Skip to content

Support normal-topic consumption with the gRPC SimpleConsumer #137

Description

@lizhimins

Search before asking

  • I had searched in the issues and found no similar issues.

Motivation

The gRPC connector added in #135 / #136 only supports the LITE consumption model: the source binds a
main lite topic through LiteSimpleConsumer and defers acknowledgement to a downstream operator,
which is what enables fair per-sub-topic throttling.

That leaves the most common case uncovered: consuming a plain normal topic over gRPC. Users who
just want topic + tag/SQL92 filter today have to either fall back to the remoting connector or
model their topic as a lite topic, and they inherit a downstream ack operator they do not need.

Solution

Add a SIMPLE consumption mode alongside LITE, selected by a single rocketmq.source.mode
(SQL: source.mode) option:

  • Subscribe one normal topic through the SDK SimpleConsumer with an optional tag / SQL92 filter
    expression.
  • Acknowledge on the source side, aligned with checkpoints — the same approach the GCP Pub/Sub
    connector uses: receipt handles of emitted records are grouped per checkpoint and acked from
    notifyCheckpointComplete, giving at-least-once. Un-acked messages are redelivered once their
    invisible duration expires.
  • Issue those acks through the very consumer that received the messages, so no second gRPC
    connection and no downstream ack operator are involved.
  • Abstract the two SDK Pop clients (SimpleConsumer / LiteSimpleConsumer) behind one internal
    interface, since they expose the same receive / ack / changeInvisibleDuration operations but share
    no common type.
  • Expose the mode and the filter options through the SQL connector as well.

LITE remains fully supported for the lite-topic + downstream-throttling scenario.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions