Search before asking
Motivation
The gRPC connector added in #135 / #136 only supports the LITE consumption model: the source binds a
main lite topic through LiteSimpleConsumer and defers acknowledgement to a downstream operator,
which is what enables fair per-sub-topic throttling.
That leaves the most common case uncovered: consuming a plain normal topic over gRPC. Users who
just want topic + tag/SQL92 filter today have to either fall back to the remoting connector or
model their topic as a lite topic, and they inherit a downstream ack operator they do not need.
Solution
Add a SIMPLE consumption mode alongside LITE, selected by a single rocketmq.source.mode
(SQL: source.mode) option:
- Subscribe one normal topic through the SDK
SimpleConsumer with an optional tag / SQL92 filter
expression.
- Acknowledge on the source side, aligned with checkpoints — the same approach the GCP Pub/Sub
connector uses: receipt handles of emitted records are grouped per checkpoint and acked from
notifyCheckpointComplete, giving at-least-once. Un-acked messages are redelivered once their
invisible duration expires.
- Issue those acks through the very consumer that received the messages, so no second gRPC
connection and no downstream ack operator are involved.
- Abstract the two SDK Pop clients (
SimpleConsumer / LiteSimpleConsumer) behind one internal
interface, since they expose the same receive / ack / changeInvisibleDuration operations but share
no common type.
- Expose the mode and the filter options through the SQL connector as well.
LITE remains fully supported for the lite-topic + downstream-throttling scenario.
Are you willing to submit a PR?
Search before asking
Motivation
The gRPC connector added in #135 / #136 only supports the LITE consumption model: the source binds a
main lite topic through
LiteSimpleConsumerand defers acknowledgement to a downstream operator,which is what enables fair per-sub-topic throttling.
That leaves the most common case uncovered: consuming a plain normal topic over gRPC. Users who
just want
topic + tag/SQL92 filtertoday have to either fall back to the remoting connector ormodel their topic as a lite topic, and they inherit a downstream ack operator they do not need.
Solution
Add a SIMPLE consumption mode alongside LITE, selected by a single
rocketmq.source.mode(SQL:
source.mode) option:SimpleConsumerwith an optional tag / SQL92 filterexpression.
connector uses: receipt handles of emitted records are grouped per checkpoint and acked from
notifyCheckpointComplete, giving at-least-once. Un-acked messages are redelivered once theirinvisible duration expires.
connection and no downstream ack operator are involved.
SimpleConsumer/LiteSimpleConsumer) behind one internalinterface, since they expose the same receive / ack / changeInvisibleDuration operations but share
no common type.
LITE remains fully supported for the lite-topic + downstream-throttling scenario.
Are you willing to submit a PR?