[GLUTEN-12597][CORE] Remodel Kafka read onto ReadRel.ExtensionTable (Substrait 0.98) by nielspardon · Pull Request #12841 · apache/gluten

nielspardon · 2026-08-20T15:55:12Z

What changes are proposed in this pull request?

Part of #12597.

This makes ReadRel.read_type verbatim upstream Substrait 0.98 by removing Gluten's last graft on it -- the bool stream_kafka = 1000 discriminator and the nested ReadRel.StreamKafka message -- and remodels Gluten's ClickHouse Kafka streaming read onto the official extension_table = 8 path, the same mechanism MergeTree and Range already use. read_type now matches upstream exactly, which never allocated field 1000, so the retired tag is not reserved.

The StreamKafka payload moves out of algebra.proto into a new Gluten-owned kafka.proto (package gluten, org.apache.gluten.proto), packed into a google.protobuf.Any and carried in ReadRel.ExtensionTable.detail. The native consumer discriminates a Kafka read by the detail's type_url via detail().Is<gluten::StreamKafka>() -- mirroring the already-merged Velox Iceberg idiom enhancement().Is<gluten::IcebergReadExtension>(). This Any-in-an-official-extension-field pattern is the preferred way to carry Gluten payloads going forward, rather than grafting new fields onto the vendored Substrait messages; WriteRel.bucket_spec (field 1000) remains the one legacy graft.

Compatibility: this is a JVM-producer + ClickHouse-consumer change with no wire-compatibility constraint -- Gluten plans are transient, and the JAR and native library are generated from one proto source and ship together. Two side effects are worth calling out. The Kafka split-info payload changes from a bare StreamKafka to a ReadRel.ExtensionTable wrapping it; and read_type, which the unconditional stream_kafka flag used to set on every ReadRel, is now left unset for non-Kafka scans (no native code reads read_type_case). A JAR and native library must therefore be rebuilt together. This path is ClickHouse-only -- Kafka has no Velox path.

How was this patch tested?

Locally: standalone protoc confirms the post-delete read_type oneof is well-formed and that the relocated gluten.StreamKafka generates cleanly under both native codegen styles (flat kafka.pb.h). The JVM producer builds green -- mvn -Pspark-3.5 -Pkafka -pl gluten-core,gluten-substrait,gluten-kafka -am -DskipTests clean install (-Pkafka is required; clean clears stale generated .java from the removed message) -- with no lingering references to ReadRel.StreamKafka / setStreamKafka and scalastyle/checkstyle passing in-phase.

The native ClickHouse local-engine build (ReadRelParser.cpp, StreamKafkaRelParser.cpp, generated kafka.pb.*) needs the Linux/Docker toolchain and is verified here by inspection only. The Kafka runtime suites (GlutenKafkaScanSuite / ClickhouseGlutenKafkaScanSuite) require -Pkafka plus an external localhost:9092 broker and are not run by any in-repo GitHub Actions job, so the end-to-end Kafka path is not exercised in CI. A maintainer running -Pkafka against a broker before merge would be the real end-to-end check; the JVM toProtobuf and the CH deserialize are kept as literal mirrors of the existing extension-table readers to minimize that risk.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

🤖 Generated with AI

github-actions · 2026-08-20T16:57:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:00:01Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:04:24Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:08:34Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:32:36Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-20T17:56:07Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

This PR continues the Substrait 0.98 ReadRel.read_type migration by removing Gluten’s legacy Kafka graft from the vendored Substrait proto and modeling ClickHouse Kafka reads via the official ReadRel.ExtensionTable path, carrying a Gluten-owned gluten.StreamKafka payload in google.protobuf.Any.

Changes:

Remove stream_kafka / ReadRel.StreamKafka from algebra.proto and rely on ReadRel.ExtensionTable + Any type discrimination for Kafka.
Introduce gluten-owned kafka.proto (gluten.StreamKafka) and update JVM producer + ClickHouse native parser accordingly.
Add a shared JVM helper to pack ReadRel.ExtensionTable and adjust build wiring (protobuf dependency + CMake glob).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
gluten-substrait/src/main/scala/org/apache/gluten/utils/SubstraitUtil.scala	Adds helper to pack Gluten payloads into `ReadRel.ExtensionTable.detail`.
gluten-substrait/src/main/scala/org/apache/gluten/utils/SubstraitPlanPrinterUtil.scala	Extends TypeRegistry so `Any`-carried Gluten messages can print/parse as JSON.
gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto	Makes `ReadRel.read_type` match upstream (removes Kafka graft; keeps official fields).
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/StreamKafkaSourceNode.java	Emits split-info payload as `ReadRel.ExtensionTable` containing `gluten.StreamKafka` in `Any`.
gluten-substrait/src/main/java/org/apache/gluten/substrait/rel/ReadRelNode.java	Replaces boolean Kafka discriminator with an `extension_table.detail` `Any`.
gluten-kafka/src/main/scala/org/apache/gluten/execution/MicroBatchScanExecTransformer.scala	Marks Kafka reads by stamping `gluten.StreamKafka` type into the in-plan `extension_table`.
gluten-kafka/pom.xml	Moves `protobuf-java` to provided scope for compilation/runtime needs.
gluten-core/src/main/resources/org/apache/gluten/proto/kafka.proto	Adds Gluten-owned `StreamKafka` message definition for `Any` payloads.
docs/developers/SubstraitModifications.md	Updates Substrait modification inventory (removes old `StreamKafka` entry).
cpp-ch/local-engine/proto/CMakeLists.txt	Makes proto glob reconfigure-aware (`CONFIGURE_DEPENDS`).
cpp-ch/local-engine/Parser/SerializedPlanParser.cpp	Ensures Kafka reads always consume a split-info entry (asserts extension_table present).
cpp-ch/local-engine/Parser/RelParsers/StreamKafkaRelParser.cpp	Parses split-info as `ReadRel::ExtensionTable` and unpacks `gluten.StreamKafka` from `Any`.
cpp-ch/local-engine/Parser/RelParsers/ReadRelParser.cpp	Detects Kafka reads by `extension_table.detail().Is<gluten::StreamKafka>()`.
backends-clickhouse/src/main/java/org/apache/spark/sql/execution/datasources/clickhouse/ExtensionTableNode.java	Reuses the new helper to build extension tables.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-21T05:50:48Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-08-21T06:34:49Z

Run Gluten Clickhouse CI on x86

nielspardon · 2026-08-21T08:31:31Z

Run Gluten Clickhouse CI

github-actions · 2026-08-27T12:59:01Z

Run Gluten Clickhouse CI on x86

…Substrait 0.98) Part of apache#12597. Stacked on apache#12832 (the iceberg_table half of the ReadRel read_type migration). Makes ReadRel.read_type verbatim upstream Substrait 0.98 by removing Gluten's last graft on it -- the `bool stream_kafka = 1000` discriminator and the nested `ReadRel.StreamKafka` message. read_type now matches upstream exactly, which never allocated field 1000, so the retired tag is not reserved. Gluten's Kafka streaming read is remodeled onto the official `extension_table = 8` path, the mechanism MergeTree and Range already use. The StreamKafka payload moves to a new Gluten-owned `kafka.proto` (`package gluten`, `org.apache.gluten.proto`), packed into a `google.protobuf.Any` and carried in `ReadRel.ExtensionTable.detail`. The native consumer discriminates a Kafka read by the detail's type_url (`detail().Is<gluten::StreamKafka>()`) -- mirroring the already-merged Velox Iceberg idiom (`enhancement().Is<gluten::IcebergReadExtension>()`). This Any-in-an-official-extension-field pattern is preferred over grafting new fields; `WriteRel.bucket_spec` (field 1000) remains the one legacy graft. Gluten plans are transient and the JAR + native library are generated from one proto source and ship together, so there is no wire-compatibility constraint. Two side effects: the Kafka split-info payload changes from a bare StreamKafka to a ReadRel.ExtensionTable wrapping it, and read_type -- previously set on every ReadRel by the unconditional stream_kafka flag -- is now left unset for non-Kafka scans (no native code reads read_type_case). A JAR and native library must therefore be rebuilt together. ClickHouse-only; Kafka has no Velox path. Generated-by: Claude Code (Claude Opus 4.8)

github-actions · 2026-09-28T08:16:23Z

Run Gluten Clickhouse CI on x86

nielspardon · 2026-09-28T11:35:43Z

Rebased and fixed conflicts

zzcclp · 2026-10-03T08:48:13Z

LGTM

zhouyuan

👍

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch 4 times, most recently from 550be87 to bedfb96 Compare August 20, 2026 16:55

github-actions Bot added CORE works for Gluten Core CLICKHOUSE DOCS labels Aug 20, 2026

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from bedfb96 to 0f4d98c Compare August 20, 2026 17:46

zhouyuan requested a lite review from Copilot August 20, 2026 19:17

Copilot AI reviewed Aug 20, 2026

View reviewed changes

Comment thread gluten-substrait/src/main/resources/substrait/proto/substrait/algebra.proto

Comment thread gluten-substrait/src/main/scala/org/apache/gluten/utils/SubstraitPlanPrinterUtil.scala

Copilot started reviewing on behalf of zhouyuan August 20, 2026 22:18

View session

nielspardon marked this pull request as ready for review August 21, 2026 05:44

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from 0f4d98c to 54cd984 Compare August 21, 2026 05:50

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from 54cd984 to 3264a7f Compare August 21, 2026 06:32

zhouyuan requested a review from exmy August 21, 2026 09:08

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from 3264a7f to 14f322f Compare August 27, 2026 12:39

exmy requested a review from zzcclp September 2, 2026 02:24

nielspardon force-pushed the feat/substrait-0.98-kafka-extension-table branch from 14f322f to e31e44a Compare September 28, 2026 08:14

zhouyuan approved these changes Oct 6, 2026

View reviewed changes

zhouyuan merged commit 6c1baac into apache:main Oct 6, 2026
60 checks passed

ReemaAlzaid mentioned this pull request Oct 20, 2026

[VL] Translate Kafka Substrait reads #13236

Merged

Conversation

nielspardon commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

nielspardon commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

nielspardon commented Sep 28, 2026

Uh oh!

zzcclp commented Oct 3, 2026

Uh oh!

zhouyuan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nielspardon commented Aug 20, 2026 •

edited

Loading