Skip to content

Spark: Emit geometry bounds in Parquet writers - #18501

Open
huan233usc wants to merge 5 commits into
apache:mainfrom
huan233usc:xin-huang_data/spark-geometry-bounds
Open

huan233usc wants to merge 5 commits into
apache:mainfrom
huan233usc:xin-huang_data/spark-geometry-bounds

Conversation

@huan233usc

@huan233usc huan233usc commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Spark 4.1 and 4.2 Parquet writers currently emit value-size metrics for geometry but omit its spatial bounds. Keep the shared geospatial WKB write path and let each private writer update its existing metrics builder: geometry uses GeometryFieldMetrics.Builder for XY bounds, while geography retains size-only metrics. Preserve the CRS from the Parquet annotation, null counts, and byte totals. Spark 4.2 continues to slice the original WKB payload to preserve byte order. Core metrics classes and public APIs are unchanged.

Test plan:

  • 55 tests: core metrics (7), Parquet writers/file metrics (30), and Spark TestSparkParquetWriter on 4.1 and 4.2 (9 each). Covers file bounds, null/empty inputs, CRS, byte totals, and WKB round-trips.
  • Modified modules: assemble, spotlessCheck, checkstyleMain, and checkstyleTest.

AI Disclosure

  • Model: GPT-6
  • Platform/Tool: Codex
  • Human Oversight: reviewed
  • Prompt Summary: Implement geometry bounds emission in Spark 4.1/4.2 while preserving the common GeospatialWriter path.
  • Generated content: Writer integration and regression tests.

@github-actions github-actions Bot added the spark label Oct 14, 2026
Share the geospatial WKB write path while preserving existing metrics builder signatures.

Generated-by: Codex
@github-actions github-actions Bot added the core label Oct 14, 2026
Xin Huang added 3 commits October 14, 2026 13:32
Keep the existing geometry counters while sharing the binary metrics contract with Spark writers.

Generated-by: Codex
Preserve shared WKB writes through a private metrics hook and keep the core metrics API unchanged.

Generated-by: Codex
Use shared assertions for all-null and empty WKB inputs while checking their distinct counts and byte totals.

Generated-by: Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant