Skip to content

[Go] Avoid importing jobopts in xlangx to prevent flag collisions - #40944

Open
PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:go-xlangx-avoid-jobopts-flags
Open

PDGGK wants to merge 1 commit into
apache:masterfrom
PDGGK:go-xlangx-avoid-jobopts-flags

Conversation

@PDGGK

@PDGGK PDGGK commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #40730.

#40729 removed the gcpopts import from gcsx. This change handles the other regression listed in the issue. #40379 made core/runtime/xlangx import options/jobopts to read --resource_hints. The root beam package imports xlangx, so any binary that imports beam now registers all of jobopts' flags on flag.CommandLine: --endpoint, --job_name, --async, --experiments, --parallelism and the rest. A binary or test that defines a flag with one of those names panics at startup.

The repro from the issue, run unchanged as a test in a separate module against each Beam version:

# github.com/apache/beam/sdks/v2 v2.77.0
ok  	repro	0.504s

# github.com/apache/beam/sdks/v2 v2.78.0-RC1
panic: /var/folders/…/repro.test flag redefined: endpoint
FAIL	repro	0.367s

# master at 99dbd880e07
panic: /var/folders/…/repro.test flag redefined: endpoint
FAIL	repro	0.377s

# master with this change
ok  	repro	0.359s

Both #40379 and #40729 are in sdks/v2.78.0-RC1. 2.78.0 therefore fixes the --project collision but still has this one.

Change

  • xlangx now looks up --resource_hints by name with flag.Lookup, as gcsx.GetGcsEndpoint does since [Go] Avoid importing gcpopts in gcsx and dataflowlib to prevent flag collisions #40729. If the flag isn't registered, nothing could have set it, and no hints are forwarded.
  • The hint parsing moves from jobopts.GetPipelineResourceHints into a new resource.ParseHints, which both packages can use without registering flags. jobopts.GetPipelineResourceHints stays as a wrapper, so its API is unchanged.
  • jobopts' stringSlice.Get() returns a []string rather than the unexported stringSlice type, so the value can be read through flag.Getter from outside the package.

After this, go list -deps ./go/pkg/beam no longer lists options/jobopts, as in v2.77.0.

The issue's long-term point, that runners register flags in init(), is out of scope here.

Testing

New tests:

  • TestNoJobSubmissionFlags (xlangx): asserts the xlangx test binary has no --endpoint, --job_name or --resource_hints flag. If the jobopts import is put back in expand.go, it fails with flag --endpoint is registered (usage "Job service endpoint (required)."), plus the same message for the other two flags.
  • TestPipelineResourceHints (xlangx): covers the lookup with the flag unregistered, and with two --resource_hints values.
  • TestResourceHintsFlagGetter (jobopts): pins the []string result of Get() that xlangx relies on. If Get() returns s again, it fails.
  • TestParseHints (resource): the previous jobopts parsing test, moved with the same inputs and assertions.

I also checked that the #40379 behaviour still works end to end, through exported API only. A test module imports jobopts (as a runner does), sets --resource_hints=min_ram=2GB, and calls beam.TryCrossLanguage against a handler registered with xlangx.RegisterHandler. The captured ExpansionRequest has beam:option:resource_hints:v1 = ["min_ram=2000000000B"], both on v2.78.0-RC1 and with this change.

Run locally:

  • go test for options/..., core/runtime/xlangx/..., runners/universal/..., runners/dataflow and the root beam package: all pass.
  • gofmt -l, go vet --copylocks=false --unsafeptr=false and staticcheck 2026.2 on the touched packages: all clean.

The full ./go/pkg/... run and the cross-language integration tests are left to CI.

xlangx imported jobopts to read the --resource_hints flag. The root beam
package imports xlangx, so every binary importing beam registered the
jobopts flags (--endpoint, --job_name, --async, --experiments and more)
on flag.CommandLine, and binaries defining flags with any of those names
panicked at startup with "flag redefined".

Look the flag up by name instead, and move the hint parsing into the
resource package, keeping jobopts.GetPipelineResourceHints as a wrapper.
@github-actions github-actions Bot added go core size/m M: 50-249 lines changed labels Nov 2, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly resolves flag redefinition panics (#40730) caused by xlangx importing jobopts (which is transitively imported by the root beam package). Moving resource hint parsing to resource.ParseHints, updating jobopts.stringSlice.Get() to return []string, and looking up --resource_hints via flag.Lookup in xlangx effectively decouples the packages while preserving cross-language resource hint forwarding.

🔍 General Feedback

  • The architectural separation between options/resource, options/jobopts, and core/runtime/xlangx is clean and aligns well with the approach used in #40729.
  • Unit tests thoroughly cover the new flag lookup behavior, flag.Getter contract on stringSlice, hint parsing, and regression prevention against job submission flag registration in xlangx.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@codecov

codecov Bot commented Nov 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.57895% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.43%. Comparing base (12bbb17) to head (9d06578).
⚠️ Report is 20 commits behind head on master.

Files with missing lines Patch % Lines
sdks/go/pkg/beam/options/resource/hint.go 79.31% 4 Missing and 2 partials ⚠️
sdks/go/pkg/beam/core/runtime/xlangx/expand.go 85.71% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #40944       +/-   ##
===========================================
- Coverage   79.83%   56.43%   -23.41%     
===========================================
  Files         494      876      +382     
  Lines       83575   154761    +71186     
===========================================
+ Hits        66722    87339    +20617     
- Misses      16853    65566    +48713     
- Partials        0     1856     +1856     
Flag Coverage Δ
go 28.98% <81.57%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label go.
Backup reviewer: lostluck (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@jrmccluskey

Copy link
Copy Markdown
Contributor

Bringing in @derrickaw since he worked on the first PR addressing this issue

@derrickaw

Copy link
Copy Markdown
Collaborator

Bringing in @derrickaw since he worked on the first PR addressing this issue

@shunping might be better with the option changes since he has been doing a lot of design work with this recently. Thanks

@derrickaw
derrickaw requested a review from shunping November 2, 2026 17:10
@shunping

shunping commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Ack! I will take a look this week.

@derrickaw
derrickaw removed their request for review November 2, 2026 18:48
@jrmccluskey
jrmccluskey removed their request for review November 3, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Go SDK pipeline options define global flags with generic names, causing flag redefinition panics

4 participants