Repository navigation
Go SDK: read the Dag run conf into a struct with typed fields - #75602
Merged
Merged
Conversation
jason810496
reviewed
Nov 1, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
Nice, thanks for the PR to fill up the gap.
IIUC, this PR will only focus on the UnmarshalConf and #74674 will focus on exposing the conf attribute directly right?
Member
Author
jason810496
reviewed
Nov 7, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
I just merged #74674 btw, please feel free to ping me when this PR is ready.
A conf read from the map gives the value of a key as an any, so a task that reads several keys carries a type assertion per key, and a wrong one fails as a panic or as a silent zero value rather than as an error the task can report. Reading the conf into a struct leaves the types to the task author, but the names do not line up on their own: a conf key is written as target_table, where the Go field has to be TargetTable, and encoding/json matches a key to a field only when the two differ at most in case. Folding case and underscores is how a TaskFlow argument already reaches an untagged field, so a conf key reaches its field by the same folding.
henry3260
force-pushed
the
go-sdk-dag-run-conf
branch
from
November 11, 2026 06:38
d5586dc to
67583ab
Compare
A number that lands in a field of type any, or in the values of a map[string]any or []any field, was decoded as a float64, which cannot hold every integer a conf carries: an id above 2^53 could reach the task with its last digits changed and no error to show for it. Decoding with UseNumber keeps the literal, so the task reads an integer exactly through Int64. A field with a Go number type is unaffected, because the decoder converts into it as before.
Member
Author
|
cc @jason810496 |
jason810496
reviewed
Nov 13, 2026
encoding/json cannot allocate an embedded pointer whose field name is unexported, so a conf key that reached one of its fields through a nil pointer failed the whole read. Keys are matched per type, and a type cannot tell a nil pointer from one the task set before the call, so the SDK matches no conf key to the fields of such a pointer either way. Reading a DagRun whose Conf is nil, such as one a task's unit test builds, changed the target it was given: - a struct or map target that does not decode itself got an empty object, which allocated a nil map or a nil pointer to either; - any other target got null, which reset an any, a slice or a pointer to nil and passed null to UnmarshalJSON. Airflow stores an empty conf for a run triggered without one, so such a run still reads as an empty object.
henry3260
force-pushed
the
go-sdk-dag-run-conf
branch
from
November 13, 2026 11:40
46840cc to
cfae309
Compare
The Go SDK docs listed the key-to-field matching rules and the error cases that DagRun.UnmarshalConf's godoc already states, and the two had already drifted apart, so a reader could follow either and get a different answer. The example stays, so the page still shows how a task makes the call. The godoc also did not say that the rest of a `json:` tag keeps its encoding/json meaning, which left a `,string` field looking as though a numeric conf value would reach it.
The Dag run conf reader and the TaskFlow argument binder each carried a byte-identical fold helper, so the two could start folding names differently without anything failing. pkg/binding imports sdk, so the helper cannot live in pkg/binding, and in sdk it would have to be exported; an internal package shares it without adding to the API.
Member
Author
|
Thanks All commets are addressed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Go task reads the Dag run conf from
actx.DagRun().Conf, which is amap[string]any. Every key therefore costs a type assertion, and a wrong one fails as a panic or as a silent zero value rather than as an error the task can report. A task that reads several keys carries a block of assertions in front of the work it actually does.Reading the conf into a struct puts the types where the task author declares them, but the names do not line up on their own: a conf key is written
target_tablewhere the Go field has to beTargetTable, andencoding/jsonmatches a key to a field only when the two differ at most in case. Folding case and underscores is how a TaskFlow argument already reaches an untagged field, so a conf key can reach its field by the same folding.The Java SDK already offers this as
context.dagRun.getConf(type), with the same folding rule (seeairflow-core/docs/authoring-and-scheduling/language-sdks/java.rst). This gives the Go SDK the equivalent.related: #74674 — that PR exposed
Confitself and has merged; this one adds the typed read on top of it, which is the split agreed in review.What
go-sdk/sdk/conf.go(new) addsDagRun.UnmarshalConf(target any). It renames each conf key to the name of the field that key matches, then decodes the result withencoding/json:json:tag renames the field and matches the same way, andjson:"-"takes no key — including on an embedded struct, whose fieldsencoding/jsonalso skips;time.Time, is left alone;Field indexes are cached per struct type in a
sync.Map, so a conf holding a list of many objects indexes each struct once.A number that lands in a field of type
any, or in the values of amap[string]anyor[]anyfield, is decoded as ajson.Numberrather than afloat64, since afloat64cannot hold every integer a conf carries — an id above 2^53 would otherwise reach the task with its last digits changed and no error to show for it. The task reads the exact value throughInt64orFloat64; a field with a Go number type is unaffected.go-sdk/sdk/context.gopoints theConfdoc comment atUnmarshalConf.airflow-core/docs/authoring-and-scheduling/language-sdks/go.rstadds a "Reading the Dag run conf" section with those rules and links to it from the existingConfparagraph.go-sdk/sdk/conf_test.go(new) covers the matching rules, nested objects,json:tags, promoted and shadowed fields, the skipped embedded struct, thejson.Numberdecoding above, a run triggered without a conf, and each error case.go test ./...andgo test -race ./...pass ingo-sdk, andbreeze build-docs apache-airflow task-sdkbuilds the new docs section.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines
🤖 Generated with Claude Code