Skip to content

Go SDK: read the Dag run conf into a struct with typed fields - #75602

Merged
jason810496 merged 5 commits into
apache:mainfrom
henry3260:go-sdk-dag-run-conf
Nov 14, 2026
Merged

jason810496 merged 5 commits into
apache:mainfrom
henry3260:go-sdk-dag-run-conf

Conversation

@henry3260

@henry3260 henry3260 commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Why

A Go task reads the Dag run conf from actx.DagRun().Conf, which is a map[string]any. Every key therefore costs a type assertion, and a wrong one fails as a panic or as a silent zero value rather than as an error the task can report. A task that reads several keys carries a block of assertions in front of the work it actually does.

Reading the conf into a struct puts the types where the task author declares them, but the names do not line up on their own: a conf key is written target_table where the Go field has to be TargetTable, and encoding/json matches a key to a field only when the two differ at most in case. Folding case and underscores is how a TaskFlow argument already reaches an untagged field, so a conf key can reach its field by the same folding.

The Java SDK already offers this as context.dagRun.getConf(type), with the same folding rule (see airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst). This gives the Go SDK the equivalent.

related: #74674 — that PR exposed Conf itself and has merged; this one adds the typed read on top of it, which is the split agreed in review.

What

  • go-sdk/sdk/conf.go (new) adds DagRun.UnmarshalConf(target any). It renames each conf key to the name of the field that key matches, then decodes the result with encoding/json:

    • a key reaches the field whose name matches it once case and underscores are ignored, and a key that names a field exactly wins over one that only folds onto it;
    • a json: tag renames the field and matches the same way, and json:"-" takes no key — including on an embedded struct, whose fields encoding/json also skips;
    • objects nested in the conf follow the same rule, including objects in slices, arrays and map values, while the keys of a map field stay as written, because they are data rather than field names;
    • a type that decodes itself, such as time.Time, is left alone;
    • a key that matches no field is ignored, because another task in the run may read it, so a field with no matching key keeps the value the target already holds and defaults can be set before the call;
    • it returns an error when a value does not fit the type of its field, when two keys match one field and neither names it exactly, and when two fields of a struct differ only in case or underscores.

    Field indexes are cached per struct type in a sync.Map, so a conf holding a list of many objects indexes each struct once.

  • A number that lands in a field of type any, or in the values of a map[string]any or []any field, is decoded as a json.Number rather than a float64, since a float64 cannot hold every integer a conf carries — an id above 2^53 would otherwise reach the task with its last digits changed and no error to show for it. The task reads the exact value through Int64 or Float64; a field with a Go number type is unaffected.

  • go-sdk/sdk/context.go points the Conf doc comment at UnmarshalConf.

  • airflow-core/docs/authoring-and-scheduling/language-sdks/go.rst adds a "Reading the Dag run conf" section with those rules and links to it from the existing Conf paragraph.

  • go-sdk/sdk/conf_test.go (new) covers the matching rules, nested objects, json: tags, promoted and shadowed fields, the skipped embedded struct, the json.Number decoding above, a run triggered without a conf, and each error case.

go test ./... and go test -race ./... pass in go-sdk, and breeze build-docs apache-airflow task-sdk builds the new docs section.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5) following the guidelines

🤖 Generated with Claude Code

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the PR to fill up the gap.
IIUC, this PR will only focus on the UnmarshalConf and #74674 will focus on exposing the conf attribute directly right?

Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
@henry3260

Copy link
Copy Markdown
Member Author

Nice, thanks for the PR to fill up the gap. IIUC, this PR will only focus on the UnmarshalConf and #74674 will focus on exposing the conf attribute directly right?

Yes, I’ll wait until #74674 is merged

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just merged #74674 btw, please feel free to ping me when this PR is ready.

A conf read from the map gives the value of a key as an any, so a task
that reads several keys carries a type assertion per key, and a wrong
one fails as a panic or as a silent zero value rather than as an error
the task can report.

Reading the conf into a struct leaves the types to the task author, but
the names do not line up on their own: a conf key is written as
target_table, where the Go field has to be TargetTable, and
encoding/json matches a key to a field only when the two differ at most
in case. Folding case and underscores is how a TaskFlow argument
already reaches an untagged field, so a conf key reaches its field by
the same folding.
@henry3260 henry3260 changed the title Go SDK: expose the Dag run conf and read it into a typed struct Go SDK: read the Dag run conf into a struct with typed fields Nov 11, 2026
A number that lands in a field of type any, or in the values of a
map[string]any or []any field, was decoded as a float64, which cannot
hold every integer a conf carries: an id above 2^53 could reach the
task with its last digits changed and no error to show for it.

Decoding with UseNumber keeps the literal, so the task reads an integer
exactly through Int64. A field with a Go number type is unaffected,
because the decoder converts into it as before.
@henry3260
henry3260 marked this pull request as ready for review November 11, 2026 09:21
@henry3260

Copy link
Copy Markdown
Member Author

cc @jason810496

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update.

Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
Comment thread go-sdk/sdk/conf_test.go Resolved
Comment thread go-sdk/sdk/conf.go Resolved
encoding/json cannot allocate an embedded pointer whose field name is
unexported, so a conf key that reached one of its fields through a nil
pointer failed the whole read. Keys are matched per type, and a type
cannot tell a nil pointer from one the task set before the call, so the
SDK matches no conf key to the fields of such a pointer either way.

Reading a DagRun whose Conf is nil, such as one a task's unit test
builds, changed the target it was given:

- a struct or map target that does not decode itself got an empty
  object, which allocated a nil map or a nil pointer to either;
- any other target got null, which reset an any, a slice or a pointer
  to nil and passed null to UnmarshalJSON.

Airflow stores an empty conf for a run triggered without one, so such a
run still reads as an empty object.
The Go SDK docs listed the key-to-field matching rules and the error
cases that DagRun.UnmarshalConf's godoc already states, and the two had
already drifted apart, so a reader could follow either and get a
different answer. The example stays, so the page still shows how a task
makes the call.

The godoc also did not say that the rest of a `json:` tag keeps its
encoding/json meaning, which left a `,string` field looking as though a
numeric conf value would reach it.
The Dag run conf reader and the TaskFlow argument binder each carried a
byte-identical fold helper, so the two could start folding names
differently without anything failing. pkg/binding imports sdk, so the
helper cannot live in pkg/binding, and in sdk it would have to be
exported; an internal package shares it without adding to the API.
@henry3260

henry3260 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member Author

Thanks All commets are addressed

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update.

@jason810496
jason810496 merged commit 2935fcb into apache:main Nov 14, 2026
85 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants