feat(gateway): broker stable sandbox SPIFFE provider tokens by drew · Pull Request #4732 · NVIDIA/OpenShell

drew · 2026-10-20T02:05:13Z

Summary

Let the Kubernetes shared supervisor use each sandbox's stable SPIFFE identity while keeping SPIRE access, JWT-SVIDs, and provider exchanges on the gateway. The supervisor authenticates with its gateway-issued sandbox session and receives only the final provider access token to inject into an admitted upstream request; sandbox workloads receive no SPIFFE sockets or credentials.

Stacked on #4702, above #4701, #4695, and #4656. The previous supervisor-owned implementation is preserved on the backup branch.

Related Issue

Closes #4731

Changes

Add the sandbox-authenticated GetSandboxProviderAccessToken RPC. Its only input is the gateway-issued, endpoint-bound credential key. The gateway verifies the current session JWT and persisted generation, credential epoch, token lineage, and active state, then resolves provider attachment, endpoint, audience, and scopes from trusted configuration.
Move the SPIRE Delegated Identity API client into the gateway. Derive <prefix>/gateway/<gateway-id>/sandbox/<sandbox-id> from verified identity, require exactly the expected SVID, and verify its signature, audience, and expiry against the agent's trust bundle. Identity survives stop/start; recreating a sandbox produces a new identity.
Perform RFC 7523 grants or both RFC 8693 exchange stages on the gateway. Cache final tokens by sandbox generation, credential epoch, provider revision, and grant configuration. Reuse dedicated-mode cache TTL rules, including exact overrides and the default when endpoint expiry is omitted, then cap caching by SVID, session, and subject-credential expiry. Recheck authorization and configuration before returning either a new or cached token. Shared supervisors request tokens on every use and hold no independent grant cache.
Configure delegation under openshell.gateway.experimental_spiffe_delegation; Helm mounts the private admin socket only on the gateway. Preserve the existing gateway Workload API setting for its component identity. Remove shared-supervisor SPIRE flags, socket mounts, and session verification-key plumbing. Dedicated supervisors retain their existing Workload API and exchange behavior.
Update the SPIRE fixture, generated Go bindings, and existing gRPC mocks for the additive RPC. Public documentation is deferred.

SPIRE still trusts the gateway as a delegate: operators must provision sandbox registrations within the gateway agent's scope and control node placement. This does not implement assignment-scoped authorization inside SPIRE, HA, automatic registration, other-driver enablement, or validated Kata support. Already-issued provider bearer tokens retain their normal expiration.

Release Notes

Testing

Workspace Clippy (--all-targets, warnings denied), Rust formatting, and pre-commit checks.
Provider library tests: 184 passed; supervisor-network: 1,572 passed; Kubernetes driver configuration: 86 passed.
Real SPIRE integration using pinned SPIRE 1.15.4: distinct sandbox SVIDs and rejection of missing, mismatched, or ambiguous registrations.
Focused Kubernetes + SPIRE lifecycle E2E: two sandbox identities and both RFC 8693 exchanges, cache reuse, stop/start preserving identity with renewed authorization, same-name recreation changing identity, missing registration failing closed, and no SPIFFE access in sandbox or shared-supervisor Pods (34.82 s).
Helm: 358 tests across 26 suites; chart lint/render checks.
Protobuf lint and full Go SDK CI, including generated-binding checks.
Documentation validation: zero errors, three warnings.

The Kubernetes fixture pins chart versions 0.30.3 / 0.6.1 (SPIRE 1.15.3).

Review follow-up: 5 existing token TTL tests pass (including missing-expiry, exact/fractional/zero overrides); scoped Clippy for core, supervisor-network, and server passes with warnings denied; Rust formatting passes. The workspace-overlay task fix is also carried from feat(kubernetes): host sandboxes on the shared supervisor #4702. No live SPIRE/Kubernetes E2E was rerun for this follow-up.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture and operator setup documentation updated

copy-pr-bot · 2026-10-20T02:05:17Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-10-20T02:05:45Z

🌿 Preview your docs: https://nvidia-preview-pr-4732.docs.buildwithfern.com/openshell

…PIRE delegation A shared supervisor's own Workload API identity names the supervisor, so provider token grants failed closed for every hosted sandbox. With SPIRE delegation configured, a hosted session now obtains the JWT-SVID registered for its sandbox from the SPIRE agent's private admin socket through the Delegated Identity API, and uses it in the existing RFC 7523 and RFC 8693 grant flows. The SPIFFE ID derives from the immutable sandbox ID, <prefix>/gateway/<gateway-id>/sandbox/<sandbox-id>, so it survives restarts and runtime replacement. Each generation must earn its own authorization: - The registration now carries the gateway ID and session verification keys from the gateway's launch authentication. The session's current gateway JWT must verify against them and name exactly the registered sandbox, runtime generation, and credential epoch. - Before issuance, an authenticated gateway read confirms that the gateway still accepts the session's current credentials. - The agent is asked for one selector naming the gateway and sandbox. Exactly one SVID with the expected ID must come back, verified against the agent's JWT bundle; nothing falls back to another identity. - The session is re-authorized after every wait and before any cached token is served or a new one cached; a failed check discards the session's cache. Dedicated supervisors keep their Workload API identity unchanged. `mise run test:spire-delegation` runs these rules against a pinned SPIRE server and agent. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

The Kubernetes driver accepts an experimental_supervisor_spiffe_delegation table with the gateway topology: the SPIRE agent admin socket, the prefix of sandbox SPIFFE IDs, and the group that owns the admin socket. With it, the driver also accepts provider_spiffe_workload_api_socket_path for the gateway's own identity; without it, the topology still refuses provider SPIFFE identity. Both socket paths must be canonical and in separate, non-nested directories, the group must not be root, and the gateway ID must be a SPIFFE path segment. The chart passes the settings to the shared supervisor, mounts only the admin socket's directory from the node into the shared supervisor Pod, adds the socket group to its supplemental groups, and runs it as a service account of its own, so SPIRE can authorize exactly that Pod as a delegate. Gateway and workload Pods receive nothing new. SPIRE registration stays with the operator. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

…ties The e2e:kubernetes:shared-supervisor-spiffe lane installs pinned SPIRE hardened charts with the agent admin socket exposed for the shared supervisor, enables SPIRE delegation, and registers sandbox identities with ClusterStaticEntry resources as an operator would. Two sandboxes on one shared supervisor exchange a stored subject token through the existing RFC 8693 flow, each under its own SPIFFE ID, with a token service that only lets the identity an intermediate token names redeem it. The test checks that an unregistered sandbox is refused rather than served with the supervisor's identity, that a restart acquires again under the same SPIFFE ID, that a sandbox recreated under the same name has a new identity, and that no workload Pod or sandbox holds SPIFFE sockets, environment, or SVIDs. The wrapper removes the SPIRE fixture, including its CRDs, on exit. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

…visor Document the experimental_supervisor_spiffe_delegation table, the stable SPIFFE ID mapping, the SPIRE setup operators provide (admin socket and group, authorized delegate, per-sandbox registration entries), and how restarts, recreation, removal, and key rotation affect identities. Update the crate READMEs and the cluster debugging and Helm development skills. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

…ture provider_spiffe_workload_api_socket_path names a CSI mount inside gateway and dedicated supervisor Pods, while the SPIRE admin socket is a node hostPath mounted only into the shared supervisor, so comparing the two paths established nothing. The driver and chart now validate only the admin socket path itself, and the docs no longer require separate directories. The e2e SPIRE fixture refuses a cluster that already has SPIRE CRDs, creates its own namespace (openshell-e2e-spire by default) before claiming ownership, installs rather than upgrades, and removes only that namespace, its CRDs, and registration entries labeled with that namespace. The SPIRE delegation harness uses uv for host Python. The configuration reference now states that gateway session JWTs must expire and what cache hits check: the session's current JWT locally, with the gateway confirming the generation only before new issuance. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

… entries The SPIRE Controller Manager only reconciles entries of its own class, <namespace>-<release>. The fixture namespace is now openshell-e2e-spire by default and can be overridden, so the test derives the class from the fixture namespace instead of assuming the old spire-spire. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

…shell-crypto Main routes first-party key generation through the openshell-crypto facade. Generate the SPIRE delegation fixture's Ed25519 gateway key with it instead of calling rcgen's backend directly. Signed-off-by: Drew Newberry <anewberry@nvidia.com>

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew added this pull request to stack #4698 October 20, 2026 02:05

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from db063a9 to 8dfbd78 Compare October 20, 2026 16:24

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 2 times, most recently from a876a1a to 437619a Compare October 20, 2026 18:39

drew changed the title ~~feat(supervisor): delegate stable sandbox SPIFFE identities~~ feat(gateway): broker stable sandbox SPIFFE provider tokens Oct 20, 2026

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 3da8000 to 5fbece5 Compare October 20, 2026 22:10

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 5fbece5 to 1899b5b Compare October 20, 2026 22:38

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 2 times, most recently from 96ba1b7 to e0b6cec Compare October 20, 2026 22:53

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch 3 times, most recently from b4c1940 to b733887 Compare October 21, 2026 00:20

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from b733887 to 8e6786b Compare October 21, 2026 00:55

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 8e6786b to 6dd2cbb Compare October 21, 2026 01:05

FrostGod mentioned this pull request Oct 21, 2026

docs(rfc): add RFC 0018 for multi-sandbox and HA supervisors #4377

Draft

3 of 6 tasks

drew added 4 commits October 21, 2026 01:48

drew added 6 commits October 21, 2026 01:48

feat(gateway): broker sandbox SPIFFE provider access tokens

d412e91

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

test(helm): cover disabled gateway SPIRE delegation

4709a20

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

refactor(supervisor): retain shared mode entry point with SPIFFE broker

85cba03

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

drew force-pushed the feat/4731-stable-sandbox-spiffe/drew branch from 6dd2cbb to 85cba03 Compare October 21, 2026 01:48

drew added 2 commits October 20, 2026 19:22

fix(gateway): share provider token cache TTL semantics

720a166

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

chore(stack): carry workspace overlay fix into SPIFFE branch

11df558

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

Navigation Menu

Repository navigation

feat(gateway): broker stable sandbox SPIFFE provider tokens - #4732

feat(gateway): broker stable sandbox SPIFFE provider tokens#4732
drew wants to merge 12 commits into
feat/4700-shared-kubernetes-supervisor/drewfrom
feat/4731-stable-sandbox-spiffe/drew

drew commented Oct 20, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

drew commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Release Notes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

drew commented Oct 20, 2026 •

edited

Loading