Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
drew
added this pull request to stack #4698
October 19, 2026 02:33
|
🌿 Preview your docs: https://nvidia-preview-pr-4702.docs.buildwithfern.com/openshell |
7 tasks done
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
from
October 20, 2026 16:24
b3441ae to
76c6e76
Compare
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
2 times, most recently
from
October 20, 2026 18:39
736a940 to
ed9a02d
Compare
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
from
October 20, 2026 22:10
ed9a02d to
7047f1b
Compare
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
from
October 20, 2026 22:38
7047f1b to
021efef
Compare
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
2 times, most recently
from
October 21, 2026 00:01
a8be6b9 to
a936c68
Compare
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
2 times, most recently
from
October 21, 2026 00:55
6e9de0f to
c2f0af9
Compare
drew
changed the title
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
from
October 21, 2026 01:05
c2f0af9 to
c8a10fc
Compare
… sessions A supervisor shared by several sandboxes needs one listener for every session's runtime-initiated Sandbox Protocol connections. The runtime now sends a fixed, bounded routing preface (magic, version, length, and an opaque route handle) before its pinned TLS handshake when its bootstrap names a route. RuntimeRouter treats the handle only as a hint that selects which generation-pinned TLS identity to demand, closes unknown, removed, and replaced handles before any handshake, bounds unauthenticated handshakes globally and per route, and queues an authenticated connection only for its own session, so no peer can displace a live channel. A routed session waits up to the bootstrap grace for its first connection, because it is registered before its workload Pod is scheduled. The session_control module defines the trusted contract between a compute driver and a shared supervisor: registering one generation's runtime descriptor and gateway-issued credentials, reading its status, and removing it, over mutually authenticated TLS 1.3. Dedicated descriptors and bootstrap listeners keep their encodings. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
openshell-supervisor experimental-host replaces the idle bootstrap process with a session host. A mutually authenticated session-control API accepts registrations only from the admission certificate holder, the gateway. Each registration starts a SessionHost session that acts only with the gateway credentials issued for its generation, keeps its proxy CA, trust directory, and SSH socket under a private directory, and receives its runtime connections through the routed listener. Status reports each session's own readiness; the health socket keeps reporting process health only. Removing a session runs on a task the host owns, so it completes when the caller disconnects, concurrent removers share its result, and the route stays reserved until the session has ended and its files are gone. An identical registration is accepted again and a different one for a registered route is refused. Sessions get no upstream proxy, extra CA bundle, or provider workload identity; the Kubernetes driver refuses configurations that need them. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
generate-certs --tls-only creates just a server and a client TLS Secret from a fresh CA, with the same create-if-missing contract and a recovery hint for partial state. The Helm certgen hook uses it for private PKIs that must not carry a sandbox-JWT signing key. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
…ssions With experimental_supervisor_topology = "gateway", each new runtime generation becomes a session on the gateway's shared supervisor instead of a supervisor Pod and boundary Service of its own. The generation records its placement and session route on its Sandbox resource when it starts, and every later decision reads that record, so changing the setting never reinterprets a running generation and unknown records fail closed. The runtime identity ends with the session route, which no Pod UID can equal, so Pod authentication cannot vouch for a shared generation. Create and start add a per-sandbox egress allowance to the Pods the shared supervisor's Service selects, bind it into the boundary evidence, and register the session over session control before releasing the workload; registration does not wait for the runtime, so create still returns before the gateway persists the generation. Readiness follows the session's own status. Stop, restart, and delete remove only that session, and reconciliation suspends a workload whose session is gone and removes sessions that no current generation may run. Configurations with an upstream proxy or provider SPIFFE identity are refused. Closes #4700 Signed-off-by: Drew Newberry <anewberry@nvidia.com>
With the gateway topology enabled, the chart runs the session host as a single-replica Deployment with its Service and a NetworkPolicy that admits session control only from the gateway and runtime connections only from sandbox workload Pods. The certgen hook issues a private session-control PKI whose client credential only the gateway mounts, the gateway may read only that Service, and the chart renders the matching experimental_shared_supervisor table. The supervisor Pod has no service account token. Settings the shared supervisor cannot isolate, and a hand-written connection table, fail at render time. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
The e2e:kubernetes:shared-supervisor lane enables the gateway topology and creates two sandboxes with different policies and provider credentials. It checks that one shared supervisor process hosts both, that each session enforces its own policy and injects its own credential, that stopping one leaves the other working, that the routed listener refuses stale and unknown routes and drops a peer that names a live route without its pinned identity, and that a restart registers a fresh session on the same process. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Document the gateway supervisor topology, its chart-owned connection settings and limits, and how to debug and test it. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
…robes The shared supervisor's resource name appended its suffix before truncating, so a release fullname of 51 or 52 characters produced the gateway Service's own name. A fullname without room for the suffix now keeps a prefix and a hash of the whole fullname. Removing a session waits up to 20 s for its workload to stop and 5 s for the session runtime to shut down, but the driver gave DELETE the same 15 s budget as other session-control calls and reported normal removals as failures. The contract now defines the removal grace and a 45 s removal deadline, and deleting a sandbox no longer waits on a session whose workload is already going away. The shared host runs no supervisor OTLP trace or OCSF exporter. The driver warns at startup when the gateway configures that export, and the docs state the limit. experimental-host keeps its Unix-only imports in its Unix entry point. The e2e route probe fails when the supervisor neither refuses nor challenges a connection, or keeps a challenged connection open after a malformed TLS answer, and it checks that every probe reached the supervisor Pod. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
…l-crypto Main now routes first-party TLS providers and digests through the openshell-crypto facade. Build the session-control TLS configurations and the routed-runtime test server from its configuration provider, install the experimental host's default provider through it, and hash shared session registrations with its SHA-256 instead of a direct sha2 dependency. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
experimental-host takes --health-port instead of a Unix health socket, matching experimental-bootstrap. It binds the wildcard health listener after the session-control and runtime listeners and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks; session control stays on 5501, the Service exposes only runtime and session control, and /run/openshell remains for per-session files. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
The shared supervisor's kubelet tcpSocket probes now connect to the mTLS session-control listener (5501) instead of a separate health listener. Drop the experimental-host --health-port flag, its accept loop, and the chart's 5502 container port. Probes open and close a TCP connection; every session-control request still requires mTLS. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Drop the experimental label from the shared host module documentation so it matches the --mode=shared entry point. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
drew
force-pushed
the
feat/4700-shared-kubernetes-supervisor/drew
branch
from
October 21, 2026 01:48
c8a10fc to
a2677ef
Compare
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Merge the sandbox instance naming from the foundation branch and apply it to the shared supervisor layer. Registrations, status reports, removal bounds, the shared host's per-instance files, and the Kubernetes driver's shared placement now describe sandbox instances: one running generation of a stable Sandbox resource. The session-control protocol keeps its wire names: the /v1/sessions paths and the "sessions" list field, the openshell.ai/sandbox-runtime-supervisor-session annotation, the kubernetes.supervisor_session claim key, the --session-control-tls-dir and --session-root flags, TLS identities, and Helm rendering are unchanged. Only Rust names, comments, and diagnostics change. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Merge the single launch configuration and prepared bindings from refactor/4699-instance-launch-config/drew and build the shared host on them. SandboxInstanceHost is now the only registry of hosted instances. Each entry owns its admission fingerprint, sandbox identity, runtime thread, stop signals, and private directory, and one serial-fenced removal task stops the runtime, revokes its credentials, drops its route, and deletes its files before releasing the key. A directory that cannot be deleted keeps the key reserved and the next removal retries it. SharedHost keeps no registry, counter, or removal state; it validates registrations, writes their launch material, and serves session control. The Kubernetes driver resolves a generation's supervisor once into a GenerationLaunchTarget whose dedicated and shared variants carry the Service address, route, shared supervisor client, and verified control policy, so installation no longer matches optional targets with impossible fallbacks. Signed-off-by: Drew Newberry <anewberry@nvidia.com>
6 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turn the opt-in Kubernetes gateway supervisor Deployment into one process hosting independent sandbox instances. New generations dial its routed listener; the gateway registers trusted launch material through the private mTLS control API. Stacked on #4701.
Related Issue
Closes #4700
Changes
SandboxInstanceHostthe only registry, serial counter, runtime threads, private directories, and removal operation.SharedHostis a thin control API adapter. Repeated admissions remain idempotent, concurrent removals join one operation, and cancelled callers cannot interrupt cleanup. Cleanup failure keeps the key reserved; a later removal retries.supervisor_topology: gateway. The--addresslistener also serves native TCP health probes.Existing
/v1/sessionsroutes, JSON fields, routing preface, pinned TLS, authentication, CLI flags, and persisted annotation names remain compatible. Admission is gateway-initiated; sandbox runtime connections are sandbox-initiated.This prototype uses one replica without HA, scaling, or transparent recovery. Supervisor loss ends its hosted instances; affected sandboxes must be started again. This layer rejects upstream proxies and provider SPIFFE identity; #4732 adds the gateway SPIFFE broker. Shared instances provide console and gateway logs but do not export supervisor OTLP traces or OCSF events. Operator workspace namespaces require a
key=valuelabel selector.Release Notes
Testing
The focused shared-supervisor, SPIFFE, and dedicated lifecycle scenarios compile. A fresh-cluster live run was blocked before OpenShell deployment: containerd could not create an inotify watcher (
too many open files; hostfs.inotify.max_user_instances=128), so the node never registered. No live scenarios ran; the temporary cluster was removed. Earlier full Kubernetes runs had a credential-gating timeout in both shared and dedicated modes; isolated runs passed without raising its 300-second limit. That broader timing caveat remains.Checklist