Skip to content

feat: inject sandbox UID via compute driver instead of requiring it in container images #1959

Description

@derekwaynecarr

Problem Statement

Every container image used as an OpenShell sandbox must bake in a sandbox user and group at a specific UID/GID. The bring-your-own-container docs require UID 1000660000, the VM driver hardcodes UID 10001, and the policy engine rejects any run_as_user value other than "sandbox". This creates friction for image authors, prevents compatibility with environments that allocate their own UID ranges (e.g., OpenShift SCCs), and ignores the Kubernetes-native securityContext mechanism for injecting user identity at runtime.

The goal is to eliminate the requirement that the sandbox user exist in the container image and instead have the compute driver inject the desired UID/GID at sandbox creation time. On OpenShift, the UID should be auto-detected from namespace SCC annotations.

Technical Context

The sandbox user identity flows through 6 distinct layers today — from image build time, through policy validation, to supervisor privilege dropping. Each layer assumes a user named "sandbox" exists in the container's /etc/passwd. The supervisor starts as root (UID 0) and drops privileges to the sandbox user via setuid()/setgid() after setting up network namespaces, Landlock, and seccomp. The Kubernetes driver already constructs securityContext on the pod spec (setting runAsUser: 0 for the supervisor), so extending it to pass the sandbox UID is architecturally straightforward. The proto already uses a string field for run_as_user, so numeric UIDs require no wire format change.

Affected Components

Component Key Files Role
Policy engine crates/openshell-policy/src/lib.rs Validates run_as_user/run_as_group, currently rejects anything other than "sandbox"
Supervisor (process) crates/openshell-supervisor-process/src/process.rs Validates sandbox user exists, drops privileges via setuid()/setgid(), chowns filesystem
Supervisor (SSH) crates/openshell-supervisor-process/src/ssh.rs Derives USER/HOME env vars from policy user
Kubernetes driver crates/openshell-driver-kubernetes/src/driver.rs Constructs pod spec, sets securityContext, manages PVC init containers
VM driver crates/openshell-driver-vm/src/rootfs.rs Writes sandbox user into guest rootfs at image prep time
Docker driver crates/openshell-driver-docker/src/lib.rs Uses --user on docker run
Proto proto/sandbox.proto Defines ProcessPolicy.run_as_user as string (no change needed)
BYOC example examples/bring-your-own-container/ Documents and demonstrates sandbox user creation

Technical Investigation

Architecture Overview

The sandbox user identity is established at image build time and consumed at 6 points during sandbox lifecycle:

  1. Image build — groupadd/useradd creates the sandbox user at a fixed UID in the Dockerfile
  2. Policy normalization — ensure_sandbox_process_identity() defaults empty run_as_user/run_as_group to "sandbox"
  3. Policy validation — validate_sandbox_policy() hard-rejects any non-"sandbox" value
  4. Supervisor user validation — validate_sandbox_user() calls User::from_name("sandbox") against /etc/passwd
  5. Privilege dropping — drop_privileges() resolves "sandbox" to a UID via User::from_name(), then calls setgid()/setuid() with verification
  6. Filesystem prep — prepare_filesystem() resolves the sandbox user for chown of read_write directories

The supervisor runs as root (UID 0) to create network namespaces, set up the proxy, and configure Landlock/seccomp. It drops to the sandbox UID only for child processes. The Kubernetes driver forces securityContext.runAsUser = 0 on the main container for this reason.

Code References

Location Description
openshell-policy/src/lib.rs:660-668 ensure_sandbox_process_identity() — defaults empty user/group to "sandbox"
openshell-policy/src/lib.rs:756-772 validate_sandbox_policy() — hard-rejects non-"sandbox" values for run_as_user/run_as_group
openshell-policy/src/lib.rs:680-697 PolicyViolation enum — would need new UidOutOfRange variant
openshell-supervisor-process/src/process.rs:758-786 validate_sandbox_user() — calls User::from_name("sandbox"), fails if missing from image
openshell-supervisor-process/src/process.rs:892-998 drop_privileges() — resolves name → UID via User::from_name(), calls setgid()/setuid() with verification
openshell-supervisor-process/src/process.rs:788-870 prepare_filesystem() — resolves sandbox user/group for chown of read_write directories
openshell-supervisor-process/src/ssh.rs:221-225 SSH session — derives USER/HOME from policy run_as_user, defaults to "sandbox"/"/sandbox"
openshell-driver-kubernetes/src/driver.rs:970-981 K8s driver — forces securityContext.runAsUser = 0 on supervisor container
openshell-driver-kubernetes/src/driver.rs:994+ PVC workspace init container — seeds PVCs, needs sandbox UID for chown
openshell-driver-vm/src/rootfs.rs:755-772 VM driver — hardcodes SANDBOX_UID = 10001 / SANDBOX_GID = 10001 in rootfs
proto/sandbox.proto:47-52 ProcessPolicy — run_as_user and run_as_group are string fields
examples/bring-your-own-container/Dockerfile:20-21 BYOC example — groupadd -g 1000660000 sandbox && useradd -m -u 1000660000 -g sandbox sandbox
e2e/rust/tests/custom_image.rs:27-28 E2E test image — same 1000660000 pattern

Current Behavior

When a sandbox is created:

  1. The policy's run_as_user is defaulted to "sandbox" if empty, then validated — only "sandbox" is accepted.
  2. The supervisor calls User::from_name("sandbox") against the container's /etc/passwd. If the user doesn't exist, startup fails with: "sandbox user 'sandbox' not found in image; all sandbox images must include a 'sandbox' user and group".
  3. drop_privileges() resolves "sandbox" → numeric UID via User::from_name(), then calls setgid()/setuid() with post-drop verification (defense-in-depth: confirms UID changed, confirms root can't be re-acquired).
  4. prepare_filesystem() resolves the sandbox user for chown of read_write directories before forking the child process.

What Would Need to Change

Policy engine — Relax the hard "sandbox" string check to also accept numeric UID strings within a platform-level range:

const MIN_SANDBOX_UID: u32 = 1000;
const MAX_SANDBOX_UID: u32 = 2_000_000_000;

Accept "sandbox" (existing) or any u32 in [MIN_SANDBOX_UID, MAX_SANDBOX_UID]. Reject "root", UID 0, system UIDs below 1000, and non-numeric garbage. Add UidOutOfRange violation variant for clear error messages. The range is a platform safety constant, not a per-policy knob.

Supervisor — validate_sandbox_user(), drop_privileges(), and prepare_filesystem() must accept numeric UIDs:

  • If the value parses as u32, skip /etc/passwd lookup and use the UID directly (setuid()/setgid() do not require a passwd entry).
  • If it's a name, keep the existing name-based lookup.
  • SSH session should derive USER=sandbox and HOME=/sandbox as defaults when no passwd entry exists.

Kubernetes driver — Add sandbox_uid/sandbox_gid to driver config. Pass the UID to the supervisor through the policy's run_as_user field. The supervisor container stays runAsUser: 0. The PVC init container uses the injected UID for chown.

OpenShift SCC-aware UID resolution — On OpenShift, read namespace annotations to auto-select the sandbox UID:

  1. Read namespace metadata via Api<Namespace>::get() (driver doesn't currently do this — requires adding the call).
  2. Parse openshift.io/sa.scc.uid-range annotation (format: <start>/<size>, e.g., 1000660000/10000). Use range start as sandbox UID.
  3. Parse openshift.io/sa.scc.supplemental-groups for GID. Fall back to UID range start if absent.
  4. If neither annotation is present (vanilla Kubernetes), fall back to configured sandbox_uid/sandbox_gid.
  5. Validate resolved UID/GID against [MIN_SANDBOX_UID, MAX_SANDBOX_UID].

This is passive detection (annotation presence) — no explicit "OpenShift mode" config flag needed.

VM driver — Use configurable UID instead of hardcoded 10001. The VM driver controls its own rootfs, so it can continue creating the user at rootfs prep time.

BYOC / docs — Remove groupadd/useradd requirement from examples and documentation.

Alternative Approaches Considered

  1. NSS module in the supervisor — Synthesize a sandbox passwd entry via custom NSS module. Adds runtime dependency and complexity. Rejected: directly using numeric UIDs is simpler and more portable.

  2. Init container running useradd — Create the user at container start. Requires the image to have user management tools and writable /etc/passwd. Rejected: many minimal images lack these tools.

  3. Better documentation only — Just improve the BYOC docs. Doesn't solve the underlying friction or OpenShift UID range incompatibility.

Patterns to Follow

  • The Kubernetes driver already constructs securityContext on pod specs (driver.rs:970-981) — the sandbox UID injection follows the same JSON manipulation pattern.
  • The policy engine already has a PolicyViolation enum with descriptive variants and Display impls — the new UidOutOfRange variant should follow the same pattern.
  • The supervisor's drop_privileges() already has defense-in-depth verification (confirms UID changed, confirms root can't be re-acquired) — numeric UID support must maintain these checks.

Proposed Approach

Implement in three phases. Phase 1 teaches the policy engine and supervisor to accept numeric UIDs within a safe range ([1000, 2_000_000_000]), removing the hard dependency on a /etc/passwd entry. Phase 2 adds sandbox_uid/sandbox_gid config to the Kubernetes driver and injects it via the policy, with passive OpenShift SCC annotation detection for automatic UID selection on OpenShift clusters. Phase 3 removes the image-side user requirement from examples, docs, and e2e tests.

Scope Assessment

  • Complexity: Medium
  • Confidence: High — clear path for Phases 1-2, Phase 3 is straightforward cleanup
  • Estimated files to change: ~11
  • Issue type: feat

Risks & Open Questions

  1. Programs requiring a passwd entry — Some programs (sudo, ssh) fail if the running UID has no /etc/passwd entry. Should the supervisor write a synthetic passwd entry at startup before dropping privileges?

  2. Home directory creation — If the image doesn't have /sandbox created, who creates it? The supervisor could create and chown it during prepare_filesystem(), but this needs to happen before the child process starts.

  3. File ownership in image layers — Files in the image owned by the old sandbox UID will appear as owned by a different user. Only affects images previously built with the sandbox user.

  4. Security boundary — The range check (MIN_SANDBOX_UID = 1000 through MAX_SANDBOX_UID = 2_000_000_000) replaces the current "sandbox" string check as the non-root invariant. It rejects UID 0, system UIDs below 1000, and unreasonably large values. The range is enforced as platform-level constants, not per-policy configurable.

  5. Docker driver — The Docker driver uses --user on docker run. Should it also adopt the configurable UID pattern, or is this Kubernetes-only initially?

  6. Gateway config docs — If sandbox_uid/sandbox_gid are added to the Kubernetes driver config, docs/reference/gateway-config.mdx and relevant compute-driver setup docs must be updated.

Test Considerations

  • Unit tests — Policy validation tests must cover: "sandbox" (pass), numeric UID in range (pass), numeric UID out of range (fail), "root" (fail), "0" (fail), non-numeric string (fail). Supervisor tests must cover numeric UID privilege dropping without passwd entry.
  • Integration tests — Kubernetes driver tests must verify pod spec includes correct securityContext when sandbox_uid is configured, and that OpenShift annotation parsing works correctly.
  • E2E tests — Update e2e/rust/tests/custom_image.rs to use an image without a baked-in sandbox user. Verify sandbox creation, privilege dropping, and filesystem ownership work with injected UIDs.
  • Existing test patterns — openshell-driver-kubernetes/src/driver.rs has tests like supervisor_sideload_injects_run_as_user_zero() that verify securityContext — new tests should follow this pattern. Policy validation tests in openshell-policy/src/lib.rs use validate_sandbox_policy() assertions — extend these.

Created by spike investigation. Use build-from-issue to plan and implement.

Activity

  1. added
    area:policyPolicy engine and policy lifecycle work
    area:sandboxSandbox runtime and isolation work
    test:e2eRequires end-to-end coverage
    area:clusterRelated to running OpenShell on k3s/docker
    on Jun 18, 2026
  2. removed
    test:e2eRequires end-to-end coverage
    on Jun 22, 2026
  3. sjenning commented on Jun 22, 2026

    @sjenning
    Collaborator

    🏗️ build-plan

    Implementation Plan

    Issue type: feat
    Complexity: Medium
    Confidence: High — the policy engine and supervisor changes are well-scoped; Kubernetes driver changes follow established patterns. Phase 3 cleanup is trivial once Phases 1–2 land.

    Summary

    Remove the hard dependency on a baked-in "sandbox" user in container images by allowing numeric UIDs in the process identity fields (run_as_user / run_as_group) within a safe range, and adding sandbox_uid/sandbox_gid config to compute drivers that inject the resolved UID at sandbox creation time. On Kubernetes/OpenShift, the driver auto-detects the sandbox UID from namespace SCC annotations.

    Scope

    • crates/openshell-policy/src/lib.rs — Relax policy validation: accept numeric UIDs in [1000, 2_000_000_000] alongside "sandbox"; reject UID 0 and system ranges
    • crates/openshell-supervisor-process/src/process.rs — Accept numeric UIDs for privilege dropping without requiring /etc/passwd entry; handle filesystem chown with numeric UIDs
    • crates/openshell-supervisor-process/src/ssh.rs — Derive USER/HOME env vars gracefully when no passwd entry exists
    • crates/openshell-driver-kubernetes/src/config.rs — Add sandbox_uid/sandbox_gid optional config fields with OpenShift annotation detection
    • crates/openshell-driver-kubernetes/src/driver.rs — Inject sandbox UID into supervisor container securityContext and PVC init container chown; resolve from annotations or config
    • crates/openshell-driver-vm/src/rootfs.rs — Accept configurable sandbox_uid/sandbox_gid instead of hardcoded constants (add to driver config)
    • examples/bring-your-own-container/Dockerfile — Remove groupadd/useradd; document that images no longer need a baked-in sandbox user
    • docs/reference/gateway-config.mdx / docs/reference/sandbox-compute-drivers.mdx — Document new driver config options
    • Tests — Unit tests for policy validation, supervisor UID handling; integration test for K8s driver annotation parsing

    Implementation Steps

    Phase 1: Policy engine accepts numeric UIDs

    1. Add constants MIN_SANDBOX_UID = 1000 and MAX_SANDBOX_UID = 2_000_000_000 to openshell-policy/src/lib.rs.
    2. Replace PolicyViolation::InvalidProcessIdentity with PolicyViolation::UidNotInSandboxRange { field, value } (or keep existing variant but change error message). The Display impl should clarify that "sandbox" or a valid numeric UID is accepted.
    3. In validate_sandbox_policy(), instead of hard-checking for "sandbox", accept either:
      • "sandbox" (backward-compatible string name)
      • A string that parses as u32 within [MIN_SANDBOX_UID, MAX_SANDBOX_UID]
    4. Reject: UID 0, strings < 1000 or > 2_000_000_000, non-numeric non-"sandbox" values (e.g., "root", "nobody").
    5. Update ensure_sandbox_process_identity() — only default to "sandbox" when empty; never inject a UID. The UID comes from the driver config at runtime.
    6. Add unit tests covering: "sandbox" (pass), numeric UID in range (pass), UID out of range (fail), "root" (fail), "0" (fail), non-numeric string like "nobody" (fail).

    Phase 2: Supervisor accepts numeric UIDs

    1. In validate_sandbox_user() (process.rs), accept both name-based and UID-based identities:
      • If value parses as u32, skip passwd lookup (write OCSF log confirming numeric UID usage)
      • If it is "sandbox", keep existing passwd lookup behavior
      • Return error for non-numeric non-"sandbox" values that fail passwd lookup
    2. In prepare_filesystem(), accept numeric UIDs:
      • If uid is a parsed u32, pass directly to nix::unistd::chown() (no passwd lookup needed)
      • If it is a name, keep existing passwd-based resolution
    3. In drop_privileges(), accept numeric UIDs:
      • If uid parses as u32, use it directly (skip User::from_name())
      • Maintain all defense-in-depth checks (UID/GID verification after drop)
      • For groups: if gid is numeric, skip passwd lookup; otherwise keep name-based resolution
    4. In ssh.rs session_user_and_home(), when the value is a numeric UID, fall back to default HOME (/sandbox) instead of trying a passwd lookup that will fail. The USER env var should be set to the UID string so the agent session still has a meaningful identifier.

    Phase 3: Kubernetes driver config + OpenShift detection

    1. Add sandbox_uid: Option<u32> and sandbox_gid: Option<u32> fields to KubernetesComputeConfig in crates/openshell-driver-kubernetes/src/config.rs. Both default to None.
    2. In the K8s driver, implement passive OpenShift SCC detection:
      • On sandbox creation, read namespace annotations via Api<Namespace>::get() (add k8s-api import for namespace API)
      • Parse openshift.io/sa.scc.uid-range annotation (format: <start>/<size>) to extract UID
      • Parse openshift.io/sa.scc.supplemental-groups for GID if present, otherwise use UID as GID
      • If annotations are absent or not in OpenShift format, fall back to configured sandbox_uid/sandbox_gid from config
      • Validate resolved UID/GID against [MIN_SANDBOX_UID, MAX_SANDBOX_UID] (reuse policy constant)
    3. In the supervisor container spec construction (apply_supervisor_sideload), pass the resolved sandbox UID to the supervisor via an environment variable (SANDBOX_UID) in addition to setting it in the policy's run_as_user. The policy field carries the UID to child processes; the env var helps the supervisor validate its own UID.
    4. For PVC init containers that chown workspace data, use the resolved sandbox UID/GID instead of hardcoded user names.

    Phase 4: VM driver and examples/docs cleanup

    1. Add sandbox_uid/sandbox_gid config to the VM driver (check if there's a shared driver config or if it needs its own). Replace hardcoded SANDBOX_UID = 10001 in rootfs prep with the configurable value.
    2. In the BYOC example Dockerfile, remove the groupadd/useradd lines. Update comments to explain that sandbox UIDs are now injected at runtime.
    3. Update gateway config docs (docs/reference/gateway-config.mdx) to document new driver config options.
    4. Update compute-driver setup docs if applicable.

    Test Plan

    • Unit tests:
      • Policy validation: test all acceptance/rejection paths for run_as_user and run_as_group (uid, name, out-of-range, invalid)
      • Supervisor privilege dropping: test with numeric UID (no passwd entry), "sandbox" name, and edge cases (gid mismatch verification)
      • Kubernetes driver: test OpenShift annotation parsing with sample annotations; test fallback to config values; test that resolved UID is within range
    • Integration tests:
      • Kubernetes driver pod spec generation includes correct securityContext.runAsUser for supervisor container and runAsUser/init container chown with numeric UID
      • Supervisor process.rs: verify privilege dropping works when uid has no passwd entry (mock /etc/passwd or test in isolated environment)
    • E2E tests:
      • Update e2e/rust/tests/custom_image.rs — the example image currently creates user at UID 1000660000. Verify it still works with the new policy engine (the UID is now valid). Optionally add a test that uses an image without a pre-created user, relying entirely on injected UID.

    Risks & Open Questions

    1. Programs requiring passwd entry — The spike flagged sudo/ssh failure without /etc/passwd. For now, this is acceptable: the supervisor drops privileges before starting child processes, and most sandbox workloads don't need sudo. If future issues report broken tools, a synthetic passwd write in prepare_filesystem() can be added later.
    2. Docker driver — The Docker driver already uses --user 0 for the supervisor container (confirmed at line 2283). It passes the policy's run_as_user to child processes. The Docker driver can adopt the configurable UID pattern in a follow-up issue if needed, but it's not in scope for this build.
    3. Image ownership drift — Images built with sandbox user at a different UID will show file ownership as "owned by 1000660000" when viewed from outside the container. This is benign and noted in the spike.

    Documentation Impact

    • crates/openshell-driver-kubernetes/src/config.rs — new config fields
    • docs/reference/gateway-config.mdx — document new sandbox_uid/sandbox_gid driver config options (required by AGENTS.md when adding gateway TOML keys)
    • docs/reference/sandbox-compute-drivers.mdx — if compute-driver setup docs cover UID requirements, update to note the injected UID pattern
    • examples/bring-your-own-container/Dockerfile — comment changes explaining new behavior

    Revision 1 — initial plan based on spike investigation (issue #1959)

  4. added this to the OpenShell Beta milestone on Jun 23, 2026
  5. 5 remaining items

  6. github-actions commented on Jul 8, 2026

    @github-actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  7. matthewgrossman commented on Jul 16, 2026

    @matthewgrossman
    Member

    Tracking split:

    • #2331 completes Docker and Podman runtime UID/GID injection, shared child environment normalization, and userless-image validation. A single OpenShell PR should close this issue.
    • NVIDIA/OpenShell-Community#87 migrates the Community images and release/tag rollout after a compatible OpenShell release is available.

    #1959 remains the umbrella for the completed and remaining work; neither child should close it directly.

  8. mikemarrotte-dell commented on Jul 21, 2026

    @mikemarrotte-dell

    Following this closely — we've been prototyping the driver-injected UID path against a stock K8s cluster (no OpenShift) and hit one small plumbing gap that would be worth folding into this issue's scope, or tracking as a follow-up if you'd prefer to keep #1959 tightly scoped.

    The Kubernetes driver already accepts sandbox_uid / sandbox_gid (via TOML at [openshell.drivers.kubernetes] or the OPENSHELL_K8S_SANDBOX_UID / _GID env vars — see crates/openshell-driver-kubernetes/src/main.rs:119-123 and src/config.rs). But the helm chart's gateway ConfigMap template (deploy/helm/openshell/templates/gateway-config.yaml) doesn't render either field, even though it already renders every other optional KubernetesComputeConfig field with a {{- if .Values... }} pattern (image_pull_policy, image_pull_secrets, workspace_default_storage_size, default_runtime_class_name, app_armor_profile, supervisor_image_pull_policy, provider_spiffe_workload_api_socket_path). So on a helm-installed gateway the only reachable knob today is the openshift.io/sa.scc.uid-range namespace annotation — which does work on vanilla K8s (we're using it that way), but is an awkward, OpenShift-branded lever for non-OpenShift clusters and hides the driver's platform-agnostic sandbox_uid config from helm users entirely.

    Rough shape of the fix (happy to send a PR):

    • deploy/helm/openshell/values.yaml: add server.sandboxUid / server.sandboxGid (both optional, unset by default).
    • deploy/helm/openshell/templates/gateway-config.yaml: render them under [openshell.drivers.kubernetes] when set, matching the {{- if .Values... }} pattern already used by the sibling fields listed above.

    That would make the driver's own sandbox_uid config a first-class helm value instead of requiring either a namespace annotation or a manual ConfigMap patch, and would line up with the docs/reference/gateway-config.mdx update already listed under "Risks & Open Questions" in the issue body.

    Happy to open a standalone issue if you'd rather keep #1959 focused on the driver / policy / supervisor work.

  9. mikemarrotte-dell commented on Aug 11, 2026

    @mikemarrotte-dell

    For anyone following: I filed the Helm plumbing scope as a standalone issue so #1959 can stay tightly scoped to driver/policy/supervisor work — see #2697 (feat(helm): expose sandbox_uid / sandbox_gid as first-class Helm values). Design details, alternatives, and code refs in that issue's body.

  10. added a commit that references this issue on Aug 24, 2026
  11. removed this from the OpenShell 0.1.0 milestone on Aug 26, 2026
  12. github-actions commented on Sep 10, 2026

    @github-actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:clusterRelated to running OpenShell on k3s/dockerarea:policyPolicy engine and policy lifecycle workarea:sandboxSandbox runtime and isolation workspikestate:review-readyReady for human reviewstate:staleInactive item at risk of automatic closure.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions