[matthewgrossman]

Summary

Ports the old conformance, oci-image, provider-refresh, and Podman userns suites onto testkit and adds the ones the runner can provision to the cpu-tests plan, so they run in Branch E2E through the plan. This is step 4 of the test harness plan, stacked on the plan-driven runner.

It adds the new suites without removing anything: the old suites, the hand-written release lanes, and the old crates are untouched. A later PR moves the release lanes onto plans and deletes the old suites once these have run green.

The sandbox-lifecycle, file-transfer, and policy-advisor ports start from @elezar's draft #4629.

Stack

Part of a stack of four; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its tests/tmachine/ paths.

1. test(testkit): add the testkit library for end-to-end suites #4724: the testkit library.
2. test(sandbox): run a testkit suite in Branch E2E #4822: the first testkit suite, run in Branch E2E against a Docker gateway. The simplest route to testkit tests in CI.
3. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: the test runner, plans, and tmachine Session API, with a plan running on CI.
4. test(suites): port the old suites onto testkit and run them from the plan #4774: ports the old suites onto testkit and adds them to the plan.

Still to come, on top of #4774: moving the release lanes onto plans and deleting the old suites, once the plan has run the ported suites green.

Related Issue

Part of #3954

Changes

• tests/suites/conformance/sandbox: test files lifecycle, cli, persistent_workspace, file_transfer, and policy_advisor next to the existing smoke. tests/suites/conformance/sandbox-image holds the oci-image port.
• tests/suites/features: new crates providers (Keycloak provider refresh) and podman (user-namespace tests, with fixtures copied).
• tests/plans/cpu-tests.yaml: the conformance suites on the Ubuntu rootful Docker environment, and oci-image there too. The runner passes OPENSHELL_TEST_CONTAINER_ENGINE through a new optional env: map in the environment YAML.
• testkit helpers for sandboxes created through the CLI: cx.track, cx.track_sandbox, cx.adopt, cx.delete_sandboxes, and cx.cli_env.
• persistent_workspace_survives_stop_start declares requires = [persistent_workspace]. mise run e2e:testkit sets OPENSHELL_TEST_CAPABILITIES for the Docker lane.
• A tmachine test-tools.yaml playbook installs Git in the Docker environment, which the Git filtering test needs.
• fmt and clippy for the new crates in Branch Checks and the mise tasks. The features workspace is now in the lockfile alignment check, which re-locks the old crates there too.

Old test to new test

conformance/cli (via crates/openshell-conformance)

Old	New	Notes
smoke (scenario smoke: status, create, get Ready, list, exec, delete, list empty)	conformance/sandbox/tests/cli.rs::cli_create_exec_list_delete	Same CLI steps and assertions. conformance/sandbox/tests/smoke.rs::create_exec_delete is the SDK variant (already in the base PR).
sandbox_lifecycle stop_start_preserves_workspace (stop, exec while stopped fails, start, workspace sentinel survives, main command re-runs)	conformance/sandbox/tests/persistent_workspace.rs::persistent_workspace_survives_stop_start (SDK, requires = [persistent_workspace]) and conformance/sandbox/tests/cli.rs::cli_stop_start_delete (CLI: stop, exec rejected while stopped, start, stop, delete)	The old scenario's workspace sentinel check is now the SDK test; the CLI commands are covered by the CLI test. Also lifecycle.rs::restart_stopped_sandbox.
sandbox_lifecycle stopped_can_be_deleted	conformance/sandbox/tests/lifecycle.rs::stopped_sandbox_can_be_deleted (SDK)	Also deleted through the CLI at the end of cli_stop_start_delete.
file_transfer::round_trip	conformance/sandbox/tests/file_transfer.rs::upload_download_round_trip	Includes upload into existing directories, symlinked and trailing-slash destinations, large binary, single file and directory download.
file_transfer::git_filtering	conformance/sandbox/tests/file_transfer.rs::upload_respects_gitignore	.gitignore filtering, single file from a Git repo, ignored-only selection rejected unless --no-git-ignore. Needs git on the test host.
file_transfer::path_safety	conformance/sandbox/tests/file_transfer.rs::download_stays_inside_workspace	Workspace-escape rejection and dash-leading names.
policy_advisor::policy_local	conformance/sandbox/tests/policy_advisor.rs::policy_local_accepts_proposals
policy_advisor::mechanistic_proposal	conformance/sandbox/tests/policy_advisor.rs::denied_address_produces_draft_rule
policy_advisor::new_hostname_proposal	conformance/sandbox/tests/policy_advisor.rs::denied_hostname_produces_draft_rule

New with no old counterpart (from #4629, Evan Lezar): lifecycle.rs::long_running_exec_delete, run_to_completion_exit_delete, default_command_sandbox_is_listed_ready.

features/oci-image (6 tests, all in conformance/sandbox-image/tests/oci_image.rs, same names)

Old	New
custom_workdir_with_named_user	same name
custom_workdir_with_numeric_user_and_private_parents	same name
userless_image_uses_default_identity	same name
policy_identity_overrides_root_image_user	same name
default_workdir_uses_managed_workspace	same name
unwritable_custom_workdir_is_rejected	same name

Moved from the OpenShellRunner helper to Context (cx.cli, cx.track_sandbox, cx.adopt, cx.delete_sandboxes). It reads the engine from OPENSHELL_TEST_CONTAINER_ENGINE. Dropped: reading OPENSHELL_BIN directly (testkit resolves the CLI itself).

features/provider-refresh/keycloak (1)

Old	New	Notes
revoked_refresh_grant_requires_user_reauthorization	features/providers/tests/refresh_keycloak.rs::revoked_refresh_grant_requires_user_reauthorization	Still #![cfg(target_os = "linux")]; uses cx.cli_env for secret-from-environment. Not in a plan (needs a Keycloak service on the guest).

drivers/podman (1 + 1 unit)

Old	New	Notes
configured_userns_matches_podman_reference	features/podman/tests/default_userns.rs::configured_userns_matches_podman_reference	Not in a plan (needs per-userns gateway reconfiguration and a reference UID map). Fixtures copied to features/podman/fixtures/.
unit test adjacent_uid_ranges_match_a_combined_mapping	same name, same file (plain #[test])	Runs with cargo test -p podman.

Not ported

Nothing is dropped. The old crates and the openshell-conformance crate stay until the cutover PR; the diagnostics and per-step timeouts of OpenShellRunner are replaced by testkit's and nextest's (slow-timeout in the runner profile).

Not in the plan yet

• Podman rpm environments (rootful and rootless): they ran the same tests in the old lane and are defined, but the runner has not run them on CI. Add them after a first run. They need Git from test-tools.yaml too.
• k3s environments: the old lane verified that the Agent Sandbox controller reconciles before the tests (k3s-ready.yaml). The runner only waits for gateway health.
• oci-image on Podman: needs the engine name (sudo -n podman or podman) and, for rootless, XDG_RUNTIME_DIR in the runner's session.
• providers (Keycloak): needs the keycloak Ansible role and the three OPENSHELL_E2E_OIDC_* variables; the runner has no per-suite guest setup step.
• podman (userns): needs the gateway reconfigured per userns mode and a reference UID map; the runner provisions one gateway configuration per environment.

Testing

• Checks appropriate to the affected code and behavior pass
  • Against a local Docker gateway: the sandbox suites pass (15/15), oci-image passes (6/6), and the Keycloak refresh test passes (1/1, with its Linux-only gate lifted temporarily). fmt, clippy, the runner tests (59), and the lockfile and alignment checks pass. The plan dry run lists the expected tests, and the Nix outputs evaluate.
  • Not run: the new suites in a tmachine VM on CI, and anything on Podman. They need Branch E2E (test:e2e).
• Unit tests added/updated
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (TESTING.md, tests/plans/README.md)

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.