feat: Firecracker compute driver — reuse the VM guest model with jailer, OCI images and full sandbox lifecycle #4351

User Story

I run Horus, an orchestrator that delegates coding-agent jobs to isolated sandboxes on runners I operate. I use OpenShell as the sandbox runtime and policy layer: the gateway, the host supervisor, network policy and credential mediation. On my KVM runner I wanted those sandboxes to be Firecracker microVMs (jailer-confined VMMs, no guest NIC, a small device model), so I built an external compute driver for Firecracker against the existing ComputeDriver protocol. It now runs real agent jobs in production on that runner. I'd like to know which shape the maintainers would accept for Firecracker support, and to offer this work as input.

Problem Statement

OpenShell's VM isolation runs on libkrun (openshell-driver-vm). There is no supported way to run OpenShell sandboxes as Firecracker microVMs. The maintainer prototype in #2934 (RFC 0012, closed 2026-09-30 with no immediate plans to merge) booted a fixed ext4 fixture and listed OCI images, stop, exec, PTY, port forwarding, mediated egress and restart recovery as follow-ups.

An external driver is possible today through the extension ComputeDriver protocol (#1907), but it has to depend on OpenShell internals that are not a published contract: the VM bootstrap rootfs, openshell-vm-sandbox-init.sh, the prepared image-disk layout the guest init expects on /dev/vdc, and the launch-authentication and boundary material formats.

Impact / Why This Matters

• Deployments that already operate and audit Firecracker must run a second hypervisor stack, or keep a fork, to use OpenShell's VM isolation.
• An external Firecracker driver works, but any OpenShell version bump can silently break it, because the guest artifacts and init contract it reuses are internal.
• Current workaround: pin OpenShell to one commit and re-qualify the driver on every bump. That is what we do, and it does not scale.

Proposed Design

We built and run an external driver (Go, extension ComputeDriver protocol, no gateway or protocol changes). Our view is that Firecracker support fits OpenShell best when it reuses the VM driver's guest model instead of inventing a new one. The design decisions below are the ones we would propose:

1. Same guest model as openshell-driver-vm. Read-only bootstrap root disk (/dev/vda), per-sandbox writable overlay (/dev/vdb), and for non-default images the prepared read-only image disk (/dev/vdc, /image-rootfs) that the existing guest init uses as the overlay lowerdir. Sandboxes therefore behave the same as libkrun ones: same init, same openshell-sandbox, same policy enforcement.
2. Host supervisor and generations. One host openshell-supervisor per sandbox. Fresh launch authentication per Create and Start, a generation-bound vsock endpoint (UDS bridge to Firecracker's vsock), and stale generations rejected. Snapshots stay driver-internal (no new RPCs, consistent with feat(driver-vm): snapshot/restore (suspend/resume) for idle sandboxes #1551).
3. No guest NIC. All egress goes through the supervisor and is policy-mediated: allowlisted hosts are reachable, while other hosts and direct IP connects fail with EPERM.
4. Jailer as the production mode. Each VMM is chrooted and dropped to a dedicated uid. Only the per-sandbox overlay belongs to that uid, and it is hardlinked into the chroot so guest writes survive Stop/Start. Kernel, root and image disks stay root-owned and are shared read-only (reflink or hardlink). The chroot base must share a filesystem with the state directory.
5. Images. Prepared image disks are recorded in a sha256-verified catalog: hashes are checked at driver start and file identity is rechecked before each boot. Unknown images fail ValidateSandboxCreate before any VM starts. Disks come from the VM driver's own image preparation, or are built directly for rootfs tars too large for its path (see bug: VM driver fails to create rootfs for images with files >2 GiB due to mke2fs -d bug #1762).
6. Guest kernel. Linux 6.12 from kernel.org, Firecracker's microvm CI config, plus a fragment matching openshell.kconfig: Landlock first in CONFIG_LSM, seccomp filter, pids/memory cgroups, and a serial console. ACPI is off, because Firecracker 1.10's ACPI tables leave virtio IRQs unrouted on 6.12. Every required symbol is asserted at build time.

Questions for maintainers, in order of preference from our side:

• (a) Keep Firecracker external, and have OpenShell publish the guest artifacts (bootstrap image, init contract, image-disk format) as a versioned contract, which would also serve docs(examples): document external compute driver authoring and extensibility #2418 and improve external compute-driver startup API and test coverage #2987. Or:
• (b) Upstream it as an in-tree compute driver, or as an Isolation Backend registered by endpoint once feat(isolation): register an out-of-process Isolation Backend by endpoint #3913 lands.
• (c) Fold it into the RFC 0012 direction of feat(firecracker): prototype host-supervised isolation #2934, with this driver's lifecycle, image and egress work as the starting point.

We are happy to share the code and test evidence for whichever shape you prefer.

Acceptance Criteria

• On a KVM host, a Firecracker-backed gateway takes sandboxes through create → Ready → exec.
• Policy-mediated egress holds: allowlisted hosts are reachable; other hosts and direct IPs are denied.
• Stop/Start produces a new generation, and stale authentication is rejected.
• Delete leaves no host leaks (VM, vsock paths, overlays, jailer trees), including over repeated cycles.
• Non-default OCI images run; unknown images are rejected before any VM starts.
• Jailer mode is the documented production default, with its host requirements stated.
• The guest artifacts a Firecracker driver depends on are either versioned upstream or owned by the driver, so a version bump cannot break it silently.

Alternatives Considered

• libkrun (openshell-driver-vm): works, but it is not the hypervisor these deployments run, and it does not offer the jailer's confinement model.
• The feat(firecracker): prototype host-supervised isolation #2934 prototype (RFC 0012 backend): the right long-term supervisor model. Today it lacks OCI images, exec, egress and stop/start; those are exactly what this driver adds.
• Docker, Podman and Kubernetes drivers with a VM-based runtime such as Kata (Docker driver: pick the container runtime (e.g. Kata) and show which one a sandbox got #3868): these give a VM boundary per container, but not OpenShell's own guest init, boundary protocol or image model.
• Middleware, interceptors and providers: none of these extension points can change the isolation layer, so they do not cover this use case.

Agent Investigation

What we verified on our side, summarized:

• Conformance on KVM (Firecracker 1.10.1, guest Linux 6.12.109, OpenShell pinned at 71c3cd957abef062eb7f37010056717cd49f2ed3, which includes feat(isolation): implement the RFC 0012 sandbox architecture #2942). All of these pass in one run:
  • capabilities negotiation;
  • create → Ready (~0.5 s);
  • CLI create and exec (~1 s);
  • policy-mediated egress;
  • stop;
  • start with a new generation;
  • delete ×30 with no leaked paths;
  • a non-default prepared image (including a 4.5 GB worker image).
• Production: real agent jobs run in jailer mode behind our orchestrator: small and >4 MiB workspaces, concurrent sandboxes, and model access only through a policy-allowlisted proxy.
• Findings that affect any VM-based driver:
  • VM sandboxes get no PID limit. The guest init leaves the sandbox in the root cgroup, which has no pids.max, so the supervisor's check (PR Make sandbox child nproc limit configurable #1497) only warns. A guest-init fix was tested on a Firecracker guest: a child cgroup with pids.max, plus a read-only cgroup view for the sandbox. A fork flood fails at the limit and the guest survives. We will file this as a separate bug.
  • mke2fs -d still fails for single files above ~4 GiB when preparing images from a rootfs tar. It reproduces at our pin (bug: VM driver fails to create rootfs for images with files >2 GiB due to mke2fs -d bug #1762).
  • sandbox exec rejected piped stdin above 4 MiB at our pin, so we upload large stdin instead (bug: sandbox exec stdin limit exceeds gateway request limit #3685, Stream exec stdin without a TTY #3802).
  • The Failed to apply compute driver event … concurrent modification warning appears once per create. It is harmless, because the 60 s reconcile sweep re-applies the driver snapshot.
  • The VM driver's main-process spec must encode an empty command as []; the supervisor rejects null. This matters for any non-Rust driver.

Related issues and resources

OpenShell issues and PRs

• feat(firecracker): prototype host-supervised isolation #2934: maintainer Firecracker prototype (RFC 0012).
• feat(isolation): register an out-of-process Isolation Backend by endpoint #3913: out-of-process Isolation Backend registration.
• Proposal: Agent Substrate as a delegated isolation backend under RFC 12 #3362: Agent Substrate as a delegated backend.
• feat: establish the Isolation Backend interface #1737: Isolation Backend interface (closed); implemented in feat(isolation): implement the RFC 0012 sandbox architecture #2942.
• feat: external compute driver extension point #1907: external compute driver extension point.
• improve external compute-driver startup API and test coverage #2987: external driver startup API.
• docs(examples): document external compute driver authoring and extensibility #2418: external driver authoring docs.
• feat(driver-vm): snapshot/restore (suspend/resume) for idle sandboxes #1551: snapshot/restore (closed, not planned).
• Evaluate sandbox isolation options: gVisor runtime, Firecracker microVMs, or cluster-in-VM #4: isolation options evaluation (closed).
• bug: VM driver fails to create rootfs for images with files >2 GiB due to mke2fs -d bug #1762: mke2fs -d large files.
• bug: sandbox exec stdin limit exceeds gateway request limit #3685, Stream exec stdin without a TTY #3802: exec stdin limits.
• bug(vm-driver): sandbox stop loses files written in the last half minute #4185: VM stop data loss.
• refactor(compute): standardize lifecycle coordination across drivers #4115: lifecycle coordination.
• SandboxConfirmEvidence requires Linux mechanisms no non-stock guest kernel can provide #3361: conformance evidence for non-stock guest kernels.
• PR Make sandbox child nproc limit configurable #1497 and fix(sandbox): use cgroup pids controller instead of RLIMIT_NPROC on K8s #1325/fix(sandbox): use cgroup pids controller instead of RLIMIT_NPROC on K8s #1390: driver-owned PID limits.
• VM sandbox SSH session disconnects with broken pipe #2587: the pids.max warning seen in a VM guest log.

OpenShell code we built on

• rfc/0012-isolation-backend
• proto/compute_driver.proto
• crates/openshell-driver-vm, including scripts/openshell-vm-sandbox-init.sh and runtime/kernel/openshell.kconfig

Firecracker

• jailer
• rootfs and kernel setup
• vsock
• snapshot support
• guest kernel configs

Linux

• cgroup v2
• Landlock

Checklist

• I've reviewed existing issues and the published docs
• This is a design proposal, not a "please build this" request