fps-bench — first-person movement benchmark for cua-driver

A benchmark that measures how reliably cua-driver delivers mouse and keyboard input to a three.js first-person game (mouse look, WASD, Space) on a Linux desktop: reach the goal at the far end of an L-shaped platform. A minimal agent acts only through cua-driver call …, and a pi-autoresearch loop patches the vendored cua-driver source on parallel CUA Fleet sandboxes to raise the score.

Brief: docs/BRIEF.md. Design: docs/plans/2026-08-29-fps-cua-driver-bench-design.md. Results: docs/results/2026-08-29-mouse-look-baseline-vs-exp4.md.

Held-key game — left: cua-driver main (score 0.0), right: with the kept Exp 4 patch (score 1.0). The right side turns at the corner and reaches the goal in ~40 s; the left walks into the corner and never turns.

Full-length, full-quality recordings: held-keys-side-by-side.webm · mouse-look-side-by-side.webm (MP4s alongside in results/videos/; the GIF above is the full run at 6 fps — the right side freezes on its final frame once it reaches the goal while the left runs to completion).

Setup

uv sync                                   # cua-bench 0.2.11, cua-sandbox (wheels.cua.ai), anthropic
.venv/bin/python scripts/game_check.py    # headless smoke test of the game (no desktop needed)

Fleet credentials come from the macOS Keychain entry cua-sandbox-fleet-api (same one pi-cua uses) or CUA_CLIENT_ID / CUA_CLIENT_SECRET.

Task

tasks/fps_lshape/ — main.py + gui/index.html (three.js r147 + PointerLockControls, vendored). Controls: mouse look, W/A/S/D move only while held (6 u/s; a press-and-release tap goes nowhere), Space jump. evaluate() → [reached, progress].

.venv/bin/cb interact tasks/fps_lshape --oracle --no-wait      # oracle solves it via window.__press
.venv/bin/cb run task tasks/fps_lshape --wait --agent-import-path fps_bench.agent:CuaDriverAgent

Local runs use docker image public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04:docker-latest (the cua-sandbox local-docker default on trycua/cua main) — but that image lacks bench_ui, which launch_window() needs, so for local runs build the arm64 variant of our image and point at it:

PLATFORM=linux/arm64 PUSH=0 image/build.sh          # -> fps-bench-cua-driver:local
FPS_BENCH_IMAGE=fps-bench-cua-driver:local .venv/bin/python -m fps_bench.runner --local --episodes 3

Agent

fps_bench/agent.py — CuaDriverAgent. Reads window.__state (privileged perception); acts only through cua-driver inside the environment: move_cursor to turn (relative mouse motion drives PointerLockControls), press_key w to walk, a click first to focus/lock. Re-plans after every action. Per-episode record includes presses vs keydowns seen (delivery_ratio) and mouse pixels sent vs seen (mouse_ratio).

Fleet image

image/Dockerfile: FROM …cua-ubuntu-24.04:docker-latest + bench_ui/pywebview + Rust 1.97.1 + sparse clone of trycua/cua (libs/cua-driver) at CUA_REF, crates pre-fetched. cua-driver is compiled natively at container boot by supervisord (image/prebuild.sh) and installed to /usr/local/bin/cua-driver.

image/build.sh        # buildx linux/amd64 → 296062593712.dkr.ecr.us-west-2.amazonaws.com/cua-gymdriver-dev:cua-driver-bench-<date>-<ref>

Fleet's pool admission only allows a fixed set of image repositories, which is why the tag lands in cua-gymdriver-dev. Pools use runtime = gvisor (plain container). Current tag: cua-driver-bench-20260830-hold (vendored driver: press_key hold_ms + Exp 4 click patch) (also runs a cua-driver serve daemon, which cua-driver call needs on this build) — the base image's supervisord.conf has no [include], so earlier tags never ran the boot-time build; this one registers it directly (build + install ≈ 60 s on a gVisor sandbox).

Autoresearch with pi-cua + pi-autoresearch (primary loop)

The loop is driven by pi with two extensions: pi-cua executes pi's tools on a CUA Fleet Linux sandbox whose workspace is synced from this repo's origin, and pi-autoresearch runs the try → measure → keep/revert loop (init_experiment / run_experiment / log_experiment).

• cua-driver/ — vendored cua-driver source (trycua/cua libs/cua-driver, ref in cua-driver/UPSTREAM_REF); the files pi edits.
• .auto/prompt.md, .auto/measure.sh, .auto/config.json — the pi-autoresearch session. measure.sh bootstraps guest deps (bench/bootstrap_guest.sh), rebuilds cua-driver, runs bench/run_in_sandbox.py on the sandbox's X display and prints METRIC score=….
• scripts/pi_sandbox.py — headless wrapper over the pi-cua backend (create/list/delete/bind).
• scripts/pi_autoresearch.sh <sandbox> [iterations] — pins a fresh pi session to a sandbox and runs the loop non-interactively; run one per sandbox for parallel hill-climbs.

The origin is private, and pi-cua clones the workspace on the sandbox, so scripts/guest_bootstrap.py (run automatically by the launcher) injects a GitHub token at runtime as a git credential-store entry for user cua — per sandbox secrets, never into the image. It uses GH_SANDBOX_TOKEN (prefer a fine-grained read-only PAT) or falls back to gh auth token.

pi install npm:pi-autoresearch                       # once (pi-cua already installed)
.venv/bin/python scripts/pi_sandbox.py create fps-a --wait
.venv/bin/python scripts/pi_sandbox.py create fps-b --wait
scripts/pi_autoresearch.sh fps-a 20 & scripts/pi_autoresearch.sh fps-b 20 & wait

Interactive alternative: pi → /sandbox fps-a → /autoresearch raise cua-driver key delivery score.

Autoresearch with the standalone Fleet loop (alternative)

.venv/bin/python scripts/fleet_smoke.py --image <image> --pool fps-bench-smoke --bench   # 1 sandbox, 1 episode
FPS_BENCH_FLEET_IMAGE=<image> .venv/bin/python -m fps_bench.autoresearch --workers 3 --iterations 5 --episodes 3

Each worker: Claude (claude-opus-5) proposes a diff → claim sandbox → git apply + cargo build --release -p cua-driver → benchmark → append to results/experiments.jsonl → release claim. Best patch is kept in results/best.diff.