Skip to content

About

Windows sandbox running in Docker with computer-use support

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

25 Commits

Folders and files

Repository files navigation

Windows Computer Use Sandbox

中文版:README.zh-CN.md

A model-agnostic Computer Use environment based on dockur/windows (Docker + QEMU/KVM running Windows 11 LTSC): any agent gets screenshot, mouse, keyboard, and PowerShell capabilities on a Windows virtual machine through a unified HTTP / MCP interface.

Architecture

Any agent (model-agnostic)                 Human
      │                                    │
      │ HTTP / MCP                         │ Browser / RDP client
      ▼                                    ▼
┌───────────────┐                   ┌───────────────┐
│  VM Gateway   │                   │ :8006 web     │
│  FastAPI :8080│                   │ viewer / :3389│
└──────┬────────┘                   └───────────────┘
       │ HTTP (guest :5000)
┌──────▼─────────────────────────────────────────────┐
│ Docker: dockurr/windows (QEMU/KVM)                 │
│  Windows 11 LTSC                                   │
│   └ guest-agent :5000 (screenshot/input/PowerShell)│
└────────────────────────────────────────────────────┘
  • guest-agent (inside the VM): the hands and eyes — MSS screenshots, SendInput input synthesis (Chinese input supported), PowerShell execution
  • gateway (host): model adaptation — screenshot scaling (max 1280×720), coordinate mapping, unified action space, Bearer auth, audit log (logs/actions.jsonl)

Agents only interact with the gateway, and all coordinates are in the scaled screenshot coordinate space.

Prerequisites

  • A Linux host with hardware virtualization/KVM enabled and read/write access to /dev/kvm and /dev/net/tun.
  • Docker with Compose v2.
  • Python 3.10+ with venv support.

Check the devices and tool versions:

ls -l /dev/kvm /dev/net/tun
docker compose version
python3 --version

Quick Start

# 1. Start the VM (first boot downloads and unattended-installs Windows, ~20–40 min)
docker compose up -d

# 2. Install host dependencies
python3 -m venv .venv
.venv/bin/pip install -r gateway/requirements.txt

# 3. Create the local configuration (edit .env if needed)
cp .env.example .env

# 4. Start the gateway (background)
mkdir -p logs
nohup .venv/bin/uvicorn --app-dir gateway app:app \
  --host 127.0.0.1 --port 8080 --env-file .env > logs/gateway.log 2>&1 &

# 5. End-to-end smoke test (once Windows installation has finished)
.venv/bin/python scripts/smoke_test.py   # expected output: SMOKE TEST PASSED

The gateway, MCP server, smoke test, and demo load the project .env; variables already set in the process environment take precedence.

Human observation channel: open http://localhost:8006 in a browser, or RDP to localhost:3389 (user Docker / password admin).

Security

This sandbox gives agents arbitrary PowerShell, mouse, and keyboard control inside the VM. Use it only with agents and tasks you trust to have that control.

  • Compose binds the web viewer, RDP, and guest-agent ports to 127.0.0.1. Do not expose these ports or the gateway to untrusted networks; use an SSH tunnel for remote access.
  • The documented RDP credentials and gateway token are local-development defaults. Change GATEWAY_TOKEN in .env; change the RDP credentials as well if remote access is required.
  • The audit log stores typed text (up to 200 characters) and complete shell commands in plaintext. Avoid entering secrets and restrict access to logs/.
  • Unattended setup disables automatic Windows updates. Install updates periodically and rebuild the golden image.

HTTP API (gateway :8080)

All requests require the header Authorization: Bearer <GATEWAY_TOKEN> (default computeruse-dev-token, see .env).

Method Path Description
GET /v1/status VM and guest-agent health, screen resolution
POST /v1/screenshot Screenshot (scaled to max 1280×720, base64 PNG + dimension metadata)
POST /v1/action Unified actions (see table below); coordinates are in screenshot space
POST /v1/shell {"command": "...", "timeout": 30} → PowerShell; returns stdout/stderr/exit_code

Action space (/v1/action)

action Parameters Description
screenshot — Same as /v1/screenshot
left_click / right_click / middle_click / double_click / triple_click coordinate: [x, y] Click
mouse_move coordinate Move
left_click_drag start_coordinate, coordinate Drag
scroll coordinate, scroll_direction (up/down/left/right), scroll_amount (default 3) Scroll
type text Type text (Unicode, Chinese supported)
key key: "ctrl+c" or keys: ["CTRL","C"] Key combo (accepts both Anthropic and OpenAI styles)
wait duration (seconds, ≤30) Wait

Out-of-range coordinates are automatically clamped into the screen and logged as a warning. An unreachable guest returns 502 for screenshot, action, and shell requests. GET /v1/status is the exception: it returns 200 with guest: "unreachable" so callers can inspect the reported state.

curl examples

TOKEN="Authorization: Bearer computeruse-dev-token"

curl -s -H "$TOKEN" http://127.0.0.1:8080/v1/status
curl -s -X POST -H "$TOKEN" http://127.0.0.1:8080/v1/screenshot
curl -s -X POST -H "$TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"left_click","coordinate":[640,360]}' http://127.0.0.1:8080/v1/action
curl -s -X POST -H "$TOKEN" -H "Content-Type: application/json" \
  -d '{"action":"type","text":"你好"}' http://127.0.0.1:8080/v1/action
curl -s -X POST -H "$TOKEN" -H "Content-Type: application/json" \
  -d '{"command":"hostname"}' http://127.0.0.1:8080/v1/shell

MCP Integration

gateway/mcp_server.py provides 4 tools: computer_status / computer_screenshot / computer_action / computer_shell.

opencode config (opencode.json in the project root):

{
  "mcp": {
    "windows-computer-use": {
      "type": "local",
      "command": [
        "/absolute/path/.venv/bin/python",
        "/absolute/path/gateway/mcp_server.py"
      ],
      "enabled": true
    }
  }
}

Note: you must use the virtualenv's Python (the system Python has no dependencies installed and will fail with No module named 'httpx2').

Operations

./scripts/make_golden.sh   # build a golden image (stops the VM, sparse-copies data.img)
./scripts/reset.sh         # reset the VM to the golden image in seconds
.venv/bin/python scripts/smoke_test.py   # 9-step end-to-end smoke test
.venv/bin/python demo/demo_notepad.py   # scripted demo (opens notepad, types, screenshots)
  • Audit log: logs/actions.jsonl (one JSON line per action; may contain sensitive input)
  • guest-agent log: C:\ComputerUse\agent.log inside the VM; install log: C:\ComputerUse\install.log
  • Host ↔ VM file exchange: ./shared on the host ↔ \\host.lan\Data inside the VM (see limitations below)

Known Limitations

  • Exclusivity: only one "driver" at a time. Input synthesis acts on the whole desktop session; a human (RDP/web viewer) and an agent operating at the same time will interfere with each other (stolen focus, lost characters). Do not operate the VM while automation or smoke tests are running.
  • Use the UNC path \\host.lan\Data, not the Z: drive: dockur's Z: mapping is unreliable across reboots and elevated contexts (shows Unavailable, directory-cache poisoning); the UNC path works in every session.
  • Processes launched via the Run dialog (Win+R) are elevated; normal operations are unaffected.
  • The guest agent is only available while Windows is running normally (no control during installation or a BSOD — extend with a VNC/QMP channel if you need that).
  • Windows is a Microsoft evaluation build (90-day license watermark); bring your own license for compliant use.

Development and Testing

# Unit tests (all run on the Linux host; the guest agent's Win32 parts are platform-guarded)
.venv/bin/python -m pytest oem/tests/ gateway/tests/ -v

# End-to-end smoke test (requires the VM and gateway to be up)
.venv/bin/python scripts/smoke_test.py

Project Structure

compose.yml             # dockurr/windows service definition (Win11 LTSC, 8G/4C/128G)
oem/
  install.bat           # unattended-install deployment of the guest agent (Python+mss, autostart, firewall)
  guest_agent.py        # HTTP service :5000 inside the VM
  tests/                # guest-agent pure-logic unit tests
gateway/
  app.py                # FastAPI gateway :8080 (auth/scaling/coordinate mapping/audit)
  coords.py             # viewport ↔ guest coordinate mapping
  guest_client.py       # httpx2 client for the guest agent
  mcp_server.py         # MCP stdio server
  tests/
scripts/                # smoke_test.py / make_golden.sh / reset.sh
shared/                 # ↔ VM \\host.lan\Data
demo/                   # notepad demo script
docs/superpowers/       # design docs and implementation plans

About

Windows sandbox running in Docker with computer-use support

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages