中文版:README.zh-CN.md
A model-agnostic Computer Use environment based on dockur/windows (Docker + QEMU/KVM running Windows 11 LTSC): any agent gets screenshot, mouse, keyboard, and PowerShell capabilities on a Windows virtual machine through a unified HTTP / MCP interface.
Any agent (model-agnostic) Human
│ │
│ HTTP / MCP │ Browser / RDP client
▼ ▼
┌───────────────┐ ┌───────────────┐
│ VM Gateway │ │ :8006 web │
│ FastAPI :8080│ │ viewer / :3389│
└──────┬────────┘ └───────────────┘
│ HTTP (guest :5000)
┌──────▼─────────────────────────────────────────────┐
│ Docker: dockurr/windows (QEMU/KVM) │
│ Windows 11 LTSC │
│ └ guest-agent :5000 (screenshot/input/PowerShell)│
└────────────────────────────────────────────────────┘
- guest-agent (inside the VM): the hands and eyes — MSS screenshots, SendInput input synthesis (Chinese input supported), PowerShell execution
- gateway (host): model adaptation — screenshot scaling (max 1280×720), coordinate mapping, unified action space, Bearer auth, audit log (
logs/actions.jsonl)
Agents only interact with the gateway, and all coordinates are in the scaled screenshot coordinate space.
- A Linux host with hardware virtualization/KVM enabled and read/write access to
/dev/kvmand/dev/net/tun. - Docker with Compose v2.
- Python 3.10+ with
venvsupport.
Check the devices and tool versions:
ls -l /dev/kvm /dev/net/tun
docker compose version
python3 --version# 1. Start the VM (first boot downloads and unattended-installs Windows, ~20–40 min)
docker compose up -d
# 2. Install host dependencies
python3 -m venv .venv
.venv/bin/pip install -r gateway/requirements.txt
# 3. Create the local configuration (edit .env if needed)
cp .env.example .env
# 4. Start the gateway (background)
mkdir -p logs
nohup .venv/bin/uvicorn --app-dir gateway app:app \
--host 127.0.0.1 --port 8080 --env-file .env > logs/gateway.log 2>&1 &
# 5. End-to-end smoke test (once Windows installation has finished)
.venv/bin/python scripts/smoke_test.py # expected output: SMOKE TEST PASSEDThe gateway, MCP server, smoke test, and demo load the project .env; variables already set in the process environment take precedence.
Human observation channel: open http://localhost:8006 in a browser, or RDP to localhost:3389 (user Docker / password admin).
This sandbox gives agents arbitrary PowerShell, mouse, and keyboard control inside the VM. Use it only with agents and tasks you trust to have that control.
- Compose binds the web viewer, RDP, and guest-agent ports to
127.0.0.1. Do not expose these ports or the gateway to untrusted networks; use an SSH tunnel for remote access. - The documented RDP credentials and gateway token are local-development defaults. Change
GATEWAY_TOKENin.env; change the RDP credentials as well if remote access is required. - The audit log stores typed text (up to 200 characters) and complete shell commands in plaintext. Avoid entering secrets and restrict access to
logs/. - Unattended setup disables automatic Windows updates. Install updates periodically and rebuild the golden image.
All requests require the header Authorization: Bearer <GATEWAY_TOKEN> (default computeruse-dev-token, see .env).
| Method | Path | Description |
|---|---|---|
| GET | /v1/status |
VM and guest-agent health, screen resolution |
| POST | /v1/screenshot |
Screenshot (scaled to max 1280×720, base64 PNG + dimension metadata) |
| POST | /v1/action |
Unified actions (see table below); coordinates are in screenshot space |
| POST | /v1/shell |
{"command": "...", "timeout": 30} → PowerShell; returns stdout/stderr/exit_code |
| action | Parameters | Description |
|---|---|---|
screenshot |
— | Same as /v1/screenshot |
left_click / right_click / middle_click / double_click / triple_click |
coordinate: [x, y] |
Click |
mouse_move |
coordinate |
Move |
left_click_drag |
start_coordinate, coordinate |
Drag |
scroll |
coordinate, scroll_direction (up/down/left/right), scroll_amount (default 3) |
Scroll |
type |
text |
Type text (Unicode, Chinese supported) |
key |
key: "ctrl+c" or keys: ["CTRL","C"] |
Key combo (accepts both Anthropic and OpenAI styles) |
wait |
duration (seconds, ≤30) |
Wait |
Out-of-range coordinates are automatically clamped into the screen and logged as a warning. An unreachable guest returns 502 for screenshot, action, and shell requests. GET /v1/status is the exception: it returns 200 with guest: "unreachable" so callers can inspect the reported state.
TOKEN="Authorization: Bearer computeruse-dev-token"
curl -s -H "$TOKEN" http://127.0.0.1:8080/v1/status
curl -s -X POST -H "$TOKEN" http://127.0.0.1:8080/v1/screenshot
curl -s -X POST -H "$TOKEN" -H "Content-Type: application/json" \
-d '{"action":"left_click","coordinate":[640,360]}' http://127.0.0.1:8080/v1/action
curl -s -X POST -H "$TOKEN" -H "Content-Type: application/json" \
-d '{"action":"type","text":"你好"}' http://127.0.0.1:8080/v1/action
curl -s -X POST -H "$TOKEN" -H "Content-Type: application/json" \
-d '{"command":"hostname"}' http://127.0.0.1:8080/v1/shellgateway/mcp_server.py provides 4 tools: computer_status / computer_screenshot / computer_action / computer_shell.
opencode config (opencode.json in the project root):
{
"mcp": {
"windows-computer-use": {
"type": "local",
"command": [
"/absolute/path/.venv/bin/python",
"/absolute/path/gateway/mcp_server.py"
],
"enabled": true
}
}
}Note: you must use the virtualenv's Python (the system Python has no dependencies installed and will fail with
No module named 'httpx2').
./scripts/make_golden.sh # build a golden image (stops the VM, sparse-copies data.img)
./scripts/reset.sh # reset the VM to the golden image in seconds
.venv/bin/python scripts/smoke_test.py # 9-step end-to-end smoke test
.venv/bin/python demo/demo_notepad.py # scripted demo (opens notepad, types, screenshots)- Audit log:
logs/actions.jsonl(one JSON line per action; may contain sensitive input) - guest-agent log:
C:\ComputerUse\agent.loginside the VM; install log:C:\ComputerUse\install.log - Host ↔ VM file exchange:
./sharedon the host ↔\\host.lan\Datainside the VM (see limitations below)
- Exclusivity: only one "driver" at a time. Input synthesis acts on the whole desktop session; a human (RDP/web viewer) and an agent operating at the same time will interfere with each other (stolen focus, lost characters). Do not operate the VM while automation or smoke tests are running.
- Use the UNC path
\\host.lan\Data, not the Z: drive: dockur's Z: mapping is unreliable across reboots and elevated contexts (showsUnavailable, directory-cache poisoning); the UNC path works in every session. - Processes launched via the Run dialog (Win+R) are elevated; normal operations are unaffected.
- The guest agent is only available while Windows is running normally (no control during installation or a BSOD — extend with a VNC/QMP channel if you need that).
- Windows is a Microsoft evaluation build (90-day license watermark); bring your own license for compliant use.
# Unit tests (all run on the Linux host; the guest agent's Win32 parts are platform-guarded)
.venv/bin/python -m pytest oem/tests/ gateway/tests/ -v
# End-to-end smoke test (requires the VM and gateway to be up)
.venv/bin/python scripts/smoke_test.pycompose.yml # dockurr/windows service definition (Win11 LTSC, 8G/4C/128G)
oem/
install.bat # unattended-install deployment of the guest agent (Python+mss, autostart, firewall)
guest_agent.py # HTTP service :5000 inside the VM
tests/ # guest-agent pure-logic unit tests
gateway/
app.py # FastAPI gateway :8080 (auth/scaling/coordinate mapping/audit)
coords.py # viewport ↔ guest coordinate mapping
guest_client.py # httpx2 client for the guest agent
mcp_server.py # MCP stdio server
tests/
scripts/ # smoke_test.py / make_golden.sh / reset.sh
shared/ # ↔ VM \\host.lan\Data
demo/ # notepad demo script
docs/superpowers/ # design docs and implementation plans