Skip to content

AGENTS.md

Latest commit

 

History

History
216 lines (191 loc) · 16.4 KB

AGENTS.md

File metadata and controls

216 lines (191 loc) · 16.4 KB
Raw

AGENTS.md: driving (and testing) MCEC with an agent

MCEC 3.0 (the Model Context Environment Controller) gives an AI agent eyes, hands, and a safe front door on Windows (Windows computer use, in the sense Claude, Codex, and similar agents use the term). This file is the self-reinforcing guidance loop: the canonical guidance an agent needs, plus the recipe to dogfood it (drive MCEC through its own MCP server) so the guidance stays honest. Each time MCEC changes, re-run the dogfood and refine the guidance below and in AgentServer.Instructions.

The built-in guidance (single source of truth)

The connect-time guidance an MCP client shows the model is authored in src/Agent/AgentInstructions.md. It is embedded into the exe at build time and returned in the MCP initialize response (result.instructions) via AgentServer.Instructions, which loads the embedded file and collapses each blank-line-separated paragraph to one line.

Work the loop observe → target → act → observe.

  1. Target a window by window (title substring), process (name without .exe), className, or foreground:true. At least one is required; a call with no target fails by design.
  2. Observe: query dumps the UI Automation tree (controlType, name, automationId, bounds, state, value; bounded by maxDepth and maxNodes); capture returns a PNG (renders composited WinUI/WPF surfaces correctly). Check results before trusting them: a capture with errorCategory: capture-blank is a black/empty frame (minimized/cloaked/occluded/locked session); restore or foreground the window and retry, don't trust the image; a capture-fallback warning means PrintWindow was refused and the picture may be wrong; a query with truncated:true (a tree-truncated warning) hit the node cap; raise maxNodes or target a deeper window. warnings are non-fatal; errorCategory tells you how to recover. (Shape: docs/agent_control.md and agent-tool-result.schema.json.)
  3. Act: prefer invoke (by name/automationId/classname; action invoke|toggle|setvalue| setfocus|expand|collapse|select) over coordinate clicks. invoke fast-fails if the control isn't present (it does not wait), so find/wait-for the control first; an invoke that returns no-target means it hasn't appeared yet; wait-for it rather than retrying blindly. Use select for TabItem/ListItem/RadioButton. To drag (resize a window by its sizing border, move one by its title bar, or drag a slider/handle; no invoke for these), use the drag tool: give a from and to, each an element { by, value } or an absolute screen pixel { x, y }, plus optional path waypoints; the whole press-move-release is dispatched atomically (prefer it over hand-rolling mouse:lbd/mouse:mt/mouse:lbu, which can interleave with other input). To click a point invoke can't reach (a custom-drawn cell, a canvas/map coordinate, a bare pixel), use the click tool. Before firing an app's own keyboard shortcut at a specific surface (e.g. a MAUI GraphicsView), focus it first; keystrokes only reach the foreground window's focused control. send_command is the raw escape hatch for any other MCEC command (keystrokes, a single mouse action, launch). Re-query after acting; bounds have moved.
  4. Verify with another query/capture.

Compose creatively. Many tasks have no single dedicated tool; build them from primitives. Launch an app with the launch tool (preferred). Use invoke action:select for tabs/list/radios. Drag/resize/move with the drag tool; switch a tab by invoke select or click; record a window by passing its query'd bounds as the record region; wait on window state with the windows tool + a timeout and a condition (appears/disappears/foreground). A capable agent uses the full command set; reach for a raw send_command before concluding something can't be done.

There is exactly one copy: edit that file. It is the observe → target → act → observe playbook (targeting; observation with query/capture/record/displays; invoke, drag, click, focus and send_command; the result envelope; creative composition of primitives; the on-screen overlay; and the security gates). Nothing here to keep in sync.

Security (do not regress)

Three independent, off-by-default gates; see docs/agent_control.md: AgentCommandsEnabled (the observation opt-in, separate from actuation), per-command Enabled, and McpServerEnabled (HTTP floor, localhost-bound). Every agent action is logged with an AGENT-AUDIT: line. An agent that hits "agent commands are disabled" should tell the user, not retry.

Three safety features layer on top (see docs/safety-emergency-stop-and-provisioning.md):

  • Emergency stop: a global "dead man's switch" hotkey (default Ctrl+Alt+Shift+S) the operator can hit from any window to instantly halt a session: it latches the actuation gate (every tool call is refused with emergency-stopped until the operator re-arms), aborts in-flight actuation, and releases held input. It reacts to physical input only (injected keys are ignored via LLKHF_INJECTED), so the agent can never trip or defeat it. Do not weaken the injected-key filter or the latch.
  • Isolated session provisioning: agents must not enable/disable commands in the installed instance (a crash leaks enabled gates). Instead, provision-session (gated by the operator's AllowSessionProvisioning opt-in, set from File ▸ Settings ▸ Agent) hands the agent a disposable directory with its own agent-ready config; teardown is deleting the directory, and MCEC reaps orphaned session dirs on launch. The installed config is never touched. That same Agent tab lists the provisioned instances, lets the operator delete any an agent leaves behind, and (with the opt-in ticked) has a Provision new… button that mints one and shows the handoff. So provisioning can bootstrap itself (#296): the installed mcec.exe --mcp serves the bootstrap only — just provision-session / end-session, every other tool refused with bootstrap-only — so a fresh install's agent has a first hop, while the full observe/act surface is still never served from Program Files. Keep the bootstrap restriction (Program.ProvisioningBootstrapOnly) and the meta-tools' own gates intact. The operator's Provision new… handoff dialog shows a two-step handoff: MCP client setup plus a ready-made agent briefing prompt (ProvisionedInstanceDialog.BuildAgentPrompt); the prompt points at the connect-time instructions rather than duplicating the playbook, keep it that way.
  • Command-access consent (#307): a command-disabled refusal is recovered by the request-command-access meta-tool, never by editing config files. It shows the OPERATOR a consent dialog (CommandAccessConsentDialog) with three choices: allow these commands, allow these plus any later requests, or deny (sticky per instance; an emergency stop dismissing the prompt is NOT a deny and records nothing). Grants are in-memory only: the loaded table's Enabled flags, never persisted; CommandInvoker.Save serializes consent-granted commands as disabled (AgentConsent.WasGrantedByConsent) so the Commands window cannot quietly write a grant into mcec.commands. The dialog is protected from the agent by three layers: the call holds AgentRuntime.InputGate while the prompt is up; dispatch freezes everything except the ToolDescriptor.ServedDuringConsent observation tools with consent-pending (default false, so a new tool stays frozen; the freeze runs BEFORE the meta-tool branches, so provision-session cannot mint a second, unfrozen instance mid-prompt); and the dialog registers itself with WindowResolver as never-a-target. Do not weaken any of the three, do not persist grants, and do not replace the dialog with MCP elicitation (that routes consent through the agent's own client; the party being constrained). Grants land on the key CommandInvoker.ResolveGateKey resolves (the entry Enqueue actually gates on); route any new name-resolution through that one resolver. To spare the agent discovering the gated set one command-disabled at a time (#324), the initialize result carries a commandAccess map (enabledTools/gatedTools/enabledRawCommands) derived from ToolCatalog.CommandAccessDefaults (the ProvisionedByDefault flags — single source of truth, so it never drifts as defaults change), and session-status stamps the LIVE table via AgentToolExecutor.LiveCommandAccess; keep both derived, never a hand-typed list. enabledRawCommands is an explicit list, not a canary boolean, so a partial raw grant (only chars:) can't read as the whole raw send_command surface being open (#340 CR).

Dogfood: test MCEC using MCEC (mcec drives mcec)

This is the proposal's success metric ("an MCP client mounts MCEC and completes a multi-step GUI task"), encoded as a real, opt-in test: MCEC drives MCEC end-to-end with genuine keyboard + mouse input; no StartProcess launch crutch.

It lives in tests/MCEControl.xUnit/Integration/AgentDesktopE2ETests.cs. Because it drives the real desktop (global keystrokes, mouse, launching apps), it is skipped unless MCEC_DESKTOP_E2E=1; dotnet test/CI never touch your desktop. Run it deliberately on an interactive session:

dotnet build src/MCEControl.csproj -c Debug
$env:MCEC_DESKTOP_E2E=1 ; dotnet test --filter Category=DesktopE2E

What it does, all via the mcec.exe --mcp server's send_command/query/capture tools:

  1. Launch via the Win+R Run dialog (keyboard): send_command winr (Win+R) → chars:<path to mcec.exe> → enter. This starts a second MCEC purely through keyboard input.
  2. Open Help ▸ About (mouse + keyboard): query the new window's UIA tree to get the Help menu's bounding rect → mouse:mt,<x>,<y> + mouse:lbc to mouse-click it → key_a to press A and open About. (Pixel coords from UIA are normalized to InputSimulator's 0..65535 space using the physical primary-display size.)
  3. Verify: capture {window:"About"} returns a PNG image block; query {window:"About"} returns the dialog's UIA tree (Window:About, the version text, License Agreement, Button:OK).
  4. Graceful shutdown: key_esc dismisses About (its OK button is the CancelButton) → alt_f opens the File menu → key_x chooses E&xit, so the controlled instance runs its normal ShutDown() (saves settings, closes) rather than being killed. The test asserts the window is gone.

The actuation commands it relies on are enabled in a temporary mcec.commands it writes next to the exe (winr, chars:, enter, mouse:, key_a, key_esc, alt_f, key_x) and removed afterward.

Operational tips (learned the hard way)

  • Give the controlled copy ActAsServer=false; otherwise its socket-server bind raises a Windows Firewall prompt that steals foreground and eats the keystrokes.
  • A freshly launched copy must be the foreground window for menu keystrokes to land; injected Win+R is reliable, but let the Run dialog focus before typing the path.
  • MCEC is a WinExe, so a console pipeline (Get-Content | exe) won't reliably capture its stdout; spawn it with explicit inherited pipe handles (the test uses Process with RedirectStandardInput/Output).

Bugs this dogfood surfaced (now fixed)

  • SetStatus set NotifyIcon.Text (a 127-char cap) to the full informational version → crashed GUI startup on long prerelease version strings. Now capped.
  • .commands loading called new Version(informationalVersion) (throws on x.y.z-branch+sha) and popped MessageBox prompts that would hang the headless --mcp process. Now uses System.Version.TryParse and suppresses load-path dialogs when AgentRuntime.Headless.

Dogfood: WinPrint hero GIF

The worked examples (the mcec hero GIF, this WinPrint one, and prompt demos like Paint → smiley → email) are cataloged for humans and agents in docs/examples.md, with a template for adding more.

MCEC driving installed WinPrint is a marquee dogfooding example, but the WinPrint hero GIF (hero-gui-win.gif) is owned and produced in the winprint repo, agent-driven via the generic scripts/Generate-HeroGif.ps1 controller (no producer script lives here). Follow winprint's authoritative recipe docs/hero-gif-win.md; it records a clean, window-only, overlay-off hero. mcec-side conveniences that recipe still hand-configures (a bootstrap -NoOverlay switch, default-enabling windows/right/down/enter/run) are tracked as controller-bootstrap follow-ups.

Working in this repo

  • User docs vs contributor docs. docs/ is published to GitHub Pages (operator guides: install, configuration, agent control, safety, remote control, examples). dev/ holds contributor and maintainer material only (CI, releasing, code signing, agent-server architecture notes, evidence bundles, doc-image regeneration recipes); it is not part of the site. Release instructions: dev/RELEASING.md (cut from main, never develop).
  • No em dashes (—) in prose, anywhere in this repo. Use parentheses or semicolons for an aside; for a bullet-point heading (* [Link](url) — description) use a plain hyphen (-) or a colon (:) instead. This applies to Markdown docs and code comments alike.
  • Agent-facing guidance is part of "Done": not optional, not "later." Any change to how an agent observes/targets/acts (a new tool, arg, failure mode, warning/error category, or driving technique) MUST update the connect-time playbook in src/Agent/AgentInstructions.md (the single source of truth, embedded into the exe and served as AgentServer.Instructions) in the same change. Updating a per-tool Tool(...) description is not a substitute (those describe args); the instructions teach recovery and strategy. Read the trigger by principle, not keyword: it fires on feature work, not just dogfooding. Treat it like updating tests.
  • Doc UI images are dogfood artifacts. Every PNG/GIF under docs/ that shows running MCEC (Settings tabs, the provision handoff, Commands window, hero.gif) must be regenerated with MCEC's own capture/record tools, following dev/doc-images.md and dev/hero-gif.md. Maintainer-only recipes (e.g. dev/paint-smiley-email.md) live under dev/ and are not site assets. No DrawToBitmap export tests or hand-taken screenshots for committed assets.
  • Build is strict: Nullable=enable, TreatWarningsAsErrors=true, and house analyzers MCEC0001 (one top-level type per file) / MCEC0002 (no nested types). New code must be warning-clean.
  • Agent subsystem lives in src/Agent/ + the MCP server in src/Services/ (AgentServer.cs is a thin static facade over McpStdioTransport/McpHttpTransport/JsonRpcDispatcher/ AgentToolExecutor); commands plug into the existing Command/CommandInvoker pattern. Dev notes: dev/agent-server-architecture.md.
  • Tests: dotnet test tests/MCEControl.xUnit/MCEControl.xUnit.csproj. Before opening a PR, run the full suite — not a --filter on only the tests you touched. Filtered runs miss regressions in other fixtures (e.g. a SessionProvisioner guard can break EndSessionTokenTests while SessionProvisionerTests still pass). When you change provisioning or anything that affects session copies, grep for SessionProvisioner.BinariesDir and update every fake-install fixture together (use ProvisionTestFixtures.SeedMinimalInstall in tests/MCEControl.xUnit/Helpers/ProvisionTestFixtures.cs). Worker/lifecycle helpers extracted from a service (e.g. #317) belong in their own source files per MCEC0002.