diff --git a/.github/skills/cracktro/content-rules.md b/.github/skills/cracktro/content-rules.md index f8f97bc4232..563b48ca099 100644 --- a/.github/skills/cracktro/content-rules.md +++ b/.github/skills/cracktro/content-rules.md @@ -37,6 +37,7 @@ Enforced by `docs/tests/talk.spec.ts`. Load before adding public text, data, ava - Exception: the For PMs slide (`pm-` classes) shows two clickable processes (feature intake, launch readiness) whose workflow names, inputs, outputs and ideas come from the public, MIT-licensed `chrizbo/agentics-beyond-code` repository, credited by a linked repo slug in the kicker and in the caption and notes (no personal names or avatars); the test's repo-reference regex only blocks `github/` and `githubnext/` refs. The issue and report contents are synthetic; its notes and caption say so. - Exception: the Scan · Fix · Learn slide (`sfm-` panel) shows measured counts from the public .NET runtime repository's `ci-failure-scan`, `ci-failure-fix` and `ci-failure-scan-feedback` workflows (window 5 May to 14 Nov 2026, 193 days, labelled on the panel: 305 `[ci-scan]` issues, 270 closed; 22 `[ci-fix]` PRs, 5 merged; 12 merged `[ci-scan-feedback]` prompt-edit PRs; 882 runs; issues per month). The loop below it is illustrative; its notes and caption say so. - Exception: the data slides before the finale. "First commit to GA" (`hist-` classes), shows gh-aw's measured public history (git on main plus GitHub search counts, 12 Aug 2025 to GA on 28 Oct 2026); its speaker notes give the sources and say account attribution is not proof of AI use. "How PRs land" (`land-` classes), shows measured time-to-merge and lines-changed distributions of merged PRs, split by the same origins; issues opened under a person's account may still have been drafted in the Copilot app or CLI, which GitHub does not record. "How workflows evolve" (`evo-` classes), shows measured first-parent commits on main that changed `.github/workflows/**/*.md` (monthly by author type, live workflow count, lines-changed histogram and statistical tests: Spearman, Mann–Whitney, Gini), 12 Aug 2025 to GA; its notes give the method. +- Exception: the CLI vs MCP slide (`cvm-` classes) has one measured caption (tool schemas only) from gh-aw's own runs of one workflow that switched from the GitHub MCP server to `gh-proxy` plus `cli-proxy`, same model, 15 runs per side, 23 Aug to 3 Sep 2026, snapshot 17 Nov 2026: median first request 16,690 → 12,766 tokens, whole-run cost about the same; its notes add a confounded 21-workflow bulk switch without a slide figure. The code scanning alerts, JSON response, commands, counts, files and paths in the flow are synthetic and its notes say so. Refresh with `refresh/cli-vs-mcp.md`. - The only URLs allowed in `deck.js` are `https://github.github.com/gh-aw/introduction/overview/`, the For PMs source link `https://github.com/chrizbo/agentics-beyond-code` and the repo-widget endpoint `https://api.github.com/repositories/`. The widget (`repoWidget(id, name, stars, cls)`) addresses repos by numeric id, fetches only on `https:` pages (never on localhost, `file://` or in tests), once per repo, and keeps the hard-coded snapshot count on any failure; it shows only the repo octicon, never an owner avatar. Index & search (`ix-rw`) shows one for the open-source qmd repo labelled `qmd`, never its owner handle (snapshot default 31k on 17 Nov 2026). - Speaker notes of every slide marked `illustrative` in `slides.md` must match `/illustrative|synthetic|simulated/i`. - Never write slide numbers or slide counts outside `slides.md`; refer to slides by heading and CSS prefix. diff --git a/.github/skills/cracktro/data-refresh.md b/.github/skills/cracktro/data-refresh.md index a2cfb7e4983..2f4e61cf32d 100644 --- a/.github/skills/cracktro/data-refresh.md +++ b/.github/skills/cracktro/data-refresh.md @@ -45,6 +45,7 @@ Every slide that shows measured data has a refresh mini skill. Load only the one | Findings that land | `refresh/findings-that-land.md` | | Personas · Probe · Patch | `refresh/personas-probe-patch.md` | | Scan · Fix · Learn | `refresh/scan-fix-learn.md` | +| CLI vs MCP | `refresh/cli-vs-mcp.md` | | Skills in review | `refresh/skills-in-review.md` | | Ponytail Reviewer | `refresh/ponytail-reviewer.md` | diff --git a/.github/skills/cracktro/refresh/cli-vs-mcp.md b/.github/skills/cracktro/refresh/cli-vs-mcp.md new file mode 100644 index 00000000000..2bd138772fe --- /dev/null +++ b/.github/skills/cracktro/refresh/cli-vs-mcp.md @@ -0,0 +1,82 @@ +--- +name: cracktro-refresh-cli-vs-mcp +description: Recompute the measured data on the cracktro "CLI vs MCP" slide and update deck.js. +--- + +# Refresh "CLI vs MCP" + +Load `.github/skills/cracktro/SKILL.md` first. Never read `docs/public/cracktro/deck.js` whole; `grep -n`, then `view_range`. + +## Workflows mined + +Same-workflow before/after natural experiments: commits that flip `tools.github.mode` between `local` (GitHub MCP server) and `gh-proxy` (`gh` CLI). Each run's `usage` artifact is mined. + +| Workflow file | Repo | Role in the slide | Signal mined | +|---|---|---|---| +| `.github/workflows/ponytail-reviewer.md` | github/gh-aw | the two measured boxes and the caption | first-request prompt tokens, run totals; switched `local` → `gh-proxy` + `cli-proxy: true` in one commit, same model | +| 21 workflows in the bulk `gh-proxy` → `local` commit (`ai-moderator.md`, `tidy.md`, `go-fan.md`, ... listed in the miner's `cases[]`) | github/gh-aw | notes only (whole-run AI credits) | same metrics; the commit also changed the sandbox runtime | +| the 14 busiest workflows with no `.md` edit within 7 days of the bulk commit | github/gh-aw | notes only (drift control) | same metrics, before vs after the bulk commit | + +## 1. Slide + +- Marker: `// CLI vs MCP:` (`grep -n "// CLI vs MCP:" docs/public/cracktro/deck.js`). CSS prefix `cvm-`. Group `cost`. Row 66 in `slides.md` (Notes `illustrative`: the flow is synthetic). +- The flow (fat `list_code_scanning_alerts` response vs `--help` → `> alerts.json` → `jq | sort | uniq -c | head`) and its `IN CONTEXT` boxes are synthetic and qualitative: no measured numbers there. +- Data lives inside that block only: + 1. The `.cvm-cap` caption: `MEASURED, schemas only`, workflow count, runs per side, window, first-request medians (16.7k → 12.8k), whole-run verdict, snapshot. + 2. `notes:` string: repeats every number, the window, the snapshot, the bulk result and its confound. +- Elsewhere: the `content-rules.md` Exception bullet "the CLI vs MCP slide (`cvm-` classes)". + +## 2. Window & snapshot + +- Each side is bounded by the workflow's previous/next `.md` edit and capped at `DAYS` (7) from the switch; the `CAP` (15) successful runs nearest the switch are kept. A side needs `MIN` 3 runs. +- Baseline (snapshot 2026-11-17): ponytail-reviewer MCP side 2026-08-23 → 08-27, gh-proxy side 08-27 → 09-03; bulk commit 2026-09-05, sides 08-29 → 09-12. + +## 3. Sources + +- `gh run download -R github/gh-aw -n usage`: `agent/token_usage.jsonl`, one line per model request (`input_tokens`, `cache_read_tokens`, `cache_write_tokens`, `output_tokens`, `ai_credits_this_response`, `path`, `model`). Skip `detection/`. +- Surface of the MCP side: the compiled lock at the MCP-side commit. `GH_AW_MCP_CLI_SERVERS_LIST` naming `` `github` `` means cli-proxy mounted the GitHub MCP as a CLI (no schemas in context); otherwise its schemas were offered to the engine. Front matter alone misses `cli-proxy` set by imports. +- Tool count: `gh run download -n agent`, `mcp-logs/tools.json` (`servers.github.length`) of an MCP-side run. + +## 4. Queries + +Run from the repo root (needs full git history and `gh auth login`); downloads are cached in `CACHE`: + +```bash +RAW=$TMPDIR/cvm-raw.json node .github/skills/cracktro/refresh/cli-vs-mcp.mjs > $TMPDIR/cvm-out.json +node -e 'const o=require(process.env.TMPDIR+"/cvm-out.json");console.log(JSON.stringify(o.summary));const c=o.cases.find(c=>c.wf==="ponytail-reviewer");console.log(c.before,c.after,c.delta)' +``` + +- To add a switch, append its short SHA to `SWITCHES`; find candidates with `git log -G'mode: (gh-proxy|local)' --format='%h %cI %s' -- .github/workflows/*.md`. +- Medians are per workflow per side; `summary.*..medianPct` is the median of per-workflow % changes (MCP side → gh-proxy side), `lower` the count that fell. + +Baseline (snapshot 2026-11-17): + +| Group | Workflows · runs | First request | Prompt tokens | AI credits | +|---|---|---|---|---| +| ponytail-reviewer, schemas → gh-proxy + cli-proxy | 1 · 30 | 16,690 → 12,766 (−23.5%) | 182,639 → 190,929 (+4.5%) | 4.15 → 4.27 (+2.9%) | +| bulk commit, all cases | 21 · 250 | ~+0.4k on MCP side (median) | — | median −19.2%, 16/21 lower, range −78% … +123% | +| bulk, GitHub MCP behind cli-proxy | 13 · 154 | −1.3% | −22.1% | −6.4% (10/13) | +| drift controls | 14 · 212 | 0% | −3% | −1.7% (9/14) | + +## 5. Update steps + +1. Update the `.cvm-cap` caption (medians, window, runs per side, snapshot); the `ctx(...)` boxes stay qualitative. +2. Update the notes: every number, window, snapshot, the bulk range and confound. Keep `illustrative and synthetic` for the flow. +3. Update the `content-rules.md` Exception bullet. `slides.md` row 66 stays free of numbers and dates. +4. If a newer clean switch (one workflow, same model, no runtime change) exists, prefer it for the boxes and say which in the notes. + +## 6. Validation + +- `node --check docs/public/cracktro/deck.js` and the preview at `#66` (row 66 of `slides.md`): boxes fit their 536-px rects; caption stays inside the 1620-px viewBox. +- The ponytail case shows `mcpSurface: "schemas"` and equal `requests` medians on both sides; controls stay near 0%. +- No URLs, SHAs, run IDs or logins on the slide or in the notes. + +## 7. Pitfalls + +- Logs and artifacts expire after ~90 days: the baseline runs (Aug–Sep 2026) become unrecomputable around early December 2026. Keep the baseline table; refresh from newer switches instead. +- `/responses` (OpenAI, Codex, MAI) `input_tokens` already include cached tokens; Anthropic counts `cache_read`/`cache_write` separately. The miner's `inclusive()` handles both; never add cache to `/responses` input. +- Claude and Codex engines appear to defer MCP tool schemas: their local-mode first request was only ~0.4k larger than gh-proxy. Only engines that load schemas up front (Copilot CLI with tool search off) show the ~3.9k jump. Do not generalise the −23% beyond that. +- Whole-run tokens and credits are dominated by task variance (n = 3–15 per side). Do not put a whole-run percentage on the slide. +- The bulk commit also moved the sandbox to cloud-hypervisor and 18 files changed engine (excluded): it is confounded. +- Many successful runs have no `usage` artifact (agent skipped); `gh run list` caps at 1000, so the miner filters by `--created`. +- macOS `awk` has no `\s`; use `[ ]`. diff --git a/.github/skills/cracktro/refresh/cli-vs-mcp.mjs b/.github/skills/cracktro/refresh/cli-vs-mcp.mjs new file mode 100644 index 00000000000..5161811d9da --- /dev/null +++ b/.github/skills/cracktro/refresh/cli-vs-mcp.mjs @@ -0,0 +1,180 @@ +#!/usr/bin/env node +// Read-only: same-workflow before/after comparison of the GitHub MCP server (tools.github.mode: local) +// versus gh-proxy (`gh` CLI, no GitHub MCP tool schemas). Mines each run's `usage` artifact. +// Run from the repo root (needs git history and `gh auth login`): +// node .github/skills/cracktro/refresh/cli-vs-mcp.mjs > out.json (RAW=rows.json dumps per-run rows) +// Env: CAP runs per side nearest the switch (15), DAYS max window per side (7), CONTROLS (15), CACHE dir. +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +const sh = promisify(execFile); +const REPO = process.env.REPO || "github/gh-aw"; +const CAP = +(process.env.CAP || 15); +const DAYS = +(process.env.DAYS || 7); +const NCTRL = +(process.env.CONTROLS || 15); +const CACHE = process.env.CACHE || join(tmpdir(), "cvm-cache"); +const MIN = 3; // runs per side for a workflow to count + +// Commits that flip tools.github.mode between local (MCP) and gh-proxy and change nothing else that +// affects the prompt. Every .md that flips in the commit becomes a case; engine/model edits are excluded. +const SWITCHES = [ + { commit: "d893c7da042", note: "gh-proxy -> local in bulk; same commit moved sandbox runtime to cloud-hypervisor" }, + { commit: "e973b8cc974", note: "ponytail-reviewer: local -> gh-proxy (+ cli-proxy: true)" }, +]; +const CONTROL_AT = "d893c7da042"; // drift control: workflows with no .md edit within DAYS of this commit + +const out = async (cmd, args) => (await sh(cmd, args, { encoding: "utf8", maxBuffer: 1 << 28 })).stdout; +const git = (...a) => out("git", a); +const iso = s => new Date(s).toISOString().replace(".000", ""); +const shift = (t, d) => iso(new Date(+new Date(t) + d * 864e5)); +const med = a => { + if (!a.length) return null; + const s = [...a].sort((x, y) => x - y), m = s.length >> 1; + return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; +}; +const pct = (a, b) => (a ? Math.round(((b - a) / a) * 1000) / 10 : null); +// /responses and OpenAI-style chat count cached tokens inside input_tokens; Anthropic counts them separately. +const inclusive = r => r.path === "/responses" || (!/claude/i.test(r.model) && (r.cache_read_tokens || 0) <= (r.input_tokens || 0)); +const promptOf = r => (r.input_tokens || 0) + (inclusive(r) ? 0 : (r.cache_read_tokens || 0) + (r.cache_write_tokens || 0)); + +async function pool(items, n, fn) { + const res = []; + let i = 0; + await Promise.all(Array.from({ length: n }, async () => { while (i < items.length) { const k = i++; res[k] = await fn(items[k], k); } })); + return res; +} + +const editsOf = async f => (await git("log", "--format=%cI", "--", `.github/workflows/${f}.md`)).split("\n").filter(Boolean).map(iso); + +async function casesFor({ commit, note }) { + const at = iso((await git("show", "-s", "--format=%cI", commit)).trim()); + const diff = await git("show", "--format=", "-U0", commit, "--", ".github/workflows/*.md"); + const files = {}; + let f; + for (const l of diff.split("\n")) { + const m = l.match(/^\+\+\+ b\/\.github\/workflows\/(.+)\.md$/); + if (m) { f = m[1]; files[f] = {}; continue; } + if (!f || f.includes("/")) continue; + const mode = l.match(/^([-+])\s+mode: (gh-proxy|local)\s*$/); + if (mode) files[f][mode[1]] = mode[2]; + if (/^[-+](engine:|model:|\s+id:|\s+model:)/.test(l)) files[f].engine = true; + if (/^\+\s+github:\s*$/.test(l)) files[f].addedGithub = true; + } + const cases = []; + for (const [wf, x] of Object.entries(files)) { + x["-"] ??= x.addedGithub ? "local" : undefined; // a new github: block replaces the default (local MCP) + if (x.engine || !x["+"] || !x["-"] || x["-"] === x["+"]) continue; + const edits = await editsOf(wf); + const prev = edits.filter(t => t < at)[0] ?? shift(at, -DAYS); + const next = edits.filter(t => t > at).at(-1) ?? shift(at, DAYS); + const before = [prev > shift(at, -DAYS) ? prev : shift(at, -DAYS), at]; + const after = [at, next < shift(at, DAYS) ? next : shift(at, DAYS)]; + // Read the MCP side's compiled lock: GH_AW_MCP_CLI_SERVERS_LIST names servers that cli-proxy mounts as CLIs + // on PATH (no tool schemas in context). Frontmatter alone misses cli-proxy set by imports. + const lock = await git("show", `${x["-"] === "local" ? commit + "^" : commit}:.github/workflows/${wf}.lock.yml`).catch(() => ""); + const mcpCli = /GH_AW_MCP_CLI_SERVERS_LIST: .*`github`/.test(lock); + cases.push({ wf, commit, note, at, from: x["-"], to: x["+"], mcpSurface: mcpCli ? "cli" : "schemas", before, after }); + } + return cases; +} + +async function controls() { + const at = iso((await git("show", "-s", "--format=%cI", CONTROL_AT)).trim()); + const edited = new Set((await git("log", `--since=${shift(at, -DAYS)}`, `--until=${shift(at, DAYS)}`, "--name-only", "--format=", "--", ".github/workflows/*.md")).split("\n").filter(Boolean)); + const all = (await git("ls-tree", "--name-only", `${CONTROL_AT}^`, ".github/workflows/")).split("\n").filter(p => p.endsWith(".md") && !edited.has(p)); + return all.map(p => ({ wf: p.replace(/^.*\//, "").replace(/\.md$/, ""), commit: CONTROL_AT, at, control: true, before: [shift(at, -DAYS), at], after: [at, shift(at, DAYS)] })); +} + +async function listRuns(c) { + try { + const j = await out("gh", ["run", "list", "-R", REPO, "--workflow", `${c.wf}.lock.yml`, "--status", "success", "--created", `${c.before[0].slice(0, 10)}..${c.after[1].slice(0, 10)}`, "--limit", "1000", "--json", "databaseId,createdAt,event"]); + const runs = JSON.parse(j); + const side = ([a, b]) => runs.filter(r => r.createdAt >= a && r.createdAt < b); + // nearest the switch first + return { before: side(c.before).sort((x, y) => y.createdAt.localeCompare(x.createdAt)), after: side(c.after).sort((x, y) => x.createdAt.localeCompare(y.createdAt)) }; + } catch { return { before: [], after: [] }; } +} + +async function mine(id) { + const file = join(CACHE, `${id}.json`); + try { return JSON.parse(await readFile(file, "utf8")); } catch {} + const dir = await mkdtemp(join(tmpdir(), "cvm-")); + let row = null; + try { + await out("gh", ["run", "download", String(id), "-R", REPO, "-n", "usage", "-D", dir]); + const reqs = (await readFile(join(dir, "agent", "token_usage.jsonl"), "utf8")).split("\n").filter(Boolean).map(l => JSON.parse(l)) + .filter(r => r.event === "token_usage" && r.status === 200).sort((a, b) => a.timestamp.localeCompare(b.timestamp)); + if (reqs.length) row = { + model: reqs[0].model, path: reqs[0].path, requests: reqs.length, + first: promptOf(reqs[0]), prompt: reqs.reduce((s, r) => s + promptOf(r), 0), + output: reqs.reduce((s, r) => s + (r.output_tokens || 0), 0), + aic: Math.round(reqs.reduce((s, r) => s + (r.ai_credits_this_response || 0), 0) * 100) / 100, + }; + } catch {} // no usage artifact: agent skipped, or artifact expired + finally { await rm(dir, { recursive: true, force: true }); } + await writeFile(file, JSON.stringify(row)); + return row; +} + +const METRICS = ["first", "prompt", "requests", "output", "aic"]; +async function measure(c, runs, raw) { + const sides = {}; + for (const s of ["before", "after"]) { + const rows = []; + for (const r of runs[s]) { + if (rows.length >= CAP) break; + const m = await mine(r.databaseId); + if (m) rows.push({ run: r.databaseId, at: r.createdAt, event: r.event, ...m }); + } + sides[s] = rows; + } + const model = Object.entries([...sides.before, ...sides.after].reduce((a, r) => ((a[r.model] = (a[r.model] || 0) + 1), a), {})).sort((a, b) => b[1] - a[1])[0]?.[0]; + const res = { wf: c.wf, commit: c.commit, control: !!c.control, from: c.from, to: c.to, mcpSurface: c.mcpSurface, at: c.at, model }; + for (const s of ["before", "after"]) { + const rs = sides[s].filter(r => r.model === model); + raw.push(...rs.map(r => ({ wf: c.wf, control: !!c.control, side: s, mode: c.control ? null : s === "before" ? c.from : c.to, ...r }))); + res[s] = { n: rs.length, window: c[s], ...Object.fromEntries(METRICS.map(k => [k, med(rs.map(r => r[k]))])) }; + } + res.ok = res.before.n >= MIN && res.after.n >= MIN; + // % change of the gh-proxy side vs the MCP side. Controls: before vs after, time-aligned with the bulk + // switch (gh-proxy was the before period), so bulk minus control removes platform drift. + const [mcp, proxy] = c.control ? [res.after, res.before] : c.from === "local" ? [res.before, res.after] : [res.after, res.before]; + res.delta = Object.fromEntries(METRICS.map(k => [k, pct(mcp[k], proxy[k])])); + return res; +} + +await mkdir(CACHE, { recursive: true }); +const raw = []; +const flips = (await Promise.all(SWITCHES.map(casesFor))).flat(); +const ctrl = await controls(); +const listed = await pool([...flips, ...ctrl], 8, async c => ({ c, runs: await listRuns(c) })); +const enough = x => x.runs.before.length >= MIN && x.runs.after.length >= MIN; +const flipSel = listed.filter(x => !x.c.control && enough(x)); +const ctrlSel = listed.filter(x => x.c.control && enough(x)).sort((a, b) => b.runs.before.length + b.runs.after.length - (a.runs.before.length + a.runs.after.length)).slice(0, NCTRL); +process.stderr.write(`cases: ${flipSel.length}/${flips.length} switches, ${ctrlSel.length}/${ctrl.length} controls\n`); +const results = await pool([...flipSel, ...ctrlSel], 6, async x => { + const r = await measure(x.c, x.runs, raw); + process.stderr.write(`${r.control ? "ctrl" : "flip"} ${r.wf}: ${r.before.n}/${r.after.n} ${JSON.stringify(r.delta)}\n`); + return r; +}); +const summarize = rs => { + const ok = rs.filter(r => r.ok); + return { workflows: ok.length, runs: ok.reduce((s, r) => s + r.before.n + r.after.n, 0), ...Object.fromEntries(METRICS.map(k => [k, { medianPct: med(ok.map(r => r.delta[k]).filter(v => v != null)), lower: ok.filter(r => r.delta[k] < 0).length }])) }; +}; +const flipsOut = results.filter(r => !r.control); +const report = { + repo: REPO, snapshot: new Date().toISOString().slice(0, 10), cap: CAP, days: DAYS, + metric: "per run: first = prompt tokens of request 1; prompt = sum of prompt tokens over requests; aic = AI credits. delta = % change of gh-proxy vs GitHub MCP (controls: before vs after, same orientation as the bulk switch)", + switches: SWITCHES, + summary: { + mcpAsCli: summarize(flipsOut.filter(r => r.mcpSurface === "cli")), // GitHub MCP behind cli-proxy vs gh + mcpSchemas: summarize(flipsOut.filter(r => r.mcpSurface === "schemas")), // GitHub MCP tool schemas in context vs gh + controls: summarize(results.filter(r => r.control)), + }, + cases: results, +}; +if (process.env.RAW) await writeFile(process.env.RAW, JSON.stringify(raw)); +console.log(JSON.stringify(report, null, 1)); diff --git a/.github/skills/cracktro/slides.md b/.github/skills/cracktro/slides.md index ed2f0c9102f..ae381982145 100644 --- a/.github/skills/cracktro/slides.md +++ b/.github/skills/cracktro/slides.md @@ -68,6 +68,7 @@ Editing: add, remove or move one row per slide in the same change as `deck.js`, | Budget steering | illustrative | As a run nears its AI-credit or time budget, the firewall's API proxy appends a notice to the end of the agent's next model request saying what is left and what to do, once per conversation, so the agent submits what it has before the hard stop and audit records every delivery. | | Model routing | illustrative | Experimental: with model: router, each run gets the cheapest model and reasoning effort that is good enough for its task and stakes, and audit shows why. | | DeterministicOps | illustrative | Run history reveals data-gathering calls repeated on every run; audit flags them, and moving them into deterministic steps cuts token use. | +| CLI vs MCP | illustrative | A fat MCP call such as listing code scanning alerts pours every field of every alert into the context; with gh-proxy and cli-proxy the agent works in bash, discovers each CLI through --help, writes the full response to a file and pipes out only the answer with jq, sort and head, so data grows on disk instead of in context. | | Audit & Optimize | illustrative | A health agent spots a failing or costly workflow, audits its unified session through the agentic-workflows skill, and tunes it so the next run is cheaper. | | Agentic Engineering | | Band header (Agentic Engineering): skip needless runs and give agents memory, hand-offs, expiry, search and measurement. | | Agents that debug agents | illustrative | The agentic-workflows MCP server exposes logs, audit and audit-diff so any agent can debug, audit and optimize other workflows from unified sessions. | diff --git a/docs/public/cracktro/deck.js b/docs/public/cracktro/deck.js index 9966a376b45..75ffd0190bd 100644 --- a/docs/public/cracktro/deck.js +++ b/docs/public/cracktro/deck.js @@ -1039,6 +1039,21 @@ .dto-m { transition: width 1.4s cubic-bezier(.3,0,.2,1); } .dto-big { fill: #3fb950; font: 700 64px "Mona Sans"; text-anchor: middle; opacity: 0; transition: opacity .5s; } .dto-big.on { opacity: 1; } .dto-cap { fill: #9198a1; font: 500 16px var(--mono); text-anchor: middle; } + /* --- cli vs mcp --- */ + .cvm .slide-head .l-grad .ch { background-image: linear-gradient(90deg, #d29922, #3fb950, #4493f8, #d29922); } + .cvm-k { fill: #9198a1; font: 600 14px var(--mono); letter-spacing: .1em; } + .cvm-ln { opacity: 0; transition: opacity .3s; } .cvm-ln.on { opacity: 1; } .cvm-ln.gone { opacity: .15; } + .cvm-ln text { fill: #9198a1; font: 500 15px var(--mono); white-space: pre; } + .cvm-ln .cmd { fill: #f0f6fc; } .cvm-ln .ok { fill: #3fb950; } .cvm-ln .ed { fill: #d29922; } .cvm-ln .jq { fill: #ffa657; font-weight: 700; } .cvm-ln .fl { fill: #79c0ff; font-weight: 700; } .cvm-js text { font-size: 13px; fill: #d29922; } .cvm-js .ed { fill: #f0f6fc; } + .cvm-yaml text { fill: #e6edf3; font: 500 20px var(--mono); white-space: pre; } + .cvm-yaml .k { fill: #79c0ff; } .cvm-yaml .v { fill: #d29922; transition: fill .3s; } .cvm-yaml .v.on { fill: #3fb950; } + .cvm-hl { fill: transparent; stroke: transparent; stroke-width: 2; transition: fill .3s, stroke .3s; } .cvm-hl.on { fill: #3fb9501a; stroke: #3fb950; } + .cvm-bin { opacity: 0; transition: opacity .35s; } .cvm-bin.on { opacity: 1; } + .cvm-bin rect { fill: #161b22; stroke: #3fb950; stroke-width: 2; } + .cvm-bin .n { fill: #f0f6fc; font: 600 19px var(--mono); } .cvm-bin .d { fill: #9198a1; font: 500 14px var(--mono); } + .cvm-ctx { opacity: 0; transition: opacity .4s; } .cvm-ctx.on { opacity: 1; } + .cvm-ctx rect { fill: #ffffff08; stroke-width: 1.5; } .cvm-ctx text { font: 500 16px var(--mono); white-space: pre; } + .cvm-cap { text-anchor: middle; font-size: 13px; letter-spacing: .04em; } #sflOut0.on rect { stroke: #3fb950; filter: drop-shadow(0 0 14px #3fb950); } #sflOut1.on rect { stroke: #d29922; filter: drop-shadow(0 0 14px #d29922); } .fm-dot { filter: drop-shadow(0 0 8px currentColor); opacity: 0; } @@ -8917,6 +8932,220 @@ }); } + // CLI vs MCP: a fat MCP call like list_code_scanning_alerts dumps every field of every alert into the context; with gh-proxy and cli-proxy the agent discovers CLIs with --help, writes the response to a file and pipes out only the answer + { + resetIndex(); + const VW = 1620, + VH = 684; + const esc = x => x.replace(/&/g, "&").replace(//g, ">"); + // synthetic list_code_scanning_alerts response: one alert of many, every field into context + const CALL = '→ list_code_scanning_alerts {"state":"open"}'; + const WALL = [ + '← [{"number":1284,"state":"open","created_at":"…",', + ' "url":"…","html_url":"…","instances_url":"…",', + ' "rule":{"id":"js/sql-injection","severity":"error",', + ' "security_severity_level":"high","name":"…",', + ' "description":"Database query built from user-…",', + ' "tags":["security","external/cwe/cwe-089"]},', + ' "tool":{"name":"CodeQL","guid":null,"version":"…"},', + ' "most_recent_instance":{"ref":"refs/heads/main",', + ' "analysis_key":"…","environment":"{}","state":"open",', + ' "commit_sha":"…","classifications":[],', + ' "message":{"text":"This query depends on a user-…"},', + ' "location":{"path":"src/db.js","start_line":42,', + ' "end_line":42,"start_column":9,"end_column":31}}},', + ' {"number":1283,"state":"open", … × every alert', + '→ page 2 · page 3 · … every page into context', + ]; + const BINS = [ + ["gh", "gh-proxy · pre-authenticated, no MCP"], + ["safeoutputs", "cli-proxy · via the MCP gateway"], + ["docs-search", "cli-proxy · via the MCP gateway"], + ]; + // discover with --help, write the response to a file, pipe out only the answer + const SHELL = [ + ["cmd", "$ gh help | grep api"], + ["", " api: Make an authenticated GitHub API request"], + ["cmd", "$ gh api --help | grep paginate"], + ["", " --paginate fetch all pages of results"], + ["cmd", "$ gh api --paginate \\"], + ["cmd", " repos/{owner}/{repo}/code-scanning/alerts > alerts.json"], + ["ok", "✓ every alert, every page on disk · 0 lines in context"], + ["cmd", "$ jq -r '.[].rule.severity' alerts.json | sort | uniq -c"], + ["", " 12 error"], + ["", " 85 warning"], + ["cmd", "$ jq -r '.[].most_recent_instance.location.path' \\"], + ["cmd", " alerts.json | sort | uniq -c | sort -rn | head -2"], + ["", " 9 src/db.js"], + ["", " 3 src/api/user.js"], + ]; + let g = ` + `; + g += ``; + [ + [0, 600, "MCP · RESPONSE INTO CONTEXT"], + [600, 1020, "FRONT MATTER"], + [1020, VW, "BASH · DISCOVER · FILES · PIPES"], + ].forEach(([x0, x1, t], i) => { + g += `${t}`; + if (x0) g += ``; + }); + g += ``; + g += ``; + // what ends up in the context: qualitative, synthetic + const ctx = (x, id, col, txt) => + `IN CONTEXT` + + `${esc(txt)}`; + // MCP: one fat call; the whole response, every page, goes straight into the context + g += `agent context · github: local`; + g += `TOOL CALL`; + g += `${esc(CALL)}`; + g += `RESPONSE · STRAIGHT INTO CONTEXT`; + WALL.forEach((t, i) => (g += `${esc(t)}`)); + g += ctx(16, "cvmXm", "#d29922", "every field of every alert, every page") + ``; + // front matter: two lines move GitHub reads to gh and mount every MCP server as a CLI + g += `.github/workflows/triage.md`; + g += ``; + g += `tools: github:`; + g += ` mode: local`; + g += ` cli-proxy: false`; + g += `ON PATH · SAME SERVERS`; + BINS.forEach(([n, d], i) => { + const y = 306 + i * 64; + g += `${n}${esc(d)}`; + }); + g += `--helpCLIs describe themselves`; + // bash: the agent discovers the CLI, writes the response to a file and pipes out only the answer + g += `agent shell · gh-proxy + cli-proxy`; + g += `PROMPT · ONE LINE PER CLI`; + g += `gh · safeoutputs · docs-search · run --help`; + g += `SHELL · DISCOVER → FILE → PIPES`; + // highlight files and pipe stages + const pipe = t => + esc(t) + .replace(/(> )?[\w-]+\.json/g, m => `${m}`) + .replace(/\b(jq|head|sort|uniq|grep)\b/g, m => `${m}`) + .replace(/ \| /g, ` | `); + SHELL.forEach(([c, t], i) => (g += `${pipe(t)}`)); + g += ctx(1036, "cvmXc", "#3fb950", "two help lines · two counts · two paths") + ``; + g += `MEASURED, schemas only · 1 workflow, MCP → gh-proxy, 15 runs/side, 23 Aug – 3 Sep 2026 · first request 16.7k → 12.8k tokens, whole run ≈ same · snapshot 17 Nov 2026`; + g += ``; + const svg = document.createElementNS("http://www.w3.org/2000/svg", "svg"); + svg.setAttribute("class", "fm-svg ao-svg"); + svg.setAttribute("viewBox", `0 0 ${VW} ${VH}`); + svg.innerHTML = g; + const $ = id => svg.querySelector("#" + id), + dot = svg.querySelector(".mra-dot"); + const ease = u => (u < 0.5 ? 2 * u * u : 1 - (-2 * u + 2) ** 2 / 2); + const clamp = u => Math.min(1, Math.max(0, u)); + // phases: [start ms, duration ms] + const P = { CALL: [400, 500], SCH: [1100, 3000], W0: [4400, 700], FLIP: [5100, 700], BIN: [5800, 1200], W1: [7100, 700], SH: [7800, 3600], BIG: [11600, 1] }; + const CYC = 15600; + let t0 = 0, + cyc = -1, + done = new Set(); + const once = (k, fn) => { + if (!done.has(k)) { + done.add(k); + fn(); + } + }; + const reveal = (ids, f, snd) => + ids.forEach((id, i) => { + if (f >= (i + 0.2) / ids.length && !$(id).classList.contains("on")) { + $(id).classList.add("on"); + if (snd) snd(i); + } + }); + const reset = () => { + svg.querySelectorAll(".on, .hot, .gone").forEach(x => x.classList.remove("on", "hot", "gone")); + $("cvmV0").textContent = "local"; + $("cvmV1").textContent = "false"; + done = new Set(); + $("cvmLane0").classList.add("on"); + $("cvmMcp").classList.add("on"); + }; + const wire = (id, f) => { + const path = $(id), + n = path.getTotalLength(), + q = path.getPointAtLength(n * ease(Math.min(1, f / 0.85))); + path.classList.add("hot"); + dot.style.opacity = 1; + dot.setAttribute("cx", q.x); + dot.setAttribute("cy", q.y); + dot.setAttribute("fill", id === "cvmW0" ? "#d29922" : "#3fb950"); + }; + const shellIds = ["cvmP", ...SHELL.map((_, i) => "cvmS" + i)]; + slides.push({ + group: "cost", + notes: + "CLI vs MCP: the workflow, alerts, counts, files and paths on this slide are illustrative and synthetic; only the caption's figures are measured. Take a fat API call such as listing a repository's code scanning alerts. Through the GitHub MCP server, list_code_scanning_alerts returns every alert as a full JSON object, rule, tool, latest instance, location, message and links, and the whole response goes straight into the context, page after page. A bigger repository means a fuller context, and the agent has nowhere else to put it. Two front matter lines change the surface without changing what the agent can reach. tools.github.mode: gh-proxy drops the GitHub MCP server and gives the agent a pre-authenticated gh CLI through a proxy sidecar. tools.cli-proxy: true mounts every other MCP server, safe outputs included, as a command on PATH that still calls through the MCP gateway, so policies and guardrails stay the same. Now the agent works in bash. It discovers what it needs by itself: the prompt has one line per command, and gh help, gh api --help or any server's --help explain the rest, a few lines at a time. It writes the full response to a file, every alert and every page, and none of it enters the context. Then pipes pull out only the answer: jq picks fields, sort and uniq count by severity and by file, head keeps the top two. The data grows on disk, not in the context, and the agent can query the same file again without fetching it again. The caption is the one thing we measured, and it is about tool schemas only: one gh-aw workflow that switched from the GitHub MCP server to gh-proxy plus cli-proxy, same model, 15 runs on each side between 23 August and 3 September 2026, had a median first request of 16,690 tokens before and 12,766 after, while its whole-run tokens and AI credits stayed within a few percent. A larger, confounded switch of 21 workflows swung from minus 78 to plus 123 percent per workflow, so we show no run-level number. Snapshot taken 17 November 2026. Both modes need bash; strict workflows with bash disabled stay on MCP.", + enter() { + t0 = 0; + cyc = -1; + }, + tick(t) { + if (!t0) t0 = t; + const e = t - t0, + c = Math.floor(e / CYC), + u = e % CYC; + if (c !== cyc) { + cyc = c; + reset(); + } + const ph = k => clamp((u - P[k][0]) / P[k][1]); + svg.querySelectorAll(".fm-wire.hot").forEach(w => w.classList.remove("hot")); + dot.style.opacity = 0; + // one fat call + if (u >= P.CALL[0]) once("call", () => (sfx.key(), $("cvmC0").classList.add("on"))); + // its whole response floods the context + if (u >= P.SCH[0]) { + reveal( + WALL.map((_, i) => "cvmT" + i), + ph("SCH"), + i => i % 3 === 0 && sfx.pop(i / 3), + ); + if (ph("SCH") >= 1) $("cvmXm").classList.add("on"); + } + if (u >= P.W0[0] && u < P.FLIP[0]) wire("cvmW0", ph("W0")); + // two front matter lines flip the surface + if (u >= P.FLIP[0]) + once("flip", () => { + $("cvmLane1").classList.add("on"); + $("cvmFm").classList.add("on"); + $("cvmV0").textContent = "gh-proxy"; + $("cvmV1").textContent = "true"; + ["cvmV0", "cvmV1", "cvmH0", "cvmH1"].forEach(id => $(id).classList.add("on")); + svg.querySelectorAll("#cvmMcp .cvm-ln").forEach(x => x.classList.add("gone")); + sfx.compile(); + }); + if (u >= P.BIN[0]) + reveal( + BINS.map((_, i) => "cvmB" + i), + ph("BIN"), + i => sfx.pop(4 + i), + ); + if (u >= P.W1[0] && u < P.SH[0]) wire("cvmW1", ph("W1")); + // discover with --help, write the file, pipe out the answer + if (u >= P.SH[0]) { + once("sh", () => { + $("cvmLane2").classList.add("on"); + $("cvmCli").classList.add("on"); + }); + const f = ph("SH"); + reveal(shellIds, f, i => i % 2 === 0 && sfx.pop(i / 2)); + if (f >= 1) $("cvmXc").classList.add("on"); + } + if (u >= P.BIG[0]) + once("big", () => { + $("cvmBig").classList.add("on"); + sfx.pop(7); + }); + }, + el: h("section", { class: "slide cvm" }, h("div", { class: "slide-head" }, h("div", { class: "kicker" }, "gh-proxy + cli-proxy · bash, files and pipes, not context"), h("h2", {}, letters("CLI"), letters("vs", "l-out"), letters("MCP", "l-grad"))), svg), + }); + } + // Agents that debug agents: any agent calls the agentic-workflows MCP server; logs, audit and audit-diff read runs as unified sessions to debug, audit and optimize other agents { const AG = pickAgent();