Skip to content

cracktro: add CLI vs MCP slide to Cost Optimization - #78656

Merged
pelikhan merged 5 commits into
mainfrom
pelikhan/cracktro-cli-vs-mcp-slide
Nov 17, 2026
Merged

pelikhan merged 5 commits into
mainfrom
pelikhan/cracktro-cli-vs-mcp-slide

Conversation

@pelikhan

@pelikhan pelikhan commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a CLI vs MCP slide to the cracktro deck's Cost optimization band (row 66 in .github/skills/cracktro/slides.md).

Message

Fat API calls are where MCP hurts most. The left card shows a synthetic list_code_scanning_alerts call: the response carries every field of every alert (rule, tool, latest instance, location, message, links), and every page of it goes straight into the context.

With tools.github.mode: gh-proxy and tools.cli-proxy: true, the agent works in bash:

  1. Self-discovery. It runs gh help | grep api and gh api --help | grep paginate, so CLIs describe themselves a few lines at a time.
  2. Files. It runs gh api --paginate …/code-scanning/alerts > alerts.json, so every alert lands on disk and none of it enters the context.
  3. Pipes. jq … | sort | uniq -c | sort -rn | head -2 pulls out only the answer: counts by severity and the top files.

The IN CONTEXT boxes are qualitative. Measured schema-overhead data is now just a caption footnote.

Measured data (rooted in run traces)

The new miner .github/skills/cracktro/refresh/cli-vs-mcp.mjs reads the usage artifact of gh-aw runs before and after commits that switch tools.github.mode. The method is documented in refresh/cli-vs-mcp.md. Snapshot 2026-11-17.

Comparison Workflows · runs First request Whole run
ponytail-reviewer: GitHub MCP (16 schemas loaded) → gh-proxy + cli-proxy, same model 1 · 30 16,690 → 12,766 (−23%) prompt +4.5%, AI credits +2.9%
Bulk switch commit (also changed the sandbox runtime) 21 · 250 MCP side ~+0.4k AI credits median −19%, range −78% … +123%
Drift controls (unchanged workflows) 14 · 212 0% AI credits −1.7%
  • On the slide: one MEASURED, schemas only caption with the ponytail first-request medians (16.7k → 12.8k), window, runs per side, snapshot and "whole run ≈ same". The flow itself is synthetic.
  • In the notes only: the bulk result, because it is confounded and noisy.
  • Engines: Claude and Codex appear to defer MCP schemas, so the large first-request saving applies only where schemas load up front.

Validation

  • node --check docs/public/cracktro/deck.js passes.
  • Layout was checked via SVG bounding boxes in a local preview.
  • ⚠️ Screenshot capture renders blank in this environment. Please attach a 1920×1080 screenshot of the slide at its final step.
  • ⚠️ Playwright tests/talk.spec.ts not run (docs/node_modules missing).

pelikhan and others added 4 commits November 16, 2026 22:29
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…request data

Mine same-workflow gh-proxy switches from run usage artifacts and add the
cracktro refresh mini-skill to recompute them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…p discovery, files and pipes

The MCP side now shows list_code_scanning_alerts pouring every field of
every alert into context; the bash side discovers gh api with --help,
writes every page to alerts.json and pipes out counts with jq/sort/uniq/head.
Measured schema-overhead data moves to a single caption footnote.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review November 17, 2026 09:14
Copilot AI adaptive review requested due to automatic review settings November 17, 2026 09:15
…-vs-mcp-slide

# Conflicts:
#	.github/skills/cracktro/content-rules.md
@pelikhan
pelikhan merged commit bef5588 into main Nov 17, 2026
8 checks passed
@pelikhan
pelikhan deleted the pelikhan/cracktro-cli-vs-mcp-slide branch November 17, 2026 09:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The cache can follow attacker-controlled symlinks and truncate developer-writable files, the new heading violates an exact-text test assertion, and block-scalar parsing misclassifies measurements, compromising the schema-overhead comparison.

4 open issues
What changed in this PR

Adds a CLI vs MCP slide to the cracktro deck’s Cost Optimization band, illustrating how CLI discovery, files, and pipes can reduce data entering agent context.

Changes:

  • Adds an animated comparison with synthetic code-scanning examples and a measured first-request token caption.
  • Adds a run-artifact miner and refresh instructions, including measurement limits and confounding factors.
  • Updates the slide index, refresh registry, and public-content rules.
File Description
docs/​public/​cracktro/​deck.js Adds slide visuals, animation, commands, and speaker notes.
.github/​skills/​cracktro/​slides.md Registers the slide and its takeaway.
.github/​skills/​cracktro/​refresh/​cli-vs-mcp.mjs Mines workflow usage artifacts for before/after comparisons.
.github/​skills/​cracktro/​refresh/​cli-vs-mcp.md Documents measurement sources, refresh steps, and limitations.
.github/​skills/​cracktro/​data-refresh.md Links the new refresh guide.
.github/​skills/​cracktro/​content-rules.md Records the slide’s measured-data exception.

Review effort: Balanced (adaptive)

const CAP = +(process.env.CAP || 15);
const DAYS = +(process.env.DAYS || 7);
const NCTRL = +(process.env.CONTROLS || 15);
const CACHE = process.env.CACHE || join(tmpdir(), "cvm-cache");
sfx.pop(7);
});
},
el: h("section", { class: "slide cvm" }, h("div", { class: "slide-head" }, h("div", { class: "kicker" }, "gh-proxy + cli-proxy · bash, files and pipes, not context"), h("h2", {}, letters("CLI"), letters("vs", "l-out"), letters("MCP", "l-grad"))), svg),
// Read the MCP side's compiled lock: GH_AW_MCP_CLI_SERVERS_LIST names servers that cli-proxy mounts as CLIs
// on PATH (no tool schemas in context). Frontmatter alone misses cli-proxy set by imports.
const lock = await git("show", `${x["-"] === "local" ? commit + "^" : commit}:.github/workflows/${wf}.lock.yml`).catch(() => "");
const mcpCli = /GH_AW_MCP_CLI_SERVERS_LIST: .*`github`/.test(lock);

## 6. Validation

- `node --check docs/public/cracktro/deck.js` and the preview at `#66` (row 66 of `slides.md`): boxes fit their 536-px rects; caption stays inside the 1620-px viewBox.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants