Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/skills/cracktro/content-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Enforced by `docs/tests/talk.spec.ts`. Load before adding public text, data, ava
- Exception: the For PMs slide (`pm-` classes) shows two clickable processes (feature intake, launch readiness) whose workflow names, inputs, outputs and ideas come from the public, MIT-licensed `chrizbo/agentics-beyond-code` repository, credited by a linked repo slug in the kicker and in the caption and notes (no personal names or avatars); the test's repo-reference regex only blocks `github/` and `githubnext/` refs. The issue and report contents are synthetic; its notes and caption say so.
- Exception: the Scan · Fix · Learn slide (`sfm-` panel) shows measured counts from the public .NET runtime repository's `ci-failure-scan`, `ci-failure-fix` and `ci-failure-scan-feedback` workflows (window 5 May to 14 Nov 2026, 193 days, labelled on the panel: 305 `[ci-scan]` issues, 270 closed; 22 `[ci-fix]` PRs, 5 merged; 12 merged `[ci-scan-feedback]` prompt-edit PRs; 882 runs; issues per month). The loop below it is illustrative; its notes and caption say so.
- Exception: the data slides before the finale. "First commit to GA" (`hist-` classes), shows gh-aw's measured public history (git on main plus GitHub search counts, 12 Aug 2025 to GA on 28 Oct 2026); its speaker notes give the sources and say account attribution is not proof of AI use. "How PRs land" (`land-` classes), shows measured time-to-merge and lines-changed distributions of merged PRs, split by the same origins; issues opened under a person's account may still have been drafted in the Copilot app or CLI, which GitHub does not record. "How workflows evolve" (`evo-` classes), shows measured first-parent commits on main that changed `.github/workflows/**/*.md` (monthly by author type, live workflow count, lines-changed histogram and statistical tests: Spearman, Mann–Whitney, Gini), 12 Aug 2025 to GA; its notes give the method.
- Exception: the CLI vs MCP slide (`cvm-` classes) has one measured caption (tool schemas only) from gh-aw's own runs of one workflow that switched from the GitHub MCP server to `gh-proxy` plus `cli-proxy`, same model, 15 runs per side, 23 Aug to 3 Sep 2026, snapshot 17 Nov 2026: median first request 16,690 → 12,766 tokens, whole-run cost about the same; its notes add a confounded 21-workflow bulk switch without a slide figure. The code scanning alerts, JSON response, commands, counts, files and paths in the flow are synthetic and its notes say so. Refresh with `refresh/cli-vs-mcp.md`.
- The only URLs allowed in `deck.js` are `https://github.github.com/gh-aw/introduction/overview/`, the For PMs source link `https://github.com/chrizbo/agentics-beyond-code` and the repo-widget endpoint `https://api.github.com/repositories/`. The widget (`repoWidget(id, name, stars, cls)`) addresses repos by numeric id, fetches only on `https:` pages (never on localhost, `file://` or in tests), once per repo, and keeps the hard-coded snapshot count on any failure; it shows only the repo octicon, never an owner avatar. Index & search (`ix-rw`) shows one for the open-source qmd repo labelled `qmd`, never its owner handle (snapshot default 31k on 17 Nov 2026).
- Speaker notes of every slide marked `illustrative` in `slides.md` must match `/illustrative|synthetic|simulated/i`.
- Never write slide numbers or slide counts outside `slides.md`; refer to slides by heading and CSS prefix.
Expand Down
1 change: 1 addition & 0 deletions .github/skills/cracktro/data-refresh.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Every slide that shows measured data has a refresh mini skill. Load only the one
| Findings that land | `refresh/findings-that-land.md` |
| Personas · Probe · Patch | `refresh/personas-probe-patch.md` |
| Scan · Fix · Learn | `refresh/scan-fix-learn.md` |
| CLI vs MCP | `refresh/cli-vs-mcp.md` |
| Skills in review | `refresh/skills-in-review.md` |
| Ponytail Reviewer | `refresh/ponytail-reviewer.md` |

82 changes: 82 additions & 0 deletions .github/skills/cracktro/refresh/cli-vs-mcp.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
name: cracktro-refresh-cli-vs-mcp
description: Recompute the measured data on the cracktro "CLI vs MCP" slide and update deck.js.
---

# Refresh "CLI vs MCP"

Load `.github/skills/cracktro/SKILL.md` first. Never read `docs/public/cracktro/deck.js` whole; `grep -n`, then `view_range`.

## Workflows mined

Same-workflow before/after natural experiments: commits that flip `tools.github.mode` between `local` (GitHub MCP server) and `gh-proxy` (`gh` CLI). Each run's `usage` artifact is mined.

| Workflow file | Repo | Role in the slide | Signal mined |
|---|---|---|---|
| `.github/workflows/ponytail-reviewer.md` | github/gh-aw | the two measured boxes and the caption | first-request prompt tokens, run totals; switched `local` → `gh-proxy` + `cli-proxy: true` in one commit, same model |
| 21 workflows in the bulk `gh-proxy` → `local` commit (`ai-moderator.md`, `tidy.md`, `go-fan.md`, ... listed in the miner's `cases[]`) | github/gh-aw | notes only (whole-run AI credits) | same metrics; the commit also changed the sandbox runtime |
| the 14 busiest workflows with no `.md` edit within 7 days of the bulk commit | github/gh-aw | notes only (drift control) | same metrics, before vs after the bulk commit |

## 1. Slide

- Marker: `// CLI vs MCP:` (`grep -n "// CLI vs MCP:" docs/public/cracktro/deck.js`). CSS prefix `cvm-`. Group `cost`. Row 66 in `slides.md` (Notes `illustrative`: the flow is synthetic).
- The flow (fat `list_code_scanning_alerts` response vs `--help` → `> alerts.json` → `jq | sort | uniq -c | head`) and its `IN CONTEXT` boxes are synthetic and qualitative: no measured numbers there.
- Data lives inside that block only:
1. The `.cvm-cap` caption: `MEASURED, schemas only`, workflow count, runs per side, window, first-request medians (16.7k → 12.8k), whole-run verdict, snapshot.
2. `notes:` string: repeats every number, the window, the snapshot, the bulk result and its confound.
- Elsewhere: the `content-rules.md` Exception bullet "the CLI vs MCP slide (`cvm-` classes)".

## 2. Window & snapshot

- Each side is bounded by the workflow's previous/next `.md` edit and capped at `DAYS` (7) from the switch; the `CAP` (15) successful runs nearest the switch are kept. A side needs `MIN` 3 runs.
- Baseline (snapshot 2026-11-17): ponytail-reviewer MCP side 2026-08-23 → 08-27, gh-proxy side 08-27 → 09-03; bulk commit 2026-09-05, sides 08-29 → 09-12.

## 3. Sources

- `gh run download <id> -R github/gh-aw -n usage`: `agent/token_usage.jsonl`, one line per model request (`input_tokens`, `cache_read_tokens`, `cache_write_tokens`, `output_tokens`, `ai_credits_this_response`, `path`, `model`). Skip `detection/`.
- Surface of the MCP side: the compiled lock at the MCP-side commit. `GH_AW_MCP_CLI_SERVERS_LIST` naming `` `github` `` means cli-proxy mounted the GitHub MCP as a CLI (no schemas in context); otherwise its schemas were offered to the engine. Front matter alone misses `cli-proxy` set by imports.
- Tool count: `gh run download <id> -n agent`, `mcp-logs/tools.json` (`servers.github.length`) of an MCP-side run.

## 4. Queries

Run from the repo root (needs full git history and `gh auth login`); downloads are cached in `CACHE`:

```bash
RAW=$TMPDIR/cvm-raw.json node .github/skills/cracktro/refresh/cli-vs-mcp.mjs > $TMPDIR/cvm-out.json
node -e 'const o=require(process.env.TMPDIR+"/cvm-out.json");console.log(JSON.stringify(o.summary));const c=o.cases.find(c=>c.wf==="ponytail-reviewer");console.log(c.before,c.after,c.delta)'
```

- To add a switch, append its short SHA to `SWITCHES`; find candidates with `git log -G'mode: (gh-proxy|local)' --format='%h %cI %s' -- .github/workflows/*.md`.
- Medians are per workflow per side; `summary.*.<metric>.medianPct` is the median of per-workflow % changes (MCP side → gh-proxy side), `lower` the count that fell.

Baseline (snapshot 2026-11-17):

| Group | Workflows · runs | First request | Prompt tokens | AI credits |
|---|---|---|---|---|
| ponytail-reviewer, schemas → gh-proxy + cli-proxy | 1 · 30 | 16,690 → 12,766 (−23.5%) | 182,639 → 190,929 (+4.5%) | 4.15 → 4.27 (+2.9%) |
| bulk commit, all cases | 21 · 250 | ~+0.4k on MCP side (median) | — | median −19.2%, 16/21 lower, range −78% … +123% |
| bulk, GitHub MCP behind cli-proxy | 13 · 154 | −1.3% | −22.1% | −6.4% (10/13) |
| drift controls | 14 · 212 | 0% | −3% | −1.7% (9/14) |

## 5. Update steps

1. Update the `.cvm-cap` caption (medians, window, runs per side, snapshot); the `ctx(...)` boxes stay qualitative.
2. Update the notes: every number, window, snapshot, the bulk range and confound. Keep `illustrative and synthetic` for the flow.
3. Update the `content-rules.md` Exception bullet. `slides.md` row 66 stays free of numbers and dates.
4. If a newer clean switch (one workflow, same model, no runtime change) exists, prefer it for the boxes and say which in the notes.

## 6. Validation

- `node --check docs/public/cracktro/deck.js` and the preview at `#66` (row 66 of `slides.md`): boxes fit their 536-px rects; caption stays inside the 1620-px viewBox.
- The ponytail case shows `mcpSurface: "schemas"` and equal `requests` medians on both sides; controls stay near 0%.
- No URLs, SHAs, run IDs or logins on the slide or in the notes.

## 7. Pitfalls

- Logs and artifacts expire after ~90 days: the baseline runs (Aug–Sep 2026) become unrecomputable around early December 2026. Keep the baseline table; refresh from newer switches instead.
- `/responses` (OpenAI, Codex, MAI) `input_tokens` already include cached tokens; Anthropic counts `cache_read`/`cache_write` separately. The miner's `inclusive()` handles both; never add cache to `/responses` input.
- Claude and Codex engines appear to defer MCP tool schemas: their local-mode first request was only ~0.4k larger than gh-proxy. Only engines that load schemas up front (Copilot CLI with tool search off) show the ~3.9k jump. Do not generalise the −23% beyond that.
- Whole-run tokens and credits are dominated by task variance (n = 3–15 per side). Do not put a whole-run percentage on the slide.
- The bulk commit also moved the sandbox to cloud-hypervisor and 18 files changed engine (excluded): it is confounded.
- Many successful runs have no `usage` artifact (agent skipped); `gh run list` caps at 1000, so the miner filters by `--created`.
- macOS `awk` has no `\s`; use `[ ]`.
Loading
Loading