Ten coding harnesses, one model, the same 113 tasks, one attempt each.
| harness | solved | cost per task | cost per solved issue | mean time |
|---|---|---|---|---|
| senior-dev | 62 of 113, 54.9% | 22¢ | 1x | 54 min |
| mini-swe-agent | 56, 49.6% | 38¢ | 1.9x | 44 min |
| codex | 51, 45.1% | 37¢ | 2.1x | 46 min |
| pi | 42, 37.2% | 35¢ | 2.4x | 52 min |
| omp | 31, 27.4% | 50¢ | 4.5x | 49 min |
| opencode | 30, 26.6% | 50¢ | 4.8x | 48 min |
| kilo | 30, 26.6% | 48¢ | 4.6x | 54 min |
| claude-code | 16, 14.2% | 19¢ | 3.4x | 32 min |
| deepseek-harness | 16, 14.2% | 150¢ | 26.6x | 94 min |
| muse-code | 3, 2.7% | 12¢ | 11.3x | 16 min |
senior-dev solved the most issues and paid the least for each one it solved: nearly 4x the issues claude-code solved, at about half the cost per solve of the next best harness.
Since then, on the same 113 tasks: 88 solved (77.9%, exact 95% CI 69.1% to 85.1%) with DeepSeek V4.1 Flash, and 78 (69.0%, 59.6% to 77.4%) with Kimi K3. Those runs are senior-dev alone, not a comparison.
| Benchmark | full DeepSWE set, 113 tasks, one seed per harness |
| Model | deepseek/deepseek-v4-flash-0731 through OpenRouter |
| Verifiers | official DeepSWE at 0b9fabb |
| Budget | 3 h per task |
| Isolation | four shards per harness, a dedicated OpenRouter key per harness |
| Ran | nine harnesses on 2026-09-10–11 UTC; senior-dev on 2026-09-12 UTC |
arms.csv has one row per harness: solved, reward rate, valid
grades, invalid outcomes, mean F2P and P2P, OpenRouter spend, cost per task,
mean agent seconds and model.
The nine competing harness versions below were checked against all 113 saved task records per harness. senior-dev uses its public integration and first stable release references, explained below. Release dates come from the linked upstream releases; they are not installation dates. These references describe the historical comparison and its public lineage.
| harness | tested version / public reference | source revision / integration commit | release date (UTC) |
|---|---|---|---|
| senior-dev | First stable: codeaf v0.5.0 | dd0fcc654 — #1488 |
2026-09-30 |
| mini-swe-agent | 2.4.6 | a83fcae |
2026-07-23 |
| codex (Codex CLI) | 0.154.0 | 6b9826e |
2026-09-09 |
| pi | 0.85.1 | d981de1 |
2026-09-05 |
| omp (Oh My Pi) | 18.1.17 | 3b3a6dc |
2026-09-10 |
| opencode | 1.18.30 | 3104c14 |
2026-09-09 |
| kilo | 7.6.2 | 3d04228 |
2026-09-10 |
| claude-code | 2.1.267 | — | 2026-09-09 |
deepseek-harness (@deepseek-ai/dsh) |
0.1.5-rc.1 | — | 2026-09-10 |
| muse-code | 1.1.1-R2514.1 | — | Not independently established |
versions.csv records the same versions alongside source
revisions where recorded, public release timestamps and source URLs. Its
harness values join to arms.csv; revisions are stored in full where available.
Empty revision, release timestamp, or URL fields mean that information was not
established for that row.
For senior-dev, version names the first stable codeaf release containing the
agent, and source_revision names the public integration commit from #1488.
That commit is contained in v0.5.0; it is distinct from the release tag's commit,
4949397498f775a80de16d3e6429b0f6f933f54c. The benchmark was run before the
integration. This row traces where the measured agent became publicly available
in codeaf; it does not describe a new benchmark run on v0.5.0.
Muse Code's version comes from the frozen benchmark records. Its historical publication date and whether a newer build was available at launch were not independently established.
Most competing harness versions were released within a week of the campaign. mini-swe-agent 2.4.6 was about seven weeks old, but was still the latest published release at launch. The comparison does not establish how newer versions perform.
- OpenCode was 1.x. Version 1.18.30 was the latest production 1.x release at launch; 1.18.31 followed on September 14. V2 development already existed, as a contemporaneous upstream issue shows. This row does not measure OpenCode 2.
- Claude Code was pinned before a patch release. Version 2.1.268 was published on September 10 at 20:30:54 UTC, before 112 of this arm's 113 tasks started. Its release notes include a fix for HTTP 400 responses on third-party Anthropic-compatible endpoints. The campaign used such an endpoint; whether that fix would change the result has not been established. Version 2.1.269 arrived after all tasks had started, near the end of the arm.
- DeepSeek Harness had a newer release candidate. Version
0.1.5-rc.2
was published on September 10 at 15:09:34 UTC, before this arm ran. The
frozen package lock selected npm's
latesttag at 0.1.5-rc.1. The rc.2 release notes describe feedback dialogs and interface layout changes; no run on rc.2 is included here.
- solved: tasks the verifier passed, out of 113.
- cost per task: billed OpenRouter spend divided by 113, so tasks without a verifier result stay in the denominator.
- cost per solved issue: spend divided by tasks solved, shown as a multiple of senior-dev's (39.9¢).
- mean time: mean agent wall time per task.
- One seed per harness. senior-dev's 62 against mini-swe-agent's 56 is not a statistically resolved difference.
- senior-dev departs from the sampling contract: the other nine sent temperature 1.0 and top-p 0.95, senior-dev sent neither, so provider defaults applied.
- Five tasks produced no verifier outcome: codex 1, pi 1, omp 2, opencode 1. They count as unsolved.
- omp, opencode, kilo, deepseek-harness, claude-code and muse-code did not record per-attempt cost; their spend is known at harness level only.
- senior-dev ran under an earlier name for the binary; values are rewritten to
senior-dev.