Claude Code Challenge Tester

This is a program that validates your progress on the "Build your own Claude Code" challenge.

Requirements for binary

• Following environment variables:
  • CODECRAFTERS_SUBMISSION_DIR - root of the user's code submission
  • CODECRAFTERS_TEST_CASES_JSON - test cases in JSON format

User code requirements

• A binary named your_program.sh that executes the program.
• A file named codecrafters.yml, with the following values:
  • debug

LSP extension

Five graded stages connect an LSP tool and diagnostic feedback to the existing agent loop:

1. lc1: Set up the LSP connection.
2. df3: Find the implementation.
3. rf4: Find the affected callers.
4. sy5: Keep the analysis current.
5. dg6: Catch an error after editing.

The supplied transport starts the configured server, handles JSON-RPC framing and initialization, and queues notifications while requests are running. The tester provides CODECRAFTERS_LSP_URL to the submitted program. The native starter clients retrieve extensionToLanguage through the configuration action and call request, notify, and notifications through that local URL. Document language IDs come from configuration rather than a Python default. Every LSP case supplies a connection. Setup checks the configuration request and model-facing schema; learners construct the supplied client directly. It uses Read to inspect the example and passes before navigation is implemented. The local HTTP facade is course scaffolding; the server still speaks stdio LSP. Learners construct document messages, track versions, return tool results and choose how diagnostic feedback reaches the model. The helper never runs an agent.

The reference agent uses Python's standard library. The controlled stdio fixture runs from the tester binary, so grading doesn't require installing Pyright.

make test_lsp
make test_lsp_offline

These commands cover 16 graded cases and ten deliberately broken clients: missing configuration requests, missing LSP advertisement, incomplete tool schemas, wrong coordinates, lost definition results, omitted declaration context, unsynchronized writes, discarded unversioned reports, reused cached feedback and false clean diagnostics on timeout. Separate transport tests cover fragmented startup, notifications during requests, failed initialization and process cleanup.

The exercise migrates pricing.total from integer cents to a Price TypedDict. A controlled partial caller edit still returns Price as int. The scripted model requires the preceding Write's error before selecting the repair; the repaired Write must receive an empty diagnostic publication. Logs show both the error and clearing. This is a fixed demonstration, not a guarantee that a live model will make an error. Final files, protocol traffic and matching tool results are checked.

Navigation operations are goToDefinition and findReferences, with filePath and one-based UTF-16 coordinates. Protocol positions are zero-based. Configuration uses the product's named command/args/extensionToLanguage map. Plugin management, hover stages and rename are outside the extension. Current IDs remain provisional; retired ls1/ir2/hv2 IDs are not reallocated.

Regenerate recorded output when prompts or cases change:

make record_lsp_fixtures

For live-model trials, configure CODECRAFTERS_SECRET_OPENROUTER_API_KEY and run make test_lsp_with_users_code. Live-model reliability, 100-run live flakiness checks and learner pacing trials remain release gates. Offline scripted checks do not establish those results.

Independent implementation checks

Install a pinned server outside the repository, then run:

npm install --prefix /tmp/lsp-pyright --ignore-scripts pyright@1.1.414
make test_lsp_with_pyright PYRIGHT_LANGSERVER=/tmp/lsp-pyright/node_modules/.bin/pyright-langserver
make test_lsp_with_claude_code CLAUDE_CODE_BINARY=/path/to/claude-2.1.266 PYRIGHT_LANGSERVER=/tmp/lsp-pyright/node_modules/.bin/pyright-langserver

The first check uses the supplied transport and reference integration with real Pyright. It waits for Pyright's project discovery before asserting workspace references, checks the target after an edited import, then observes the actual Price return-type error and diagnostic clearing. The repaired caller returns 500. The helper transports notifications; the reference agent implements freshness.

The second runs Claude Code 2.1.266 with a temporary plugin, Pyright and a local scripted Anthropic endpoint. It verifies definition, references, position conversion, includeDeclaration true, the same API migration, diagnostic delivery and repair. Bash stays available because this product version suppresses automatic diagnostic attachments without Bash or PowerShell. The scripted model never invokes a shell. The native check observes product diagnostic attachments separately from the learner fixture's structured Write results. Neither check uses paid model calls.

Versioned stale reports are rejected; unversioned reports are accepted in arrival order and cannot establish which edit was analyzed. Missing feedback is an error. The fixture publishes once per open/change except in its missing-feedback case; real servers determine publication timing. Cleared diagnostics do not prove behavioral correctness.

CI and repository conventions

make test runs the entire suite without Go's test cache, as Redis's tester does. make test_offline runs protocol checks, fixture comparisons, and unit tests. make test_live runs the existing TestStages scenarios against OpenRouter. Both groups run as separate CI jobs, so an invalid model credential cannot hide an offline protocol regression. No live tests are skipped or allowed to fail.

The base and Skills Python scenarios pin Python 3.14 through .python-version, matching the course starter. This prevents uv from selecting Python 3.15 when the locked native dependencies only support Python through 3.14.

The tester follows Redis's slug-to-Go-function registration, synchronized course YAML, scenario fixtures, reference implementation, teardown, and flakiness-check patterns. LSP uses a controlled stdio fixture for repeatable protocol assertions; the separate live-model and Pyright checks retain their documented scope.