Skip to content

fix(anthropic): capture extended-thinking reasoning tokens - #746

Merged
Luca Forstner (lforst) merged 3 commits into
mainfrom
fix/anthropic-direct-reasoning-tokens
Sep 7, 2026
Merged

Luca Forstner (lforst) merged 3 commits into
mainfrom
fix/anthropic-direct-reasoning-tokens

Conversation

@paultancre-bt

@paultancre-bt paultancre-bt commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Problem

The direct Anthropic integration (integrations/anthropic/) never records reasoning tokens. extract_anthropic_usage (_utils.py) only reads flat, top-level usage fields:

_ANTHROPIC_USAGE_METRIC_FIELDS = (
    ("input_tokens", "prompt_tokens"),
    ("output_tokens", "completion_tokens"),
    ("cache_read_input_tokens", "prompt_cached_tokens"),
    ("cache_creation_input_tokens", "prompt_cache_creation_tokens"),
)

But Anthropic reports extended-thinking usage nested, under usage.output_tokens_details.thinking_tokens (a subset of output_tokens). So completion_reasoning_tokens was always absent on thinking-model spans.

Flagged by the same customer as the sibling pydantic-ai bug ("the direct Anthropic integration records no reasoning tokens at all"). This is the follow-up called out in #745 / SDK-328.

Fix

Extract the nested output_tokens_details.thinking_tokens as the normalized completion_reasoning_tokens metric, matching the OpenAI/pydantic-ai integrations.

Verification

  • Cassette-backed regression test recorded against a real Anthropic thinking response (claude-haiku-4-5, thinking enabled), usage came back with output_tokens_details.thinking_tokens: 168, and the span metric now matches.
  • Red→green confirmed: without the fix the test raises KeyError: 'completion_reasoning_tokens'.
  • Existing usage/metrics/cache tests still pass (7/7).

Blast radius

  • Additive: one new metric read; no existing field/metric changes.
  • Gated on include_output, so the streaming input-only usage path (include_output=False) is unaffected — verified it stays absent there.
  • Degrades cleanly when output_tokens_details is absent: non-thinking requests don't return it, and older SDKs (e.g. anthropic 0.48.0) have no such field → skipped. Test is version-gated to latest, same pattern as the existing prompt-cache-TTL tests (no 0.48.0 cassette needed).

Relates to #745

paultancre-bt and others added 3 commits September 7, 2026 10:32
The direct Anthropic integration's `extract_anthropic_usage` read only flat
top-level usage fields, so it never surfaced extended-thinking usage. Anthropic
reports thinking tokens nested under `usage.output_tokens_details.thinking_tokens`
(a subset of `output_tokens`), so `completion_reasoning_tokens` was always absent
on thinking-model spans.

Extract the nested field as the normalized `completion_reasoning_tokens` metric,
matching the OpenAI/pydantic-ai integrations. Gated on `include_output` so the
streaming input-only usage path is unaffected, and degrades cleanly when the
field is absent (non-thinking requests, older SDKs).

Cassette-backed regression test recorded against a real thinking response.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lforst
Luca Forstner (lforst) merged commit 336ef4e into main Sep 7, 2026
83 checks passed
@lforst
Luca Forstner (lforst) deleted the fix/anthropic-direct-reasoning-tokens branch September 7, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants