Skip to content

feat(spx-gui): add Sentry tracing for SPX runtime and AI interactions - #3383

Closed
1034674309 wants to merge 6 commits into
goplus:devfrom
1034674309:feat/issue-2419-sentry-spx-runtime
Closed

1034674309 wants to merge 6 commits into
goplus:devfrom
1034674309:feat/issue-2419-sentry-spx-runtime

Conversation

@1034674309

@1034674309 1034674309 commented Jul 31, 2026

Copy link
Copy Markdown
Member

Summary

Integrate Sentry with the SPX runtime and the AI interaction lifecycle tracked by #2419.

The runtime reports actionable panics without parsing every normal game log. AI interactions get isolated Think and Archive traces with per-call propagation headers, transport/command spans, explicit terminal outcomes, and one terminal exception only when the documented flow actually stops.

The implementation follows tools/ai/SENTRY_PLAN.md; tools/ai/SENTRY_IMPLEMENTATION.md maps each contract item to its code and tests.

Runtime Sentry foundation

The branch already contains the SPX runtime error bridge used by the AI integration:

  • parse structured runtime panic records from the runner
  • report the original panic with source function/file/line/column context
  • defer Sentry SDK work until the WASM-to-JavaScript call returns, preserving Go WASM panic unwinding
  • deduplicate the generic runner/game error emitted for the same panic
  • avoid JSON parsing for ordinary game logs
  • keep Sentry failures isolated from the project runtime

Plan-by-plan verification

Slice 1: bridge and isolated Think trace

Each Player.Think creates a new Sentry root rather than inheriting the page/pageload trace:

ai.think
  ├─ ai.transport.attempt
  │    └─ http.server
  │         └─ ai.upstream
  └─ ai.command.execute

Implementation details:

  • startThink synchronously starts an independent root and returns Sentry-Trace / Baggage before the fetch.
  • All other bridge calls are deferred through setTimeout(0) so Sentry SDK work cannot interfere with Go WASM stack unwinding.
  • Every Think receives a unique ID. The parent bridge stores spans in Map<thinkId, state> rather than process-wide singleton variables.
  • Trace headers are stored in the Think context.Context; timeout child contexts inherit the correct headers.
  • The WASM transport reads propagation headers from the call context, so concurrent Players cannot overwrite each other's trace.
  • endThink is registered before startThink; bridge failures cannot leave interactionActive locked.
  • Root start is recovered defensively so AI behavior continues when the monitoring bridge is unavailable.
  • A per-run game_session_id groups Think and Archive roots from the same game session.

Transport attempts

Every Transport.Interact call creates ai.transport.attempt with:

  • turn_index
  • attempt
  • error status when the fetch fails
  • backend-provided backend_category, backend_reason, and request_id

The fetch carries only the current Think's trace headers. HTTP failures are decoded through typed errors rather than JavaScript string matching.

Command execution

Registered handlers create ai.command.execute covering argument population and handler execution.

  • tags: command_name, turn_index
  • Break finishes successfully and ends the Think successfully
  • ordinary handler errors mark the command span as an error, retain error_message, and continue to the next turn
  • invalid arguments terminate the Think as model_quality_failure / invalid_arguments
  • handler panics other than AbortThread terminate the Think as runtime_failure / handler_panic
  • unknown commands remain in history for the following backend ai.detail; they do not create a fake handler span

Slice 2: backend continuation

The transport propagation produces the complete tree with the backend HTTP transaction and provider span.

The companion backend PR is goplus/builder-backend#346. It provides:

  • ai.upstream and ai.archive.upstream
  • provider request_id
  • first_meaningful_delta_ms
  • bounded ai.detail input/output
  • structured failure JSON {category, reason, request_id}

This frontend copies backend classification only to attempt-level backend_* attributes. Backend categories are intentionally separate from frontend terminal categories.

Slice 3: Think terminal state

think() records one thinkFinish; one deferred endThink writes the root terminal state and optionally captures one exception.

Exit outcome Root status Exception
model stops after at least one command success ok no
handler returns Break success ok no
owner/game cancellation cancelled unchanged no
HTTP 403 with code=40301 quota_exhausted unchanged no
HTTP 429 retries exhausted rate_limited unchanged; attempts are errors no
timeout retries exhausted failure error one
other transport retries exhausted failure error one
no initial command failure error one
invalid command arguments failure error one
handler panic failure error one
20-turn limit failure error one

The root also records turn_count and attempt_count; failures include their frontend category/reason.

429 is intentionally not an Issue source. It can be frequent, so exhausted 429 flows use outcome=rate_limited and error attempt spans only.

Terminal exception payloads

Only outcome=failure produces AIThinkFailure.

Categories and reasons:

  • transport_failure / timeout
  • transport_failure / network
  • model_quality_failure / missing_initial_command
  • model_quality_failure / invalid_arguments
  • model_quality_failure / turn_limit
  • runtime_failure / handler_panic

Each exception includes game_session_id, category, and reason. Relevant failures add turn/attempt and command context. The current user message is retained, command name plus arguments are capped at 4 KiB, and error text is capped at 1 KiB.

Capture, root attributes, span finish, and Map.delete all happen in the same deferred bridge task, preventing duplicate terminal reports.

Independent Archive trace

Archive runs in its own owner-scoped spx.Go context and starts a separate root:

ai.archive
  └─ ai.archive.transport.attempt
       └─ http.server
            └─ ai.archive.upstream
  • Archive never reuses a Think header and never changes a Think outcome.
  • Every retry gets ai.archive.transport.attempt and backend classification attributes.
  • Success ends the root normally.
  • Cancellation and quota stop without an exception.
  • Non-cancellation retry exhaustion produces one AIArchiveFailure with archive_failure / retries_exhausted.
  • Backend empty_archive may be retried; only exhaustion of the complete frontend Archive flow creates an Issue.
  • Archive deliberately has no outcome attribute.

Slice 4: backend contract consumption

The frontend recognizes and propagates the backend contract supplied by goplus/builder-backend#346:

  • only 403 plus JSON code=40301 is quota exhaustion
  • other 403 responses remain ordinary failures
  • 429 remains TooManyRequestsError and observes Retry-After
  • backend category, reason, and request_id survive wrapped HTTP/timeout errors
  • quota stops Think/Archive retries immediately

Detailed provider/model reasons remain backend-owned and are copied verbatim to attempt spans. The frontend does not duplicate those reasons as Think exception categories.

Sampling

Independent ai.* roots use an app-provided aiSampleRate instead of inheriting page sampling.

  • VITE_SENTRY_AI_SAMPLE_RATE is exposed for Account and XBuilder apps.
  • The current default remains 1 for validation and initial rollout.
  • Choosing a lower production rate and adding backend route-specific sampling are the documented slice 5 follow-up, not part of this PR.

Explicitly out of scope

  • production sampling-rate tuning
  • sprite/Player name tags
  • backend exception capture
  • changing quota response fields
  • turning exhausted 429 into a Sentry Issue

Tests and checks

  • cd tools/ai && go test ./...
  • cd tools/ai && go vet ./...
  • cd spx-gui && pnpm run type-check
  • cd spx-gui && pnpm run lint
  • ESLint on all changed Vue/TypeScript files with zero warnings
  • Prettier check on all changed Vue/TypeScript files
  • tools/ispx/build.sh
  • git diff --check

Tests cover:

  • per-context trace-header inheritance and isolation
  • quota/429/timeout HTTP classification
  • one terminal endThink / endArchive
  • success, cancellation, quota, rate-limit, timeout, network, missing command, invalid arguments, panic, and turn-limit outcomes
  • ordinary handler errors and unknown commands continuing the Think
  • Archive success, cancellation, quota, rate-limit, and retry exhaustion
  • exception eligibility and payload truncation

Manual validation

A successful AI-Town Think was verified end-to-end as an independent trace connected through the backend HTTP transaction. Failure, exhausted 429, and empty-archive behavior are covered by unit tests; they have not all been forced in a live game session.

Dependencies

Closes #2419.

Add Sentry integration to ProjectRunner to capture and report runtime
errors from tools/ispx with structured context including file, line,
function, and column information.

Key features:
- Parse panic JSON logs from tools/ispx console output
- Report runtime panics to Sentry with SpxRuntimePanic exception type
- Add structured context (file, line, column, function) to events
- Instrument lifecycle spans for engine init, build, start, and stop
- Deduplicate errors within 5-second window to prevent duplicate reports
- Graceful degradation when Sentry is unavailable
- Async reporting to avoid interfering with WASM panic unwinding

Closes goplus#2419

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@1034674309 1034674309 changed the title feat(spx-gui): integrate Sentry monitoring for SPX runtime panics feat(spx-gui): report SPX runtime panics and lifecycle spans to Sentry Aug 2, 2026
@1034674309
1034674309 marked this pull request as ready for review August 3, 2026 01:53

@fennoai fennoai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

This change adds Sentry tracing spans around SPX runner lifecycle operations and structured SPX runtime-panic reporting. The overall structure is clean, parseRuntimePanic is strict and defensive at the iframe trust boundary, and the WASM-unwinding deferral (setTimeout(0)) is a subtle correctness detail handled well. The four review passes surfaced a few concrete concerns worth addressing, noted inline. No blocking issues; submitting as a non-approving comment review.

Highlights:

  • Performance: JSON.parse runs on every single-string console.log from the game loop — worth a cheap prefilter.
  • Correctness: grace-period (1000ms) vs recent-panic window (5000ms) mismatch, and panic state is not reset on stop(), so a stale panic can suppress a legitimate error in a subsequent run.
  • Security (low): attacker-controlled panic strings reach Sentry unbounded.


let value: unknown
try {
value = JSON.parse(args[0])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance: parseRuntimePanic runs on every single-string console.log forwarded from the game iframe, and the only guard before JSON.parse is the arity/typeof check on line 244. A game doing console.log("score: 42") once per frame will trigger a JSON.parse + throw/catch at up to 60x/sec. Consider a cheap prefilter before parsing, e.g. skip unless the string starts with { and contains "panic" — the payload requires record.msg === 'panic' anyway, so this is behavior-preserving.


function hasRecentRuntimePanic() {
const panic = lastRuntimePanic
return panic != null && Date.now() - lastRuntimePanicAt <= 5000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Magic-number mismatch / correctness: this bare 5000 recent-panic window is inconsistent with runtimePanicGracePeriod = 1000 (line 185). The deferral only waits 1s for a future panic, but a panic that arrived up to 5s earlier suppresses the generic capture. Recommend extracting both into named constants and documenting why they differ (or unifying them). A raw literal here is easy to drift out of sync with the grace period.

}

function deferRuntimeErrorCapture(err: unknown, ctx: string) {
if (hasRecentRuntimePanic()) return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-run stale-panic suppression: lastRuntimePanic/lastRuntimePanicAt are reset only at the start of runInternal (lines 429-430), not in stop(). During rerun() (which calls stop() then run()), and for up to 5s after any run ends, hasRecentRuntimePanic() can still return true from a previous run's panic. That would cause this early return to silently drop a legitimately different error from the new run. Consider resetting panic state on stop() as well, or keying "recent" to the current run/AbortController.

line,
column
})
Sentry.captureException(exception)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security (low) — unbounded attacker-controlled data to Sentry: error, functionName, and file originate from the iframe's console.log (user/game WASM) and are forwarded verbatim into Sentry as the exception message, name, and context with no length cap or sanitization. A malicious project can emit arbitrary/large panic strings (log-poisoning) and could embed runtime-harvested data into your Sentry stream. Consider truncating these strings to a bounded length and confirming Sentry PII scrubbing covers this path.

// eslint-disable-next-line no-console
console.log(...args)
const panic = parseRuntimePanic(args)
if (panic != null) handleRuntimePanic(panic.error, panic.functionName, panic.file, panic.line, panic.column)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: parseRuntimePanic already returns a RuntimePanic object, but it's destructured into 5 positional args here only for handleRuntimePanic to immediately reassemble the identical object (line 215). Passing panic directly (handleRuntimePanic(panic)) is simpler and removes a class of positional-arg ordering bugs.

})
try {
const result = await operation()
span.setStatus({ code: 1 })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor (readability): span.setStatus({ code: 1 }) / { code: 2 } use bare numeric codes for OK/ERROR. A short comment or named constants (// 1 = OK, 2 = ERROR) would match how logLevels is spelled out elsewhere in this file.

田宇琨 added 3 commits August 11, 2026 17:05
Create isolated Think and Archive traces, propagate per-call trace headers, record transport and command spans, and report terminal outcomes with bounded exception context.

Consume structured backend failure metadata while keeping quota, cancellation, and exhausted 429 flows out of Sentry Issues.
@1034674309 1034674309 changed the title feat(spx-gui): report SPX runtime panics and lifecycle spans to Sentry feat(spx-gui): add Sentry tracing for SPX runtime and AI interactions Aug 23, 2026
@1034674309

Copy link
Copy Markdown
Member Author

This PR is superseded by split PRs so runtime monitoring and AI tracing can be reviewed independently:

SPX runtime panic monitoring is not part of the AI Sentry contract. It is left on this old branch and can land later as its own PR if still wanted.

Closing this PR. Please review #3470 instead.

@1034674309 1034674309 closed this Aug 26, 2026

This branch was successfully deployed

1 active deployment
Preview – builder c103a09a Deployed Aug 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Sentry integration to tools/ispx

1 participant