Skip to content

Trace execution paths through pull-request safe outputs - #78680

Open
pelikhan wants to merge 4 commits into
mainfrom
copilot/update-create-pull-request-logging
Open

pelikhan wants to merge 4 commits into
mainfrom
copilot/update-create-pull-request-logging

Conversation

@pelikhan

@pelikhan pelikhan commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Add compact execution-path identifiers to create-pull-request and push-to-pull-request-branch, enabling downstream operators to identify hot paths and problematic recovery routes in unified sessions.

  • Branch instrumentation
    • Assign 49 unique, source-documented letters per handler; append markers in execution order.
    • Add terminal markers for successful returns, unsuccessful/skipped returns, and thrown errors.
  • Reusable tracing
    • Isolate state per invocation, preserve result/error identity, and cap identifiers at 1,024 characters.
  • Session persistence
    • Retain execution_trace across successful, failed, skipped, staged, and retry outcomes.
    • Extend the unified-session schema and document interpretation.

Example outcome fields:

{
  "type": "push_to_pull_request_branch",
  "status": "success",
  "execution_trace": "AMTdefnux"
}

Identifiers describe paths, not invocations; interpret them against the operation type and handler source revision.

pelikhan and others added 2 commits November 17, 2026 08:35
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
@pelikhan pelikhan changed the title Trace pull-request safe-output execution paths Trace execution paths through pull-request safe outputs Nov 17, 2026
@pelikhan
pelikhan marked this pull request as ready for review November 17, 2026 09:36
Copilot AI adaptive review requested due to automatic review settings November 17, 2026 09:36
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~14 AIC · ⌖ ~2 AIC · ⊞ 6.6K · ◷

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated by Ponytail Reviewer for #78680 · codex · routed: gpt61 medium · ~8 AIC · ⌖ ~1 AIC · ⊞ 185 · ◷

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt56 · ~7 AIC · ⌖ ~2 AIC · ⊞ 1.5K · ◷

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output. Detected threat types: prompt injection.

Review the workflow run logs for details.

PR Specification Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated by PR Specification Reviewer for #78680 · copilot · gpt56 · ~8 AIC · ⌖ ~2 AIC · ⊞ 1.6K · ◷

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

🧪 Test quality analysis by Test Quality Sentinel · copilot · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ⊞ 7.2K · ◷

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output. Detected threat types: prompt injection.

Review the workflow run logs for details.

Overview

Reviewed PR head 430c17272cf34e6e6e23ad5cb6da77149d379fbf against the Unified Agent Session Specification (documented version 2.10.0; affected runtime-vocabulary contract T-UAS-061, with preservation requirements T-UAS-027/T-UAS-029 and publication safety T-UAS-049).

Finding

The new data.execution_trace contract is not synchronized with the specification metadata: frontmatter/document metadata and the change log remain at 2.10.0, while the status section still says 2.8.0. Please update the version/date consistently, correct the stale status reference, and add a focused change-log entry for the bounded/redacted trace semantics. An inline comment marks the changed contract paragraph.

Coverage

Static review only; no tests or conformance scripts were run, per review contract.

Generated by PR Specification Reviewer for #78680 · copilot · gpt50 · ~8 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review-specs to run again

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-17T09:45:00Z
review_event: COMMENT
top_themes: [trace path accuracy, configuration-only markers]
files_reviewed: [actions/setup/js/create_pull_request.cjs, actions/setup/js/push_to_pull_request_branch.cjs, actions/setup/js/safe_output_trace.cjs, actions/setup/js/safe_output_outcomes.cjs, actions/setup/js/safe_output_handler_manager.cjs]
comment_count: 2
History

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~7 AIC · ⌖ ~2 AIC · ⊞ 1.5K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two actionable trace-accuracy issues: both handlers record the configured no-change mode at invocation start, so normal non-empty operations receive markers for branches they never take. Inline comments explain the impact and suggested placement. No blocking correctness or security issues found; this was a static diff review and tests were not run.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~7 AIC · ⌖ ~2 AIC · ⊞ 1.5K
Comment /review to run again

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (spec-review-history)
{"id":"ldg-084047ff-7a70-4b02-ae58-7d01eb21f321","parents":[],"payload":{"id":"ldg-d8951b92-d108-4c24-8db5-1cc2a1757f63","operation":"append","value":{"finding_count":1,"head_sha":"430c17272cf34e6e6e23ad5cb6da77149d379fbf","outcome":"updated"}},"sha":"sha256:50bf69d10f911f0c7c60a3b48c7e7fa470bf21179674a6c33e84a8c7f804a48e","timestamp":"2026-11-17T09:46:06.320Z","type":"ledger_append","version":1}
History

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (review-history)
{"id":"ldg-6b7ed1c0-250e-4a73-8ec2-648d9e4b99f9","parents":[],"payload":{"id":"ldg-0602ea42-891d-4ea9-ad06-c0b6d2f37016","operation":"append","value":{"finding_count":2,"head_sha":"430c17272cf34e6e6e23ad5cb6da77149d379fbf","outcome":"findings"}},"sha":"sha256:9a01cc640a3c4f565c28acec4b51cd2f9e168f1352a1a2c07c7734fddcb7ea5e","timestamp":"2026-11-17T09:47:02.180Z","type":"ledger_append","version":1}
History

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd and /codebase-design. Requesting changes because the execution trace can misreport the route a run actually took.

📋 Key Themes & Highlights

Key Themes

  • Trace accuracy: v/w (push) and Q (create PR) are recorded at handler entry from configuration, so they can mark routes the run never took.
  • Map drift: push handler s is labelled "unchanged checkout" in the map, but the code marks the empty-patch skip.
  • Result leak: execution_trace is enumerable and survives the result spreads in safe_output_handler_manager.cjs.
  • Test quality: one wiring assertion checks source text rather than behaviour.

Positive Highlights

  • ✅ Clean per-invocation closure, so overlapping calls keep private traces.
  • ✅ The wrapper order (trace outside transport integrity) lets the outer layer see converted failures.
  • ✅ Trace letters are unique within each handler, and the map comments match the markers.
  • ✅ Marker validation and the 1024-character bound are enforced both in the wrapper and in outcome projection.

Not Verified

The vitest suites, npm run typecheck, and npm run schema:session:check could not run in this environment: npm ci failed on the package proxy TLS certificate, and the schema generator needs ts-json-schema-generator. The schema JSON parses and the new module passes a syntax check. Please run the test and schema gates in CI before merging.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~14 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

Comment thread actions/setup/js/safe_output_trace.cjs Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Appending tracing after variable handler arguments can place temporaryIdMap in the trace callback’s position, causing handlers invoked by processMessages to throw.

2 open issues
What changed in this PR

Adds compact execution-path tracing to gh-aw’s pull-request safe outputs so operators can analyze handler routes in unified sessions.

Changes:

  • Instruments pull-request creation and branch pushes with branch and terminal-outcome markers.
  • Adds invocation-local tracing capped at 1,024 characters.
  • Persists traces across outcome types and retries, with schema, documentation, and test updates.
File Description
docs/​src/​content/​docs/​specs/​unified-agent-session-specification.md Documents trace format, persistence, and revision-aware interpretation.
docs/​public/​schemas/​unified-session.schema.json Defines and validates execution_trace fields.
actions/​setup/​js/​types/​unified_session.d.ts Adds the optional typed trace field.
actions/​setup/​js/​safe_output_trace.test.cjs Tests isolation, bounds, identity preservation, wrappers, and marker uniqueness.
actions/​setup/​js/​safe_output_trace.cjs Implements bounded tracing and result/error attachment.
actions/​setup/​js/​safe_output_outcomes.test.cjs Tests trace persistence, validation, and retry outcomes.
actions/​setup/​js/​safe_output_outcomes.cjs Includes validated traces in projected outcomes.
actions/​setup/​js/​safe_output_handler_manager.cjs Retains traces from returned failures, exceptions, and retries.
actions/​setup/​js/​push_to_pull_request_branch.test.cjs Adds expected traces for push and recovery scenarios.
actions/​setup/​js/​push_to_pull_request_branch.cjs Instruments branch-push routes and documents marker meanings.
actions/​setup/​js/​create_pull_request.test.cjs Adds expected traces for creation and fallback scenarios.
actions/​setup/​js/​create_pull_request.cjs Instruments pull-request creation routes and documents marker meanings.

Review effort: Max (adaptive)

Comment thread actions/setup/js/safe_output_trace.cjs Resolved
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "ea0dd79daa6dac3addb218299a801fdcef7221f6",
  "kind": "requested",
  "schema_version": "1",
  "state": "361e48e91a4940e10856b6c33fb407db0b2bafa2b62052fddd80b0486002d0cb",
  "work": [],
  "workflow": "pr-sous-chef"
}

Generated by 👨🍳 PR Sous Chef · pi · routed: haiku55 xhigh · ~3 AIC · ⌖ ~2 AIC · ⊞ 7K · ◷
Comment /souschef to run again

…-pull-request-logging

# Conflicts:
#	actions/setup/js/safe_output_handler_manager.cjs

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Copilot AI commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

@copilot

🍳 A little prep to help this PR along! Copilot assignment requested for the following outstanding work:...

Merged the latest main and resolved the safe-output execution rename integration in commit 9355aa4b36. make merge-main and final progress validation passed; 413 focused tests passed (3 skipped).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants