Repository navigation
matthias/eval-gates-trail-show - #2988
Draft
matthiaswenz wants to merge 3 commits into
Draft
matthiaswenz wants to merge 3 commits into
matthiaswenz wants to merge 3 commits into
Conversation
entire-api now reports gates on an agent's eval (gate_type "eval"). The
gate key is the agent id, so the old one-row-per-gate view printed a
ULID with no reading or threshold.
trail show now groups eval gates under "Evals:", named by the agent,
with the reading against the threshold ("72% must be ≤ 40%") and
whether it passed, blocks the merge, or needs attention. A gate skipped
for a configuration issue prints "not enforced: <reason>". Non-blocking
gates that failed or errored, of any type, are listed under "Needs
attention:". --json already passes each gate's value through untouched.
trail merge keeps what blocks unchanged. Before it merges, it prints the
gates that need attention and the eval gates that are not enforced as
warnings on stderr, so --json keeps them too. A blocking eval gate is
listed by its rationale, which names the agent, instead of its id.
Part of https://entire.com/gh/entirehq/-/trails/19610 (design:
https://entire.com/et/entirehq/-/trails/620).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Entire-Checkpoint: 01M67M72S4GBC0ETT6VMR2K4WQ
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
trail show listed a disabled eval gate with its last result, so a gate turned off while its agent was paused still read "not enforced", and trail merge warned about it. A result for an earlier commit printed its old verdict, also as a merge blocker. Disabled eval gates are now left out, an outdated one says it isn't judged on this commit yet, and a gate without an agent name is named "agent <id>". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
entire-api now stores the answer a yes/no eval gate requires as its
threshold ("yes" or "no"), and the agent's polarity no longer decides it.
trail show printed the requirement from polarity; it now prints the
threshold, and nothing when there is none.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This draft pull request was opened by Entire to back a trail. Feel free to edit the title or body, but keep the hidden marker below: Entire uses it to close this pull request when the trail closes.
Note
Medium Risk
Changes the merge command's user-visible behavior (stderr warnings) and gate messaging on a sensitive workflow, though blocking rules are unchanged.
Overview
Adds first-class support for agent eval merge gates in the CLI mergeability snapshot.
entire trail show(text) now splits eval gates under an Evals: section with agent names, readings vs thresholds (percent/size/yes-no), and labels for passed, blocks merge, needs attention, or not enforced. A Needs attention: list summarizes non-blocking failed/errored gates.trail show --jsonstill passes each gate'svaluethrough unchanged; new API types document the eval shape (agent_name, threshold,configuration_issue, etc.).entire trail mergereuses the same readiness view, names eval blockers by agent rationale, and warns on stderr (including with--json) right before a successful merge when it proceeds past advisory failures or skipped/not-enforced eval gates—without changing what counts as blocking. Agent-help text for projecttrail showis updated to describe eval gates.Coverage is mostly new unit/integration tests around rendering, JSON passthrough, merge warnings, and blocking eval behavior.
Reviewed by Cursor Bugbot for commit cc99df3. Configure here.