I've been reading src/art/trajectories/__init__.py — Trajectory (messages_and_choices, reward: float, metrics: dict[str, float | int | bool]) and TrajectoryGroup (grouping multiple rollouts of the same task, with its own metrics). That's a clean shape for "one rollout's outcome plus its scalar reward and side metrics" — closer to an eval result than most RL trajectory formats I've seen, since metrics already separates named auxiliary signals from the training-facing reward.
I maintain EvalPort, a JSON-Schema-based interchange spec (TestCase/Suite/Grader/Result/ResultSet) for portable LLM eval data, so eval/grading data isn't locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).
The mapping here is fairly direct: Trajectory.reward → EvalPort GraderResult.score, Trajectory.metrics → additional named GraderResults (the same "multiple named signals per outcome" pattern EvalPort's schema is built around), messages_and_choices → the Result transcript, and a TrajectoryGroup (multiple rollouts of one task, as used for GRPO's relative comparisons) → an EvalPort ResultSet grouped by task. That would let an ART training/eval run's rollouts be read by grading tooling built for other frameworks, or let a suite of tasks authored as portable EvalPort TestCases drive ART rollouts instead of a one-off rollout.py per project (as in examples/tic_tac_toe/rollout.py, examples/mcp-rl/, etc.).
I'd propose a standalone, optional adapter doing that conversion in both directions. No required dependency, no change to Trajectory/TrajectoryGroup or the training loop.
Happy to build this as a PR into ART (e.g. src/art/interop/evalport.py), or as a standalone package in EvalPort's own adapters/ directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
— Sahi, independent contributor (not affiliated with OpenPipe)
I've been reading
src/art/trajectories/__init__.py—Trajectory(messages_and_choices,reward: float,metrics: dict[str, float | int | bool]) andTrajectoryGroup(grouping multiple rollouts of the same task, with its ownmetrics). That's a clean shape for "one rollout's outcome plus its scalar reward and side metrics" — closer to an eval result than most RL trajectory formats I've seen, sincemetricsalready separates named auxiliary signals from the training-facingreward.I maintain EvalPort, a JSON-Schema-based interchange spec (
TestCase/Suite/Grader/Result/ResultSet) for portable LLM eval data, so eval/grading data isn't locked to one framework's format. It's early-stage (~35 shipped adapters, no notable star count — being upfront about that).The mapping here is fairly direct:
Trajectory.reward→ EvalPortGraderResult.score,Trajectory.metrics→ additional namedGraderResults (the same "multiple named signals per outcome" pattern EvalPort's schema is built around),messages_and_choices→ theResulttranscript, and aTrajectoryGroup(multiple rollouts of one task, as used for GRPO's relative comparisons) → an EvalPortResultSetgrouped by task. That would let an ART training/eval run's rollouts be read by grading tooling built for other frameworks, or let a suite of tasks authored as portable EvalPortTestCases drive ART rollouts instead of a one-offrollout.pyper project (as inexamples/tic_tac_toe/rollout.py,examples/mcp-rl/, etc.).I'd propose a standalone, optional adapter doing that conversion in both directions. No required dependency, no change to
Trajectory/TrajectoryGroupor the training loop.Happy to build this as a PR into ART (e.g.
src/art/interop/evalport.py), or as a standalone package in EvalPort's ownadapters/directory with zero footprint on this repo — whichever you'd prefer. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md— Sahi, independent contributor (not affiliated with OpenPipe)