feat(tier3): rebuild ATIF from OpenCode and Codex agent logs - #119
feat(tier3): rebuild ATIF from OpenCode and Codex agent logs#119mimran-khan wants to merge 5 commits into
Conversation
Parse OpenCode JSON stream events into synthetic trajectories when trajectory.json is empty, and reuse the parser for structured Codex tee logs. Fail closed on non-dict tool inputs. Fixes NVIDIA#118 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
| if not lines: | ||
| return None | ||
| if all(line.startswith("{") and line.endswith("}") for line in lines): | ||
| synth = synthetic_trajectory_from_opencode_json(text) |
There was a problem hiding this comment.
[P1] codex exec --json does not emit OpenCode tool_use events. A successful Codex CLI 0.142.5 run emitted thread.started, turn.started, item.completed with item.type="agent_message", and turn.completed; passing those JSONL records here returns None. Harbor also tees 2>&1, so one warning line prevents this branch entirely. Please parse Codex's own item events, tolerate non-JSON stderr, and add a fixture captured from real Codex stdout instead of reusing the OpenCode schema.
There was a problem hiding this comment.
Fixed: added a Codex exec --json parser for item.completed / agent_message events, stderr-tolerant JSONL scanning, and fixture tests. OpenCode-shaped JSONL still works as a fallback.
There was a problem hiding this comment.
This is still not fixed at the current head. Codex exec JSON emits top-level item.completed events; agent messages use item.text, and shell calls are separate command_execution items with command and aggregated_output fields. Feeding a successful stream in that shape still returns no synthetic trajectory, so an empty trajectory file collapses to the no-reconstructible-log result. Please parse the published ThreadEvent schema and cover it with a captured CLI fixture.
There was a problem hiding this comment.
This should be fixed now. synthetic_trajectory_from_codex_json handles top-level item.completed events, reads agent_message from item.text, and maps command_execution items with command plus aggregated_output into bash tool calls. Added fixture coverage for that Codex thread shape.
Codex tee logs use type=item / agent_message events, not OpenCode tool_use. Add a dedicated parser with stderr-tolerant JSONL scanning, keep OpenCode-shaped JSONL as a fallback, and normalize shell calls to bash for skill checks. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
|
Codex parser fixes are pushed from the earlier round. Ready for re-review when you have time. |
| meta["note"] = "Synthetic ATIF from Cline CLI JSONL log" | ||
| return synth, meta | ||
|
|
||
| opencode_path = logs / "opencode.txt" |
There was a problem hiding this comment.
[P2] This fallback is not reachable from eval-dataset refinement with collected OpenCode results. Trajectory discovery never includes opencode in agent_priority, so a persisted opencode/with-skill/trials//opencode.txt run is skipped wholesale. I reproduced an empty discovery result for that layout while the same fixture under a listed agent is found. Please add OpenCode to trajectory discovery and cover the public refinement path with an integration test.
There was a problem hiding this comment.
opencode is in agent_priority now, and I added a regression that discovers trajectories from opencode.txt when trajectory.json is missing.
There was a problem hiding this comment.
opencode is in agent_priority now, and I added a regression that discovers trajectories from opencode.txt when trajectory.json is missing.
rng1995
left a comment
rng1995
left a comment
There was a problem hiding this comment.
The current head still does not reconstruct real Codex exec events, and the new OpenCode fallback is not connected to the collected-results discovery path. Focused converter and refinement tests passed (95 tests), with Ruff and diff checks clean. Requesting changes for the two reproducible integration gaps noted in the review threads.
…ries Merge main and handle Codex exec JSON item.completed events with agent_message text and command_execution items. Include opencode in trajectory discovery so refinement can rebuild ATIF from opencode.txt when trajectory.json is missing.
…an-khan/SkillEvaluator into feat/opencode-atif-fallback
3 participants
Fixes #118
When Harbor leaves
trajectory.jsonempty, Tier 3 scoring used to fail with "No trajectory or reconstructible agent log" even though OpenCode and Codex tee structured JSON toopencode.txtandcodex.txt.This adds parsers for OpenCode
run --format=jsonevents (text + completedtool_userecords) and wires them intoload_trajectory_with_fallbackafter the existing claude/cursor/cline fallbacks. Codex reuses the same parser when every tee line is JSON. Non-dict tool inputs are ignored so raw strings are not treated as shell commands.Harbor task bundles already copy
log_converters.pythroughadapter.py, so generated verifiers pick this up automatically.Distinct from #110, which is about exec steps inside an existing trajectory.
Test plan
pytest tests/tier3/test_log_converters.pypytest tests/test_harbor_collector_runtime_failures.py