Skip to content

feat: summarize agent run logs in harness status - #68

Open
Samyra312007 wants to merge 4 commits into
rxdt:mainfrom
Samyra312007:feat/run-log-summaries
Open

Samyra312007 wants to merge 4 commits into
rxdt:mainfrom
Samyra312007:feat/run-log-summaries

Conversation

@Samyra312007

Copy link
Copy Markdown

Closes #9 harness status no longer just counts log files. It reads back the JSONL that harness run already writes under scratchpad/runs/ and renders a newest-first table of finished runs: iterations, tokens in/out with the cache split, cost, duration, and why the run stopped. --verbose adds each agent's last message and the log path. run_worker is untouched; this feature is strictly read-only.

Decision: our logs (Option A), not the agent's native session logs

The issue asks for the call to be made and argued in the PR. Option A scratchpad/runs/ , because:

  1. Repo boundary. LoopGate's own rules keep agents inside the repo; a status command that goes poking through ~/.codex/ and ~/.claude/projects/ would break that containment on principle.
  2. The native logs often don't exist. The claude preset runs with --no-session-persistence , so there is frequently no ~/.claude/... session file to read. The ralph log is the only record of those runs.
  3. Ralph's framing lines exist only in our logs and are the one thing every agent shares: iteration count and wall-clock span, whatever ran inside.

Codex's native logs would have been a duplicate of what we already captured, so there was nothing to gain.

Before / after

Before, the only read-back was a file count:

7 run log(s) in /repo/scratchpad/runs
newest:
/repo/scratchpad/runs/20260902/claude/0004.jsonl

Now ( harness status ):

DATE AGENT RUN ITERS MIN TOKENS IN TOKENS OUT CACHE R CACHE W COST MS STOP
2026-09-02T05:38 codex 0003 1 1 15376 142 9600 0 - - -
2026-09-02T06:34 claude 0003 1 0 2 13 5903 13757 $0.14 1800 end_turn

harness status --verbose appends a second table with the agent's own last words and the file to open:

                                                         Last messages

DATE AGENT RUN LAST MESSAGE LOG
2026-09-02T05:38 codex 0003 Hi — codex-0003 here. No edits made. No big issues .../20260902/codex/0003.jsonl
2026-09-02T06:34 claude 0003 I don't have access to my session ID. .../20260902/claude/0003.jsonl

An empty repo keeps the old one-line receipt, 0 run log(s) in ... .

How it reads the logs

All parsing lives in a new module, harness/runs.py (~190 lines; cli.py only got its status body rewritten) — per the challenge in the issue, since cli.py is at its complexity cap.

  • Find the line by its type , never by position. Claude's totals live in type: "result" , Codex's in type: "turn.completed" ; both are searched for by name, in that order, wherever they sit in the file.
  • Same facts, different words, mapped onto shared columns: usage.cache_read_input_tokens ↔ usage.cached_input_tokens , usage.cache_creation_input_tokens ↔ usage.cache_write_input_tokens ; total_cost_usd , stop_reason , and duration_ms are Claude-only and stay - for Codex; the last message is result for Claude or the last item.completed with item.type == "agent_message" for Codex.
  • Ralph's two framing lines give iterations and elapsed minutes for any agent, including ones that log nothing else. A run with no end line yet reports the iteration it started.
  • A dash, not a zero. Facts an agent never reported render as - (see the Codex row above). A wrong number is worse than a blank this is what aider/crush-style logs (Add more agents to [tool.harness.agents] #49) will get.
  • Defensive by construction: unparseable lines, JSON lines that aren't objects, unreadable files, and garbage timestamps are skipped, never fatal one weird log cannot crash status . Totals must be finite numbers; hypothesis caught that unicode-digit junk strings parse as float (a naive float() would have printed $nan ).
  • Ordering stays exactly what the old status did: newest-first by mtime.

What this deliberately does not do

  • No agy/copilot parsing yet there are no sample logs of either shape to design against. The reader already shows their rows correctly as dashes; when someone runs one and pastes a log (the issue notes that's a useful contribution on its own), the mapping is a small extension in runs.py .
  • No JSON output, no row cap, no log writing/pruning. Small and flat; extensions are cheap.

Testing

No test runs an agent or spends a cent, per the issue:

  • The two logs pasted in the issue are committed byte-for-byte as fixtures: harness/tests/logs/claude/0003.jsonl and harness/tests/logs/codex/0003.jsonl .
  • New harness/tests/test_runs.py : both real samples map onto identical expected facts (proving the cross-agent mapping), plus unit tests for partial/corrupt/foreign logs, ralph-span edge cases, and hypothesis property tests (arbitrary text, numbers, timestamps rendering never crashes, never invents).
  • The two test_cli.py status tests were updated to the new contract; everything else untouched.
  • Full suite: 283 passed, 100% coverage ( pytest -p no:cacheprovider -n auto --cov --cov-fail-under=100 ). ruff format/check clean on all changed files.

Notes

  • MIN comes from ralph's minute-resolution framing timestamps (rounded), so a 40-second run shows 0 that's the log's own resolution, not a bug. Claude's precise duration_ms sits in the MS column.
  • DATE is the ralph start timestamp only; nothing is inferred from directory names.
  • Mutmut has not yet been run against the new module (tracked in docs/PROJECT_STATUS.md next steps), and the project gate ( harness gate ) should be run from the package env before merge.

- add harness/runs.py: a read-only parser that maps claude's result line and codex's turn.completed line onto one shared row shape, with ralph's framing lines giving iterations and elapsed minutes for any agent; lines are found by type, never position
- harness status now renders a newest-first table (date, agent, run, iterations, minutes, tokens in/out, cache read/write, cost, duration, stop reason); --verbose adds each agent's last message and log path; a fact an agent never reports shows as a dash, never a zero
- parse defensively: unparseable lines, non-object lines, and unreadable files are skipped, so one weird log cannot crash status; totals must be finite numbers (hypothesis caught unicode-digit junk parsing as float)
- commit real claude/codex sample logs as fixtures under harness/tests/logs and cover the reader with unit and hypothesis property tests in harness/tests/test_runs.py; run_worker is untouched
@rxdt rxdt added enhancement New feature or request hacktoberfest Open for Hacktoberfest contributions labels Sep 17, 2026

rxdt commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Local docs-only commit d832ee6 is ready: docs/PROJECT_STATUS.md and docs/specs/base.md exactly match origin/main. Normal push was blocked by the mandatory pre-push gate after xdist/environment test_cli.py failures (96.1% coverage); remote PR head remains f646f01. No bypass was used.

rxdt commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Normal push to git@github.com:Samyra312007/loopgate_harness.git was attempted; mandatory pre-push gate rejected it. 4 test_cli.py failures under xdist/environment (template temp.pyproject missing, uv not found, plus coverage 96.1%), so remote head remains f646f01; no bypass used.

rxdt commented Sep 17, 2026

Copy link
Copy Markdown
Owner

GitHub API blob writes are forbidden (403 Resource not accessible by integration), so cannot update contributor fork via connector. PR #68 currently still shows docs edits and is clean/mergeable at old head.

Per maintainer request in PR rxdt#68 — the feature docs under docs/
belong to the upstream repo's workflow, not the contributor fork.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@Samyra312007

Copy link
Copy Markdown
Author

ca65ae2 on the PR head resets docs/PROJECT_STATUS.md and docs/specs/base.md to match main byte-for-byte (same blob SHAs as origin/main ), replacing the local d832ee6 which is unreachable to anyone without your clone. The full pre-push gate is green on this head: test, coverage, ruff ×2, pylint, types, security, complexity, audit all PASSED. For the record, the xdist/coverage failures you hit reproduce only when the gate runs without the venv activated ( .venv/bin missing from PATH breaks tool resolution in subprocesses) worth a note in the README if others hit it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request hacktoberfest Open for Hacktoberfest contributions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

harness status should summarize finished runs, not just count log files

2 participants