feat(trace): add JSONL timing traces to helix evolve - #50
Open
KE7 wants to merge 3 commits into
Open
Conversation
This was referenced Aug 10, 2026
helix evolve --trace: JSONL event stream with timestamps for the agent-vs-evaluator time split
KE7
force-pushed
the
feat/trace-jsonl-timing
branch
2 times, most recently
from
August 23, 2026 21:58
54e3741 to
7aae6b3
Compare
Add `helix.trace.traced(span)`: a signature-preserving decorator that
appends a `start`/`end` record pair (span id, wall/monotonic clocks,
duration, thread id, outcome, error class, cheap identity attrs) to a
JSONL sink opened once by `trace.enable(path)`. When tracing is off the
wrapper is a single global check. The `finally` classifies any
BaseException, writes the end record, and re-raises untouched; a failing
sink is reported once via `logger.error("Trace unavailable: ...")` and
never masks the exception in flight.
Decorate the five functions whose whole body is the span we want:
`run_evaluator` (evaluate), `_run_full_val_eval` (validate),
`_run_proposal_worker` (proposal), `invoke_claude_code` (agent),
`generate_seed` (seed) — plus `run_evolution` (run) so the last line of a
complete trace is always the run's end record. No function is split and
no signature changes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4
Parse the path, call `trace.enable()` once before `run_evolution`, and exit 2 with a clear message when the file cannot be opened. `HELIX_TRACE=PATH` is honoured as the environment fallback. Document the event schema, the spans, how a truncated trace is recognised, and a jq line for per-generation wall time. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4
Decorator: args/kwargs/return passthrough, start/end pairing, error outcome with class name only, a broken sink never masking a KeyboardInterrupt in flight, one well-formed line per record under a thread pool, disabled-mode no-op, explicit and registry attrs extraction. `enable`: env fallback, unopenable path, per-record flush. CLI: `--trace` on evolve and resume enables the sink before evolution. End to end: a real `run_evolution` through the evaluator/mutator override hooks emits run, validate, proposal, evaluate, and agent spans and ends with the run's end record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4
KE7
force-pushed
the
feat/trace-jsonl-timing
branch
from
September 10, 2026 00:41
93ef401 to
86552b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
helix evolve --trace PATHandhelix resume --trace PATH(orHELIX_TRACE=PATH) append a JSON Lines timing trace of the run, so you can see where the wall clock went: proposals vs. sequential validation, evaluator vs. agent time, per generation.Design: one decorator, no splits
The whole feature is
helix.trace.traced(span), afunctools.wrapsdecorator applied to functions that already exist onmainand whose whole body is exactly the span we want:runrun_evolutionendis always the last lineproposal_run_proposal_workerevaluaterun_evaluatorvalidate_run_full_val_evalagentinvoke_claude_codeproposal, seed generation insideseed, otherwise a merge)seedgenerate_seedWhy this shape:
executor.py,evolution.py,mutator.py(merger.pyuntouched): three import edits and six@traced("...")lines. Nothing to keep in sync when a signature changes.finallyclassifies anyBaseException(KeyboardInterruptincluded), writes theendrecord, and re-raises untouched. A failing sink is reported once vialogger.error("Trace unavailable: ...")and can never mask the exception in flight.trace.enable(), athreading.Lockaround each write, flushed per record. No footer protocol, no counters, no loader.Event schema
{"event": "start", "span": "evaluate", "span_id": 7, "wall_time": 1757400000.1, "monotonic": 1234.5, "thread_id": 6199, "attrs": {"candidate_id": "g1-s2", "split": "train", "evaluation_phase": null}} {"event": "end", "span": "evaluate", "span_id": 7, "wall_time": 1757400012.3, "monotonic": 1246.7, "thread_id": 6199, "attrs": {...}, "duration_seconds": 12.2, "outcome": "ok"}outcomeis"ok"or"error"; on errorerror_typecarries the exception class name only (never the message).attrsis a small per-span identity dict: candidate id and generation forproposal, candidate id / split / evaluation phase forevaluate, candidate id forvalidate, the prompt artifact name foragent. Matchstarttoendonspan_id(proposal workers interleave).Truncation
A trace is complete iff its last line is
{"event": "end", "span": "run", ...}. Anything else means the process died mid-run and later spans are missing; every record before it is still whole (each is flushed as written).Gates
uv run python -m pytest -q— 1003 passeduv run ruff check src/ tests/— cleanuv run mypy --strict src/helix/— cleangit diff --stat origin/main -- src/helix/{executor,evolution,mutator,merger}.py— 3 files, 9 insertions(+), 2 deletions(-)Previous implementation (wrapper-based) preserved at commit 93ef401 for reference.
🤖 Generated with Claude Code
https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4