Skip to content

feat(trace): add JSONL timing traces to helix evolve - #50

Open
KE7 wants to merge 3 commits into
mainfrom
feat/trace-jsonl-timing
Open

feat(trace): add JSONL timing traces to helix evolve#50
KE7 wants to merge 3 commits into
mainfrom
feat/trace-jsonl-timing

Conversation

@KE7

@KE7 KE7 commented Aug 6, 2026

Copy link
Copy Markdown
Owner

What

helix evolve --trace PATH and helix resume --trace PATH (or HELIX_TRACE=PATH) append a JSON Lines timing trace of the run, so you can see where the wall clock went: proposals vs. sequential validation, evaluator vs. agent time, per generation.

Design: one decorator, no splits

The whole feature is helix.trace.traced(span), a functools.wraps decorator applied to functions that already exist on main and whose whole body is exactly the span we want:

span decorated function measures
run run_evolution the whole run; its end is always the last line
proposal _run_proposal_worker one proposal slot (parent eval, mutation, child eval)
evaluate run_evaluator one evaluator invocation
validate _run_full_val_eval one sequential full-validation stage
agent invoke_claude_code one agent-backend call (mutation inside a proposal, seed generation inside seed, otherwise a merge)
seed generate_seed seedless-mode seed generation

Why this shape:

  • No function splits, no signature drift. The core change is 9 added / 2 changed lines in executor.py, evolution.py, mutator.py (merger.py untouched): three import edits and six @traced("...") lines. Nothing to keep in sync when a signature changes.
  • Exception-safe. The decorator's finally classifies any BaseException (KeyboardInterrupt included), writes the end record, and re-raises untouched. A failing sink is reported once via logger.error("Trace unavailable: ...") and can never mask the exception in flight.
  • Near-zero cost when disabled. The wrapper checks one module-level flag and calls through; attrs extraction does not even run.
  • Thread-safe append-and-flush sink. Opened once by trace.enable(), a threading.Lock around each write, flushed per record. No footer protocol, no counters, no loader.

Event schema

{"event": "start", "span": "evaluate", "span_id": 7, "wall_time": 1757400000.1, "monotonic": 1234.5, "thread_id": 6199, "attrs": {"candidate_id": "g1-s2", "split": "train", "evaluation_phase": null}}
{"event": "end",   "span": "evaluate", "span_id": 7, "wall_time": 1757400012.3, "monotonic": 1246.7, "thread_id": 6199, "attrs": {...}, "duration_seconds": 12.2, "outcome": "ok"}

outcome is "ok" or "error"; on error error_type carries the exception class name only (never the message). attrs is a small per-span identity dict: candidate id and generation for proposal, candidate id / split / evaluation phase for evaluate, candidate id for validate, the prompt artifact name for agent. Match start to end on span_id (proposal workers interleave).

Truncation

A trace is complete iff its last line is {"event": "end", "span": "run", ...}. Anything else means the process died mid-run and later spans are missing; every record before it is still whole (each is flushed as written).

Gates

  • uv run python -m pytest -q — 1003 passed
  • uv run ruff check src/ tests/ — clean
  • uv run mypy --strict src/helix/ — clean
  • git diff --stat origin/main -- src/helix/{executor,evolution,mutator,merger}.py — 3 files, 9 insertions(+), 2 deletions(-)

Previous implementation (wrapper-based) preserved at commit 93ef401 for reference.

🤖 Generated with Claude Code

https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4

@KE7 KE7 changed the title Add helix evolve --trace: JSONL event stream with timestamps for the agent-vs-evaluator time split feat(trace): add JSONL timing traces to helix evolve Aug 10, 2026
@KE7
KE7 force-pushed the feat/trace-jsonl-timing branch 2 times, most recently from 54e3741 to 7aae6b3 Compare August 23, 2026 21:58
KE7 and others added 3 commits September 9, 2026 17:41
Add `helix.trace.traced(span)`: a signature-preserving decorator that
appends a `start`/`end` record pair (span id, wall/monotonic clocks,
duration, thread id, outcome, error class, cheap identity attrs) to a
JSONL sink opened once by `trace.enable(path)`.  When tracing is off the
wrapper is a single global check.  The `finally` classifies any
BaseException, writes the end record, and re-raises untouched; a failing
sink is reported once via `logger.error("Trace unavailable: ...")` and
never masks the exception in flight.

Decorate the five functions whose whole body is the span we want:
`run_evaluator` (evaluate), `_run_full_val_eval` (validate),
`_run_proposal_worker` (proposal), `invoke_claude_code` (agent),
`generate_seed` (seed) — plus `run_evolution` (run) so the last line of a
complete trace is always the run's end record.  No function is split and
no signature changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4
Parse the path, call `trace.enable()` once before `run_evolution`, and
exit 2 with a clear message when the file cannot be opened.
`HELIX_TRACE=PATH` is honoured as the environment fallback.  Document the
event schema, the spans, how a truncated trace is recognised, and a jq
line for per-generation wall time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4
Decorator: args/kwargs/return passthrough, start/end pairing, error
outcome with class name only, a broken sink never masking a
KeyboardInterrupt in flight, one well-formed line per record under a
thread pool, disabled-mode no-op, explicit and registry attrs
extraction.  `enable`: env fallback, unopenable path, per-record flush.
CLI: `--trace` on evolve and resume enables the sink before evolution.
End to end: a real `run_evolution` through the evaluator/mutator
override hooks emits run, validate, proposal, evaluate, and agent spans
and ends with the run's end record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4
@KE7
KE7 force-pushed the feat/trace-jsonl-timing branch from 93ef401 to 86552b4 Compare September 10, 2026 00:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant