[rybosome]

Executive summary

Adds a private RunObserver hook to the execution engine. As a run proceeds, it receives small scalar notifications: the run plan, source start and finish, accepted and rejected records, trial start and finish, cancellation, and sink finishing. This is the first PR in the live CLI progress work (design option D: one line per case in scrollback plus a sticky status footer). Users see no change in this PR. The public API, CLI output, and artifact formats are all untouched.

Why

Today mic run prints nothing until the run finishes, so a three-minute eval looks like it has hung. Streaming progress needs signals the engine doesn't currently expose:

• Trial start. This is the only way to show "row 7 has been running for 1:25", or to say which trials Ctrl-C is waiting on.
• When dataset reading starts. The dataset factory (for example a BigQuery query) can take a while before the first row arrives.
• When cancellation begins.
• When sinks are finishing after the last trial. Remote flushes can be slow.

Design and implementation

• src/mic/_runtime/progress.py defines:

  • RunObserver, a protocol with nine synchronous methods;
  • the small frozen views it receives: RunPlan, TaskPlan, SourcePlan, TrialError;
  • Progress, a guard that forwards each call and drops the observer after its first exception.

• Why this isn't a ResultSink:

  • A failing sink stops new trials from starting and fails the run.
  • Sinks show up in run.json receipts.
  • Each sink gets a deep copy of every event, including case payloads and task outputs.
  • The sink event format has no "trial started" event, and adding one would change mic-event-v1.

  A progress display needs none of that, and must never be able to fail a run.

• Engine (engine.py):

  • The public arun() now delegates to a private execute(..., observer=None); its signature is unchanged.
  • Hooks sit next to the existing delivery points, plus trial_started in the worker and finishing before delivery.finish.
  • trial_finished fires before the awaited delivery.write, so a cancelled write can't hide a trial that was already counted.
  • A new interrupt() helper replaces the three places that set interrupted, and fires cancelling once.

• Delivery (delivery.py): a new read-only Delivery.unfinished property lists sinks that are open but haven't settled. This lets the display name the sinks it's waiting on, such as braintrust.

• What observers receive. Only scores, statuses, latencies, and the phase, exception type and scorer name for errors (TrialError). Exception messages never reach an observer, which matches how failure() already records errors. Labels come from the case's optional label.

Verification and confidence

These are the risks this PR could introduce, and how each was tested (tests/runtime/test_progress_observer.py):

Risk	How it was tested
A hook changes a run's outcome	For each hook, an observer that raises at that hook. Its run must have the same status, exit code, trial counts, score statistics, source summaries (including the digest), failures, sink receipts and recorded events as an identical run without an observer. The observer must also never be called again after it raises.
Hooks arrive out of order or inconsistently	Concurrency 4, two trials per case, and out-of-order completion. Checks: planned comes first and finishing last; each trial is accepted, then started, then finished, exactly once; the source finishes after its last accepted case; scores and the source summary match the result.
Cancellation leaves trials unaccounted for	The test waits until both workers are busy, the queue is full, and the producer is blocked, then cancels. Checks: cancelling fires exactly once; both running trials finish as cancelled; the two queued trials finish as cancelled without having started; the row that was never admitted gets no events; no trial starts after cancelling.
Private error text leaks	Exceptions in the task and the scorer carry "private" messages. Observers see only TrialError(phase, type, scorer), and "private" appears nowhere in the recorded calls.
Suites and shared datasets	Two evaluations share one dataset with different trial counts. The single SourcePlan lists both, and each one's trials start the right number of times.
Sink listing at finish	finishing lists ("jsonl", "capture") when recording, and excludes a sink that failed to open.

• I ran the new tests 30 times locally with no failures. One order-dependent assertion was rewritten to compare contents, because source completion can legitimately interleave with the last trials.
• uv run python scripts/verify.py ran 756 tests with 0 failures. Strict typing, authoring typing, lint, format and the coverage gates all pass. progress.py has 100% coverage, and every new engine line is covered (the remaining uncovered engine lines are older branches).

Compatibility and risks

• No public API change: mic.__all__ and the arun/run signatures are unchanged. execute and RunObserver live under the private _runtime package.
• No change to the artifact or event format, no new dependencies, and no new side effects.
• The cost per trial is a few function calls when an observer is set, and one None check when it isn't. No await points were added, so scheduling and cancellation behavior are unchanged.