[rybosome]

Executive summary

@mic.eval now accepts sinks=, so an eval file can declare where its results go and mic run uses those sinks with no extra flags. Declared sinks are scoped: each one receives only the evaluations that declared it. mic run --no-sinks turns them off for a run, mic list shows their names, and mic preflight validates them without opening them.

Why

Linear: TEN-1188. The microevals results pipeline needs every run, local or CI, to reach one shared sink. Today a custom sink can only be passed to mic.run(..., sinks=[...]) from Python; mic run builds only a Braintrust sink. Without this change every eval would need a Python __main__ bypassing the CLI.

The issue's open questions were decided by the maintainer:

• A declared sink receives events only for the evaluation that declared it, not every event in the run.
• mic run gets an opt-out flag (--no-sinks).

Design and implementation

• Model and decorator. Evaluation.sinks: tuple[ResultSink, ...] is a trailing field that defaults to (). It uses compare=False because sinks are caller-owned and need not be hashable. All three eval overloads accept sinks: Sequence[ResultSink] = (). No new sink type is introduced.
• Validation. resolve_options checks that each declared sink has a non-empty name and a callable open. It rejects different instances that share a name and allows the same instance to repeat. Preflight goes through resolve_options, so it validates declared sinks without opening them.
• Merging (engine.arun).
  • Order is the recorder, then explicit sinks, then declared sinks in plan order.
  • Declared sinks are deduplicated by object identity across the suite.
  • An instance that is also explicit opens once, as an unscoped explicit sink.
  • The existing duplicate-name check runs over the merged list before any source is read.
• Scoping (delivery.Scope). Each declared sink carries the set of evaluations that declared it and the source ids those evaluations read.
  • open gets a RunInfo filtered to those evaluations.
  • TrialFinished is filtered by task.
  • Source-level events (CaseAccepted, RecordRejected, SourceFinished) are filtered by source_id. With a shared dataset, the declaring eval's sink still receives the shared cases, because they are its inputs.
  • finish gets an outcome with a scoped summary, scoped sources, and only the tasks[...] requirements for those evaluations. The summary comes from an extra Aggregator per distinct scope, fed the same observations. status, exit_code and failures stay run-wide, because they describe the run.
• Failure semantics are unchanged. A write that raises disables the sink and stops admission run-wide; a finish or close failure fails the run.
• CLI.
  • --no-sinks applies dataclasses.replace(spec, sinks=()) to the selection, so run()/arun() keep their signatures.
  • list adds sinks=a,b to text output and a sinks array to eval rows in --json. It reads only sink.name.
  • resolve_evaluations now returns list[Evaluation[Any, ...]] (it is internal) so the CLI can replace the selection under strict typing.
• Not added to artifacts. Sink names are not written into RunInfo.tasks configuration, because run-v4 uses additionalProperties: false. Sink receipts already record which sinks ran.

Verification and confidence

Risks: wrong delivery scope (leaking another eval's data, or dropping the declaring eval's shared-source events), duplicate opens in suites, name collisions slipping past the pre-effect check, ordering, and changed failure behaviour.

• New tests/runtime/test_declared_sinks.py (13 tests):
  • the decorator stores sinks and does not open them;
  • order is recorder, then explicit, then declared, checked by both open order and receipt order;
  • one instance declared across a suite opens once and sees both evals;
  • scoping: RunInfo, trial events, source events, the finish summary, sources and requirements contain only the declaring eval, while explicit sinks still get everything;
  • shared-dataset source events reach the declaring eval's sink;
  • an instance that is both explicit and declared stays unscoped;
  • name collisions (across the suite, within one eval, against an explicit sink) fail before the dataset factory is called;
  • preflight validation without opening;
  • write and finish failures: disabled sink, admission stopped, healthy sink finalized, no secret text;
  • CLI list text and JSON, and run with and without --no-sinks.
• Mutation check: replacing the trial scope filter with return True makes the scoping tests fail.
• Typing: a positive fixture for sinks= that keeps the full Evaluation type, and a negative fixture rejecting a non-sink.
• uv run python scripts/verify.py: tests, coverage json/report/gates, strict typing, authoring typing, lint and format all PASS.

Compatibility and risks

• Public API: additive only. There is a new optional sinks= keyword, a new trailing Evaluation field with a default, a new CLI flag, and a new sinks key on eval rows in mic list --json. run()/arun() are unchanged.
• Formats: no change to mic-run-v4 or mic-event-v1.
• Dependencies: none.
• Side effects: this is a new kind of implicit effect. Running an eval that declares a remote sink now writes remotely with no CLI flag. This is the purpose of the feature; it is documented in docs/reporting.md and docs/cli.md, and the mic skill now tells agents to check mic list and use --no-sinks when publishing is not authorized.
• Passive construction: declared sinks are built at import, including under mic list. The existing passive-construction rule still applies, and the docs restate it.