[rybosome]

Executive summary

mic.TaskContext gains task_metadata, a mutable dict that is fresh for every trial. A context-aware task writes to it while it runs. Mic records it as the trial's task_metadata, shallow-merged under any returned TaskResult.metadata. It is also recorded when the trial fails, times out or is cancelled. The version goes to 0.4.0.

@mic.eval(dataset=cases, scorers=[...])
async def necessity(ctx: mic.TaskContext, case: Case) -> Verdicts:
    async with recording_llm_calls(ctx) as client:  # writes ctx.task_metadata in its finally
        ...

Why

In 0.3.0, a task can only emit metadata by returning a TaskResult, and writes to ctx.metadata are thrown away. Per-trial setup and teardown, like resetting a failure counter and attaching a model-call log, then needs a callback-taking runner wrapped around mic: a second framework. A failed trial also records no task metadata at all, yet that is the trial whose call log explains what went wrong. The consumer is the intent2auth microeval harness (Tenetgraph/intent2auth#910, TEN-1196). The design was approved in the handoff doc.

Design and implementation

• src/mic/models.py: adds task_metadata: JsonObject = field(default_factory=dict) to the frozen TaskContext. Reassigning it raises FrozenInstanceError; mutating it works. Because the field has a default, TaskContext(case_id, trial, metadata) still constructs.
• src/mic/_runtime/case.py, run_case:
  • Success path: once the task returns, {**ctx.task_metadata, **TaskResult.metadata} goes through the existing json_object(..., "$.task_metadata"). That call is both the snapshot and the validation, and it feeds the existing merge with dataset metadata. TaskResult keys win.
  • When the field is recorded: if the task neither writes nor returns a TaskResult, the field stays absent, so existing artifacts are unchanged.
  • Order: output validation still runs after metadata handling, so a trial whose output is invalid keeps its metadata.
  • Failure path: each except block calls a nested salvage_task_metadata() after the existing record_error. It records the snapshot without changing the status or errors[0]. If the snapshot does not serialize, it appends one schema-phase error instead. record_error takes an optional explicit phase for that.
  • returned flag: set when the task returns. From then on the success path owns the metadata. Without the flag, invalid metadata on the success path would be salvaged a second time and log a duplicate error.
• Docs:
  • docs/api.md: the contract, snapshot timing, failure semantics, and the three-level merge order (dataset, then ctx.task_metadata, then TaskResult).
  • docs/artifacts.md: task_metadata now also appears on failed, timed-out and cancelled trials.
  • The README task-context snippet now writes ctx.task_metadata.
• Release: version = "0.4.0" and uv lock.

The handoff doc said the schema error on a failed trial would name $.task_metadata. Mic's error records deliberately hold only phase, type and a generic message, so the path is not persisted. The tests check phase and type instead.

Verification and confidence

Main risks: data loss or masking on failure paths, cross-trial leakage, duplicate errors, and changes to artifacts that don't use the feature.

• tests/runtime/test_task_context_metadata.py has 18 cases and runs real evaluations through mic.run/mic.arun. It covers each behaviour row from the design:

  • plain output, and what scorers see;
  • the field stays absent with no writes;
  • TaskResult precedence, plus an empty TaskResult still recording {};
  • each trial starting empty at concurrency 1 and 4;
  • sync thread-pool tasks, and gather children writing the same dict;
  • raise, timeout and invalid output each keeping their metadata with the error unchanged;
  • cancellation;
  • invalid metadata (tuple, NaN, non-string key) failing a successful trial with exactly one error;
  • invalid metadata never masking the original failure;
  • background writes after return being ignored;
  • sink TrialFinished and JSONL artifacts agreeing and validating against the v4 schemas;
  • direct construction, independent dicts, and frozen reassignment.

  With the runtime change stashed, 17 of the 18 fail. The one that still passes is the "absent" case, which is meant to hold either way.

• README acceptance test: now asserts that the task-context snippet records {"model": "jev-latest"} on all six trials. That includes the run where every trial fails.

• Typing fixtures: positive.py writes and .updates the dict; negative.py rejects reassignment. verify_typing.py passes, rejecting all 23 deliberate mistakes.

• scripts/verify.py: tests, coverage gates, strict pyright, authoring typing, lint and format all pass. uv build and scripts/verify_packaging.py pass.

Compatibility and risks

• Public API: additive only. The new defaulted field is the only change; no break.
• Artifact format: no schema version change. task_metadata already exists in common-v4 as an optional field on any trial. Runs that don't use the feature produce the same artifacts as before.
• Behaviour change: a successful task that writes metadata which doesn't serialize now becomes task_failed. This is the existing rule for invalid TaskResult metadata, applied to the new field.
• Dependencies and side effects: none new.