feat(e5): audit runner — providers, tolerant parsing, recorded runs

[Ajay03299]

Completes the E5 harness on top of the corpus and metrics merged in #10.

corpus item → provider → raw text → parse → JSONL record → panel → metrics

Runs fully offline. Live calls are the exception, not the default.

e5/providers.py

One structural Protocol, three implementations — so tests need no mocking library.

• StubProvider emits JSON rather than returning an array directly, so stub runs exercise the same parser a live reply would and still catch parser regressions.
• ReplayProvider turns one real audit into a permanent offline fixture. A missing key raises instead of falling back to a default: a silent substitution would make a partial recording look complete.
• LiveProvider is the only one that touches the network. It reads its key from the environment, never accepts a literal, refuses the [YOUR_KEY_HERE] placeholder, and retries transient failures with backoff.

.env.example gains OPENAI_KEY and ANTHROPIC_KEY — retail_ai.py's docstring already told users to set OPENAI_KEY, but the template never defined it.

e5/parsing.py — why it parses tolerantly

Real replies are prose with a JSON block, or a markdown table, or weights summing to 0.97. The tempting move is to accept only well-formed output, but that biases the measurement: if model A emits clean JSON 99% of the time and model B 80%, discarding failures compares A's full response distribution against B's tidiest 80%, understating B's dispersion — which is precisely what E5 measures. Format compliance also plausibly correlates with templated advice, so silent rejection would push ĥ upward.

So every parse records raw_text, the extracted allocation, the pre-normalization sum, and a ParseStatus (ok / renormalized / partial / ambiguous / no_weights / empty). Exclusion is an analysis-time parameter, not a hard-coded policy.

Two bugs found during development, both of which returned plausible wrong answers rather than raising:

1. Per-ticker proximity search — in "40% in AAPL, 35% in MSFT" the 40% falls within the proximity window of MSFT and gets claimed twice.
2. Greedy nearest-neighbour with a claim set — 35% is nearer to AAPL than to MSFT, finds AAPL taken, and is discarded, leaving MSFT with no weight.

Both produced valid-looking simplex points. Only comparing against a known answer caught them, and both are now regression-tested.

Corpus prompts now carry an explicit JSON output contract, so prose parsing is a fallback. When number and ticker counts disagree — a distractor like "AAPL returned 30%" alongside the real allocation — the result is flagged AMBIGUOUS rather than silently mis-bound. Distinguishing a past return from a recommendation needs semantics, not regex, so the honest move is to record low confidence.

e5/runner.py

Append-only JSONL keyed on (query_id, model, repeat), so an interrupted run resumes and a partial run stays analyzable. Provider failures are recorded with error set and the run continues — losing a whole audit to one timeout is worse than a recorded gap.

Allocations are padded to the full universe. Items ask about ticker subsets, but e5.metrics needs a common basis to stack into the (models, repeats, assets) panel.

What stub mode does and doesn't show

A stub-only run validates the pipeline, not the phenomenon. Several archetypes — PASSIVE_INDEX most obviously — are hard-coded to equal-weight regardless of input, so market condition can't move them, and a stub run shows spread = 0 and drift = 0 by construction. That's the point of a zero-variance baseline, but any ĥ figure from stub mode is an artifact of the stub, not a measurement. Real numbers need LiveProvider and your keys.

Suggested first live run: 1 archetype × 1 ticker pair × 3 conditions × 2 models × 5 repeats = 60 calls, a few cents. Enough to confirm live replies parse, and it produces the fixture ReplayProvider needs so the run is reproducible offline forever after.

Tests

47 new (28 parsing, 19 runner), 180 total on Python 3.13 / macOS arm64. LiveProvider is tested only for key handling, never over the network.

Still open from #8

Per-query vs pooled ĥ determines how many repeats a real run needs per prompt — worth settling before spending on the full corpus.