[vdittgen]

The problem

Categoriser failures are deliberately not cached, so a flaky call retries next run:

# Don't cache fallback verdicts — we want to retry next run.

Right for a transient glitch. Ruinous for a persistent one: when the provider is unreachable or misrouted, every uncached event fails and the whole backlog re-fires on every run, forever.

Observed in production — batches of exactly 70 identical failures per pipeline run, across several days:

status_code: 404, model_name: arandu-pro,
body: {'message': "model 'arandu-pro' not found", ...}

batch	failures
Aug 5 23:36	70
Aug 5 21:46	70
Aug 5 20:56	70
Aug 4 12:04	70

In cloud mode each retry is a paid round trip, so a misroute bills repeatedly for calls that cannot succeed. The existing per-run cap doesn't help — it's 200, and 70 is well under it.

(The misroute itself is fixed separately in arandupro#72. This PR is about not amplifying the next one.)

Fix

Consecutive failures are the signal. One event failing is that event's problem; three in a row is the provider's, and every remaining event will fail identically. On tripping, the rest of the run takes the fallback category without spending a call — the same behaviour the per-run cap already has, so the marts still receive a complete row set.

Effect on the observed incident: 70 failed calls per run → 3.

Deliberately unchanged

• Failures are still not cached. A recovered provider re-categorises rather than inheriting fallbacks — there's a test for it.
• The breaker bounds one run, it does not persist across runs. Cross-run backoff is a bigger design question (where to store health, when to reset) and would make a recovered provider wait; three calls a run is a cheap probe.

Observability

The run summary now reports when the breaker tripped. The previous incident went undiagnosed for days partly because nothing in the logs distinguished "provider down" from "70 bad events".

Tests

Four, including one that pins the call count to the failure limit rather than the event count — it fails without the breaker, verified by neutralising the limit.

• caps calls when the provider is down
• every row still falls back (marts stay complete)
• failures still not cached — recovery re-categorises
• an isolated failure surrounded by successes does not trip it

2,978 passed; the 5 failures are the pre-existing WhatsApp bridge ones that fail identically on a clean tree.

🤖 Generated with Claude Code