Context
Today (Apr 20) motley hit MiniMax 500 insufficient balance (1008) at 1:18pm and 1:20pm. Both went undetected until Tim manually asked me to check at 1:22pm — ~4 min latency on a real failure. motley already writes structured errors to events.jsonl. Manual discovery → reactive pattern. Tim's ask: flip to proactive via Strix-triggered alerts.
Design (locked with Tim, Apr 20)
- Alert shape: trigger Strix (self-DM or equivalent), NOT direct DM to Tim. Strix does diagnosis pass, sends one DM with root + recommended action.
- Dedup: layered on #91 batched message processing. 50
insufficient_balance fires → 1 turn with kind+count+sample. Same infrastructure, different producer.
- Scope: generic reader with configured agent-paths (motley, verge, future siblings). Not motley-specific.
- "Fix directly" boundary: code/config errors → ship PR (existing edit-access workflow from Feb 23). Credit/account/infra errors → surface diagnosis only, don't auto-act.
Components
- Event reader — tail configured
events.jsonl paths, classify errors (structured "error": true or non-2xx response_status).
- Classifier — error kind (
insufficient_balance, rate_limit, model_not_found, etc.) so dedup has a key.
- Trigger — emit Strix-visible alert carrying
{agent, kind, count, sample_event, timestamps}.
- Dedup — one-shot-until-different-kind per agent per kind (answered 9:01am today). Reset when kind changes or after cooldown.
Depends on
Out of scope
- Auto-topup credits (account access, not mine).
- Restarting sibling processes (read-only per Feb 23 workflow).
- Fixing upstream API errors at the sibling (diagnosis + PR where applicable only).
Priority
High. Real instance today (motley 4-min latency), generalizable to verge + future siblings.
Correction (Apr 20 afternoon): Earlier version of this issue referenced "#106" which doesn't exist — the batched-processing chainlink was committed to this morning but never actually filed until afternoon (now #91). Updated references.
Context
Today (Apr 20) motley hit MiniMax
500 insufficient balance (1008)at 1:18pm and 1:20pm. Both went undetected until Tim manually asked me to check at 1:22pm — ~4 min latency on a real failure. motley already writes structured errors toevents.jsonl. Manual discovery → reactive pattern. Tim's ask: flip to proactive via Strix-triggered alerts.Design (locked with Tim, Apr 20)
insufficient_balancefires → 1 turn with kind+count+sample. Same infrastructure, different producer.Components
events.jsonlpaths, classify errors (structured"error": trueor non-2xxresponse_status).insufficient_balance,rate_limit,model_not_found, etc.) so dedup has a key.{agent, kind, count, sample_event, timestamps}.Depends on
Out of scope
Priority
High. Real instance today (motley 4-min latency), generalizable to verge + future siblings.
Correction (Apr 20 afternoon): Earlier version of this issue referenced "#106" which doesn't exist — the batched-processing chainlink was committed to this morning but never actually filed until afternoon (now #91). Updated references.