Feature proposal: opt-in capability-gap event export for machine-demand measurement

[flyoung588]

Problem

ClawMetry already observes sessions, tool calls, cron activity, costs, and failures across many Agent runtimes. Those observations can show what happened inside an Agent, but there is no optional machine-readable export that says: "this Agent attempted to obtain a capability and could not access, fit, afford, or execute it."

That missing boundary is exactly the demand signal MIX is testing.

Proposed small integration

Would you be open to an opt-in capability-gap exporter that maps already-observed, structured ClawMetry events to the MIX E01-E08 taxonomy?

Examples:

• tool/capability not found -> E01_NO_MATCH
• auth or permission rejection -> E02_NO_ACCESS
• schema/policy mismatch -> E03_NO_FIT
• fallback requested -> E04_NEED_ALTERNATIVE
• timeout/429/capacity -> E05_CAPACITY_GAP
• licensing/rights rejection -> E06_RIGHTS_GAP
• missing provenance/verification -> E07_EVIDENCE_GAP
• explicit budget boundary -> E08_CAPITAL_NEED

The exporter would be disabled by default, add no heavy dependency, and send only stable reason codes plus minimized capability metadata. It would never export prompts, task text, exception bodies, credentials, transcripts, or customer content.

Why ClawMetry is the first-choice integration

ClawMetry already supports many Agent runtimes and explicitly maintains real-fixture siblings for synthetic event shapes. An exporter here can test machine-demand capture across real daily workloads without forcing every runtime to adopt a separate framework-specific hook.

Current MIX status (no inflated claims)

• public implementation: https://github.com/flyoung588/MIX
• verified external demand events today: 0
• test and Alpha databases are isolated;
• append-only demand receipts and origin flags exist;
• MIX does not execute or reroute the observed Agent.

If the direction fits ClawMetry's scope, I can prepare a small design first and implement it following this repository's contribution rules. If outbound export is out of scope, a local JSONL callback contract would be enough for MIX to consume without adding a direct dependency.

[vivekchand]

Hi @flyoung588! Took a careful look at this. Here's the plan:

The problem
ClawMetry observes many structured failure and gap signals (tool errors, capacity limits, auth rejections, budget blocks) but has no way to surface them as a stable, taxonomy-tagged, machine-readable stream that external measurement tools can subscribe to locally.

Approach

The local JSONL callback path you mentioned is the right first step — no outbound HTTP, no new package dependencies, no direct coupling to MIX. Here's how that fits the existing codebase:

1. clawmetry/capability_gaps.py (new, ~100 LOC) — a lightweight classifier that reads already-ingested events from DuckDB and maps them to the E01–E08 taxonomy. Pattern: follows clawmetry/detectors.py's run_all model. Each classifier is a pure function (event_row) -> Optional[GapEvent]. Initial mappings (conservative — only where the signal is unambiguous):

   • Tool-not-found / capability-not-registered → E01_NO_MATCH
   • Auth/permission rejection in tool output → E02_NO_ACCESS
   • HTTP 429 / capacity exhaustion → E05_CAPACITY_GAP
   • Budget-block from clawmetry/proxy.py → E08_CAPITAL_NEED

2. clawmetry/sync.py (~10 LOC change) — hook the classifier into the daemon's main loop via _ingest_keepalive_heartbeat. When CLAWMETRY_CAPGAP_EXPORT_DIR is set, append classified gap events as JSONL to $CAPGAP_EXPORT_DIR/capability_gaps.jsonl. Off by default, append-only, no truncation.

3. routes/usage.py or routes/health.py (~30 LOC) — new GET /api/capability-gaps?window=1h endpoint reading from DuckDB. Returns recent gap events with taxonomy code, session id, tool name, and timestamp. Zero Stripe/auth/billing touch.

Test plan

• Unit tests in tests/test_capability_gaps.py: one fixture per E-code confirming the mapping
• Verify the JSONL writer only fires when CLAWMETRY_CAPGAP_EXPORT_DIR is set
• Confirm CLAWMETRY_CAPGAP_EXPORT_DIR unset leaves the daemon behaviour unchanged

Estimated change
~140 lines across 3 files (clawmetry/capability_gaps.py new, clawmetry/sync.py +10, routes/usage.py or routes/health.py +30).

Why plan-only for now

Two things I'd want a maintainer call on before shipping:

• Taxonomy sign-off: E03 (schema/policy mismatch), E04 (fallback), E06 (rights), E07 (evidence) all need a clear definition of what ClawMetry event shapes map to them. Getting E01/E02/E05/E08 right first and leaving the rest as unknown is safer than inventing mappings.
• Scope direction: outbound vs local-only — the local JSONL path is unambiguously safe, but if outbound is ever on the table it changes the architecture. Better to confirm before the first PR lands.

If you (or @vivekchand) want me to proceed with just the local-JSONL + /api/capability-gaps slice, comment bot-go and I'll pick it up next run. Happy to narrow further if the taxonomy mapping needs discussion first.

---

Generated by Claude Code