docs(ingest): generate docs/INGEST.md from one ingest contract, with a drift test

[vivekchand]

Problem

Three surfaces are about to describe the same ingest contract, and today none of them shares a source:

1. docs/CUSTOM_RUNTIME_INGEST.md — hand-written, covers the run/event API only.
2. The setup prompts (paired issue) — read by agents, where a wrong header is a silent failure.
3. The public endpoint reference on the landing site (paired issue) — where drift-bot now enforces AC-WEB-001.3 truthfulness: it greps repo docs and fails a landing claim the repo denies.

Meanwhile the OTLP receiver's real capabilities are undocumented anywhere. dashboard.py::_otlp_decode has accepted protobuf, OTLP/JSON and gzip for some time. Nothing tells a user that, so functionally it does not exist.

Scope

A generated docs/INGEST.md, on the model of scripts/gen_query_contract_doc.py (which already generates docs/QUERY_CONTRACT.md from the contract, with a drift test). One module declares the ingest contract; the doc, the setup prompts, the landing page and the server's own validation all read it.

Contents, all derived:

• Both surfaces: /v1/{logs,metrics,traces} (OTLP) and /api/v1/runs* (run/event).
• Accepted content types and encodings — application/x-protobuf, application/json, Content-Encoding: gzip.
• Every header, spelled in the case the server actually matches.
• Size caps and batch caps, as numbers read from the constants that enforce them.
• Every response code and what it means, including what happens to a partially-bad batch (state the real behaviour, whatever it is — an all-or-nothing promise we don't keep is worse than an honest "valid events are kept").
• Which gen_ai.* attributes are read, and — explicitly — which are not yet.

A drift test in the shape of tests/test_query_contract_drift.py: regenerate, diff, fail if the committed doc is stale.

Why a whole issue for a doc

Because it is the anti-drift mechanism for the other three, not the doc itself. We have shipped a wrong number to users from exactly this shape of gap before — ARCHITECTURE.md and CLAUDE.md drifted into describing a different product, and the fix was to generate docs/MODULE_MAP.md rather than to correct the prose. Same move.

Acceptance

• docs/INGEST.md is generated, and editing it by hand fails CI.
• Renaming an ingest header changes the doc, the setup prompt and the landing page from one edit.
• A stranger can push an event using only that page.
• Every claim on the landing ingest page has a matching line in this doc (the truthfulness gate's requirement).

Part of the ingest-gap plan (phase 1, item 5 of 5).

[vivekchand]

Phase 1 issue map — "open the front door"

The gap this closes: an agent is observable only if the daemon runs on the same machine as the agent. These eight issues, in dependency order:

#	Repo	Issue
1	clawmetry	#5679 — scoped ingest key authenticates /v1/* and /api/v1/*
2	clawmetry-cloud	#2344 — name the two trust classes (gates #2343)
3	clawmetry-cloud	#2343 — hosted ingest key + OTLP endpoint
4	clawmetry-pro	#230 — pushed telemetry becomes real sessions; honest control verdicts
5	clawmetry	#5682 — generated docs/INGEST.md + drift test
6	clawmetry	#5681 — per-runtime setup prompts
7	clawmetry	#5680 — "Send your data" onboarding step + live status strip
8	clawmetry-landing	#754 — public endpoint reference; reconcile reach + storage claims

Upstream dependency: clawmetry PR #5676 ships clawmetry/apikeys.py (cmk_ scoped keys, clawmetry key CLI). #5679 extends it with a write scope rather than starting a second key system.

Ordering that matters: #2344 lands before #2343 (a plaintext trust class named before it exists, not after it becomes a P0), and #5682 lands before #754 (the landing page is written from the generated doc, because drift-bot greps repo docs and fails claims the repo denies).

Deliberately out of scope for phase 1 — and mostly out of scope permanently: the 33 AWS / 11 Azure / 17 infra integration pages, generic log parsing (syslog/CEF/GELF), an LLM-authored parser, a search language and dashboard builder, and being a destination for Fluent Bit / Vector / Logstash / the Datadog Agent. Those make ClawMetry a worse log platform and a less distinctive agent platform.

[vivekchand]

Hi @vivekchand! Took a careful look. Here's the plan:

The problem
docs/INGEST.md does not exist, the OTLP receiver's real capabilities (protobuf + OTLP/JSON + gzip, for all three signals) are completely undocumented, and the three surfaces that describe the ingest contract (CUSTOM_RUNTIME_INGEST.md, the future setup prompts, and the future landing page) share no source — so drift is guaranteed.

Approach

1. clawmetry/ingest_contract.py — new module that declares the full ingest contract as Python data: OTLP endpoints (/v1/metrics, /v1/traces, /v1/logs), accepted content types (application/x-protobuf, application/json), Content-Encoding: gzip support, the 64 MB decompressed cap (CLAWMETRY_OTLP_MAX_DECOMPRESSED_MB), response codes (200/400/429/501), the gen_ai.* span attributes read by _otel_to_row in dashboard.py:2823, and the run/event endpoints from routes/runtime_ingest.py:57-87 (free GET /api/v1/runtimes + Pro-gated writes).
2. scripts/gen_ingest_doc.py — generator on the model of scripts/gen_query_contract_doc.py, reads the contract and emits docs/INGEST.md; --check mode exits 1 on drift (for CI).
3. docs/INGEST.md — the generated output (committed alongside the generator so --check passes immediately).
4. tests/test_ingest_doc_drift.py — one test: re-run the generator, assert the committed doc matches. Mirrors tests/test_query_contract_drift.py::test_committed_doc_matches_generator.

Note on CI coverage: Per CLAUDE.md, CI runs explicit file lists — tests/test_ingest_doc_drift.py will need to be added to .github/workflows/ci.yml in a follow-up or the same PR after human review (can't touch CI workflows autonomously).

Test plan

• python3 scripts/gen_ingest_doc.py --check exits 0 after the PR lands.
• python3 -m pytest tests/test_ingest_doc_drift.py -q passes.
• A stranger can read docs/INGEST.md and push a curl event using only that page.

Estimated change
~220 lines across 4 files (clawmetry/ingest_contract.py, scripts/gen_ingest_doc.py, docs/INGEST.md, tests/test_ingest_doc_drift.py).

Drafting a PR now — will link here when ready. If you'd rather drive this yourself, drop a comment and I'll back off.

---

Generated by Claude Code

[vivekchand]

Draft PR opened: #5705. Please review when you have a moment.

One follow-up needed before marking ready: add tests/test_ingest_doc_drift.py to the explicit file list in .github/workflows/ci.yml (per CLAUDE.md: CI runs file lists, not pytest tests/, so the drift test won't execute in any job until it's named there).

---

Generated by Claude Code