feat(onboarding): a "Send your data" step with a live ingest status strip

[vivekchand]

Problem

Onboarding has no data step and the product has no "is my data arriving?" surface anywhere.

clawmetry/static/js/onboarding.js is: sign in → choose cloud or self-host → done. Its own header comment states the premise — "Runtime detection is 100% automatic (daemon-side); nothing here asks the user to name or configure a runtime."

That premise is right and it is a real advantage over every vendor that makes you wire an exporter. But it has two costs today:

1. There is nowhere to put a second door for agents auto-detection cannot see (CI, containers, remote hosts) — the case the paired ingest-key issue opens up.
2. There is no confirmation. A user who installs and sees an empty dashboard has no way to tell "nothing is running" from "it is broken". That question is what kills setup funnels, and we have prior evidence it kills ours: 285 launches → 13 choices in 14 days on the old gate.

Bronto's screen answers it with one persistent strip: Waiting for events — we'll let you know when data arrives here → Connected · 22 events received, with a progress bar toward the threshold where parsing kicks in. It is the cheapest trust-builder on their site.

Scope

A. GET /api/onboarding/ingest-status (new, in routes/overview.py alongside the existing /api/onboarding/state)

Returns, cheaply enough to poll every 2s:

{
  "connected": true,
  "events_total": 4127,
  "events_recent": 22,
  "first_event_at": 1757300000.0,
  "last_event_at": 1757300412.0,
  "sources": [
    {"kind": "filesystem", "runtime": "claude_code", "events": 4100, "last_at": 1757300412.0},
    {"kind": "otlp",       "runtime": "my_langchain_app", "events": 27, "last_at": 1757300390.0}
  ]
}

sources is what makes it more useful than Bronto's counter: it says which door the data came through, so "I configured OTLP and it isn't arriving" is answerable without a log dive. Read it through routes/local_query.py::_dispatch like every other handler — not from raw files, or it returns empty on cloud.

B. A "Send your data" onboarding step, after the hosting choice, with three doors:

Door	Content	Default
This machine	What was auto-detected, listed by runtime. Nothing to configure.	Pre-selected; for most users this is the whole answer
Somewhere else	Ingest key + endpoint + a copy-paste curl that works	—
Ask your agent	The setup prompt (paired issue)	—

C. The strip. Persistent under the step: Waiting for events… → Connected · N events received. On an install where auto-detect already found sessions it lands on connected immediately, which turns the step from a chore into a confirmation. Skippable, and never blocking — the user can always reach the dashboard.

Non-goals

• No "events needed to parse" progress bar. That threshold exists because Bronto has to learn a schema from unstructured lines; our events are typed on arrival. Copying the bar would be copying an artefact of their weakness.
• No config-conversion door (paste a Vector/Fluent Bit config, get a rewrite). Only pays off for teams already running collectors, which is a minority of this audience.

Traps

• Adding a step to a flow whose whole selling point is having no steps. It must be skippable in one click and must never gate the dashboard.
• The onboarding overlay has bitten us before: a duplicate id=page-guard blanked switchTab. Check element ids before adding markup to templates/partials/overlays.html.
• Poll the status endpoint, don't SSE it — this runs before the user has any reason to trust us with a long-lived connection.

Acceptance

• A fresh install with existing local sessions shows a non-zero event count without the user clicking anything.
• A remote push (paired ingest-key issue) flips the strip to connected within one poll interval, and sources names it as otlp, not as filesystem.
• Skipping lands on the dashboard with no nag on next launch.
• The endpoint answers in under 50 ms on a store with 100k+ events (it is polled).

Part of the ingest-gap plan (phase 1, item 3 of 5).

[vivekchand]

Phase 1 issue map — "open the front door"

The gap this closes: an agent is observable only if the daemon runs on the same machine as the agent. These eight issues, in dependency order:

#	Repo	Issue
1	clawmetry	#5679 — scoped ingest key authenticates /v1/* and /api/v1/*
2	clawmetry-cloud	#2344 — name the two trust classes (gates #2343)
3	clawmetry-cloud	#2343 — hosted ingest key + OTLP endpoint
4	clawmetry-pro	#230 — pushed telemetry becomes real sessions; honest control verdicts
5	clawmetry	#5682 — generated docs/INGEST.md + drift test
6	clawmetry	#5681 — per-runtime setup prompts
7	clawmetry	#5680 — "Send your data" onboarding step + live status strip
8	clawmetry-landing	#754 — public endpoint reference; reconcile reach + storage claims

Upstream dependency: clawmetry PR #5676 ships clawmetry/apikeys.py (cmk_ scoped keys, clawmetry key CLI). #5679 extends it with a write scope rather than starting a second key system.

Ordering that matters: #2344 lands before #2343 (a plaintext trust class named before it exists, not after it becomes a P0), and #5682 lands before #754 (the landing page is written from the generated doc, because drift-bot greps repo docs and fails claims the repo denies).

Deliberately out of scope for phase 1 — and mostly out of scope permanently: the 33 AWS / 11 Azure / 17 infra integration pages, generic log parsing (syslog/CEF/GELF), an LLM-authored parser, a search language and dashboard builder, and being a destination for Fluent Bit / Vector / Logstash / the Datadog Agent. Those make ClawMetry a worse log platform and a less distinctive agent platform.

[vivekchand]

Hi @vivekchand! Took a careful look. Here's the plan:

The problem
There's no confirmation signal anywhere in ClawMetry that tells a user "your data is actually arriving" — an empty dashboard after install is indistinguishable from a broken one, which kills setup funnels.

Approach

This is a three-part issue. I'm implementing Part A (the backend endpoint) now; Parts B and C (the wizard step + UI strip in onboarding.js / templates/partials/overlays.html) are noted below and ready for a follow-up once the data layer exists.

Part A — GET /api/onboarding/ingest-status (implementing now)

1. clawmetry/local_store.py — add query_ingest_status(*, recent_window_secs=86400) -> dict. Two DuckDB passes: one aggregate (total, recent 24h count, first/last timestamps) and one GROUP BY agent_type for the per-source breakdown. created_at is BIGINT epoch-ms (already used for age-based pruning in query_events_by_ingest). The kind field ("filesystem" vs "otlp") is derived from whether agent_type is in entitlements.ALL_RUNTIMES; lazy import, no circular dependency.

2. routes/local_query.py — add "query_ingest_status" to _DAEMON_METHODS so the daemon proxy serves it. The lint_daemon_allowlist.py guard in CI pins every local_store_via_daemon("query_*") call to this set.

3. routes/onboarding.py — add the route on bp_onboarding. Follows the local_store_via_daemon → direct-fallback pattern used by routes/alerts.py. Response shape exactly matches the spec in the issue. Exempt from the trial-enforcement gate (/api/onboarding/ is already in trial_enforcement.py::_EXEMPT_PREFIXES).

Part B — the "Send your data" wizard step (plan only, for human follow-up)

• New step in clawmetry/static/js/onboarding.js after the hosting choice.
• Three doors: "This machine" (auto-detected runtimes), "Somewhere else" (curl snippet), "Ask your agent" (paired feat(onboarding): per-runtime setup prompts, generated from the ingest constants #5681).
• Note from issue: check element ids against templates/partials/overlays.html before adding markup — a duplicate id=page-guard blanked switchTab before.

Part C — the status strip (plan only)

• Polls /api/onboarding/ingest-status every 2 s; transitions from "Waiting for events…" to "Connected · N events received".
• Skippable, never gates the dashboard.

Test plan

• Existing test suite (make lint) will catch the allowlist drift guard.
• Manual: curl http://localhost:8900/api/onboarding/ingest-status on an install with existing sessions should return "connected": true with a non-zero events_total; on a fresh install, "connected": false.

Estimated change
~80 lines across 3 files (clawmetry/local_store.py, routes/local_query.py, routes/onboarding.py).

Drafting a PR now for Part A — will link here when ready. If you'd rather drive this yourself, drop a comment and I'll back off.

---

Generated by Claude Code

[vivekchand]

Draft PR opened: #5730. Please review when you have a moment.

---

Generated by Claude Code