fix(core-api): send exactly one heartbeat per container and fail loudly on a bad collector URL

[erni-a]

Summary

End-to-end validation of the heartbeat shipped in #1577 (backend 3.17.0) found that a Docker container sends two beats per cycle, that a mistyped collector URL is announced as ON and then refused silently, and a handful of smaller reporting and documentation defects. This PR fixes all of them in one place, and bumps the TypeScript client so the User-Agent header from #1561 can be published.

Defects and fixes

D1 - two heartbeats per container. core-api/Dockerfile runs uvicorn with --workers 2; every worker runs the lifespan, so each started its own loop. Observed against production on 2026-09-19: deployment 0e9d4f29... POSTed at 16:23:52Z and 16:23:59Z, deployment 1f866780... at 16:28:55Z and 16:29:21Z (both 202; the receiver's 20 h dedupe hid the second). GET /api/v1/telemetry answered differently depending on which worker took the request (last_sent_at, next_send_at, payload_preview), and the in-process clients_24h counter was split, so the stored beat carried only the sending worker's half (one worker previewed caura-client-python "21-100", the other "0").

Fix (no new dependency, worker count unchanged): a per-container state directory (CAURA_TELEMETRY_STATE_DIR, default <tmpdir>/caura-heartbeat, created 0700) and a new core_api.heartbeat.state module.

• Leader election: a non-blocking fcntl.flock on <dir>/leader.lock taken in install(). The holder runs the send loop; the others run no loop and print a "follower" line instead of a second ON line. Followers retry the lock every 60 s and take over if the leader dies (the kernel drops the lock), resuming from the leader's published next_send_at (bounded by one interval).
• Shared client counter: every worker flushes its in-memory family counter every 30 s and on shutdown to <dir>/clients-<pid>.json (write-to-temp + atomic rename). At send time the leader sums its live counter with every other live worker's file; on 202 it zeroes its own counter and writes <dir>/epoch; a worker seeing a newer epoch at its next flush subtracts what it had already flushed. Files of dead pids are pruned (os.kill(pid, 0)).
• Shared status: the leader writes <dir>/state.json after each cycle step; a follower answers GET /api/v1/telemetry from it, so every worker reports the same values. payload_preview is built by the answering worker from the summed counters.
• Unwritable directory: WARNING at boot and fall back to today's one-loop-per-worker behaviour; never a crash. Empty variable: same fallback, by choice, at INFO.

D2 - plain-http collector URL announced ON, refused silently. check_endpoint_url ran only in send_once, at DEBUG, so the boot line said ON, GET said enabled: true, and nothing was ever sent or explained. The check now lives in policy.evaluate(): an invalid URL is Disabled(reason="invalid_endpoint_url"), the boot line prints OFF with the reason and the offending URL at WARNING, and GET shows enabled: false, the reason and the endpoint. The send-time check stays as defence in depth.

D5 / docs. README said "CI set to any value"; code and docs/telemetry.md say non-empty (an empty CI= keeps the heartbeat on). README now says non-empty.

Small fixes.

• (a) Delivery failures (connection refused, timeout, non-202) left last_status/last_sent_at null with no trace. The state and GET gain last_error (short, URL-stripped, so a credential in an override can never surface) and last_attempt_at; failures are logged once per cycle at WARNING instead of DEBUG.
• (b) CAURA_VERSION=v3.17.0, the pinning form docs/self-hosting.md recommends and which core-api reads from .env, was sent verbatim. The payload version and the User-Agent now strip a leading v.
• (c) docs/telemetry.md now states: one beat per container regardless of worker count (and how), source runs report dev unless CAURA_VERSION is set (and that the collector excludes dev), providers.* report the configured kind even when the server fell back to the fake provider, the new invalid_endpoint_url row, last_error/last_attempt_at, and the new boot lines. README kept consistent. .env.example documents CAURA_TELEMETRY_STATE_DIR.

TypeScript client 1.0.2. The published @caura/client (1.0.1) predates the User-Agent header from #1561, so its requests are counted as other. clients/typescript is bumped to 1.0.2 (package.json, package-lock.json, src/version.ts; the VERSION agrees with package.json test passes). Publishing needs tag caura-client-ts-v1.0.2 on the merge commit (publish-npm-client.yml).

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• Documentation update

How Has This Been Tested?

New tests:

• tests/test_heartbeat_state.py: lock acquisition and hand-over (including a real child process that dies holding the lock), counter-file aggregation, epoch reset, stale/malformed/dead-pid files, state.json round trip.
• tests/test_heartbeat_multiworker.py: two multiprocessing worker processes sharing a state directory against a fake receiver; asserts exactly one POST per cycle over two cycles, that the beat carries the summed counts (1 + 1 -> "2-5", then "0" after the epoch reset), identical status from both workers, v3.17.0 -> 3.17.0, and that shutdown releases the lock. Runs in about 5 s.
• Extended test_heartbeat_sender.py (roles, takeover, last_error, WARNING once per cycle, URL-free error text), test_heartbeat_policy.py (URL row, boot lines, install() role election and the unwritable-dir fallback), test_api_telemetry.py (new fields, a follower worker's GET, the bad-URL response), test_heartbeat_payload.py (version normalisation).

Gates, run as in ci.yml from a fresh uv venv -p 3.12 with all four packages editable: ruff check + ruff format --check on core-api/src and tests/; cd core-api && mypy src/; scripts/tenant_scope_gate.py --base origin/main; scripts/legacy_name_ratchet.py --base origin/main (no new lines); pytest tests/ -m "not benchmark" against a pgvector/pgvector:pg16 container with CI's env after alembic upgrade head; clients/typescript: npm install + npm test (33 pass).

Docker check: built core-api/Dockerfile from this branch and ran it through docker-compose.yml (the real CMD, two uvicorn workers) with a receiver sidecar sharing the container's network namespace on 127.0.0.1:8010. Boot log: one anonymous heartbeat ON line (worker pid 10) and one anonymous heartbeat follower line (worker pid 9), where 3.17.0 printed two ON lines. Six GET /api/v1/telemetry calls spread over both workers returned the same deployment_id and next_send_at (2026-09-19T17:16:29Z); the receiver saw exactly one POST, at 17:16:29Z, whose clients_24h.caura-client-python was 6-20 (those six calls, summed across both workers). After the beat both workers reported last_sent_at 17:16:29Z, last_status 202, last_error null and the same next_send_at. The state directory held leader.lock, clients-9.json, clients-10.json, state.json and, after the send, epoch.

Checklist

• I have read CONTRIBUTING.md
• I have added tests that cover my changes
• ruff check and ruff format --check pass
• mypy passes
• pytest passes locally
• I have updated relevant documentation (README, docs/telemetry.md, .env.example)
• CHANGELOG.md is generated by release-please from the squash title

Additional Notes

• Separate containers (replicas) still each send; the collector's 20 h dedupe per deployment covers that case, as before. The lock is per container because /tmp is.
• docs/telemetry-schema-v1.json is unchanged: no payload field was added or renamed.

[github-actions]

Claude Code Review — skipped: PR author 'erni-a' is not a public member of the 'caura-ai' org

[erni-a]

@Eldad-Caura Please approve