perf(route-cache): resilient SQL circuit recovery after 12h soak

[Boym323]

Why

The 12-hour Route Cache Production Soak V3 (2026-10-11 19:46Z to 2026-10-12 07:46Z, version 1.0.432 / e36f07b) recorded 680/721 valid samples, 41 diagnostic unavailability intervals, slow route-cache DB requests, frequent caller timeouts, and circuit breaker OPEN in roughly half the diagnostic samples. Values are from process-local recent samples, not SQL server-side execution latency. CPU/GC/event-loop were not measured.

Changes

• Keep the hard 750ms optional-cache caller deadline, but retain occupied SQL slots until real completion; no pretend SQL cancellation.
• Replace uncontrolled reopen after cooldown with a single half-open recovery probe after old requests release their slots.
• Exponential circuit backoff capped at 5 minutes, reset only after a timely successful request.
• Limit concurrent background cache writes to 2/4 slots; allow foreground reads to use remaining capacity, and run maintenance only when idle.
• Reduce cache-count maintenance sampling from every 5 minutes to every 15 minutes; skip it during degraded operation.
• Isolate optional maintenance COUNT/prune errors from the foreground route-cache breaker; they still remain observable in DB failure telemetry.
• Distinguish true SQL attempt timeouts from other errors in phase telemetry, and do not classify intentionally bypassed writes as SQL failures.
• Add targeted regression coverage for late SQL completion, recovery probe single-flight, exponential backoff, and write/read fairness.
• Expose probe attempts/failures, active DB writes, failure streak and half-open state in the existing System / Route Enrichment panel (CZ/EN), with telemetry regression coverage.
• Preserve OPEN reporting after cooldown expiry until a recovery probe actually succeeds.

Safety

No database migration, no credential change, no public API or SSE changes; optional cache remains fail-soft. Do not merge/deploy on CI alone: do DEV DB integration/performance validation and a gradual rollout before the next 12h production soak. Do not assert root cause of PostgreSQL latency based only on the sampled metrics.

Verification plan

• Targeted: npm run test:targeted -- tests/route-cache-inflight-timeout.test.ts tests/route-postgres-cache.test.ts tests/route-enrichment-telemetry.test.ts
• Typecheck, lint, full required CI gates.
• DEV with injected 1-10 second SQL latency and connection failure: verify that the live radar keeps streaming, active SQL <= 4, writes <= 2 and recovery probes <= 1.
• After deploy: compare circuit open ratio, caller timeouts/hour, postgresCacheRequest P95/P99, DB cache hits, diagnostic endpoint availability, RSS P95/peak and active job backlog against V3. Capture admin login 503 separately; the soak log cannot establish its HTTP status.

[Boym323]

Production release follow-up — 2026-10-12

Code merged to main: ec5d8c72403375a9796f3dee773890b3cbcdd426.
PR release-preview CI 38283678140, main release validation, full tests, performance regression and CodeQL all PASS.

Production deployment NOT confirmed / BLOCKED: main CI 38284560087, deploy job failed in attempt 1 and attempt 2 after 5 lock-contention retries each:
[AirRadar release] ERROR: Another AirRadar release is already running.
Release preflight passed. OLD_SHA/NEW_SHA were not captured, so deployment did not proceed beyond lock acquisition. No other in-progress GitHub Actions workflows were reported when checked; this does not rule out a manual host-side release.

Safe operational follow-up, on production host (read-only):

sudo lslocks -o PID,COMMAND,PATH | grep -F 'airradar-release.lock' || true
sudo fuser -v /var/lock/airradar-release.lock 2>&1 || true
ps -ef | grep -E '[r]elease.sh|[n]pm run build|[n]ext build'
systemctl status airradar.service --no-pager

Do not remove the lock file, kill the holder, or bypass flock without establishing whether a legitimate release is still active. An on-disk lock file alone does not mean the lock is held.

Once the other holder completes and the production host is healthy, re-run failed jobs for main CI 38284560087, verify deploy job PASS, /api/version shows ec5d8c72 or descendant, and post-release health checks. The uploaded 12h production soak is baseline only; no improvement has yet been proven on production PostgreSQL.