Instrument the dark lane: per-lane history, resolve-slot skips, live contamination

[jehanazad]

What

Instrumentation only. No behaviour change to which candidates solve, which solves publish, or what reaches the map — every change here is a counter, a record field, or a query parameter.

Phase 0 of the dark-lane recovery plan. The three signals the later phases are judged on are today unreadable from a running server, so every judgement about the dark lane has had to be offline replay of captured history.

1. /api/test/mlat-history: per-lane cap and lane filter

The known lane writes ~16x the dark lane's record volume. The flat records[:1000] therefore answered a 30 min request with the newest ~6 min of dark records — and the rest of the window read as a quiet dark lane rather than as a truncated answer. That is the same failure the shared deque had before #289 split it, just moved into the reader.

• ?lane=dark|known|adsb|all (default all), classified by the existing _record_lane.
• ?limit= (default 1000, max 5000), applied per lane (_cap_per_lane), so a known-lane burst can never evict dark records from a response.
• lane_counts in the payload is reported pre-cap, so a truncated records list stays legible.
• Existing response shape and fields are unchanged; ?hex=/?all=1 answer exactly as before at the defaults.

2. Resolve-slot skips become visible

Live the suppression rule refuses ~1 537 candidates per 646 dark attempts per 30 min, and solver_resolve_skips was the only trace. Nothing said which claim blocked a candidate — and since tracker track ids are shared across the candidates of different aircraft (#290's own finding), a skip that suppressed a duplicate and one that suppressed a different aircraft looked identical.

• New state.solver_resolve_skips_recent, a separate 500-entry deque (not the solve-history deque: skips outrun dark records roughly two to one and would evict exactly the solves an investigation needs). Each entry carries ts_ms, lane, track_ids, n_nodes, blocking: [{track_id, held_ts, held_n}], guess_lat, guess_lon.
• New solver_resolve_skips_dark counter (lane by the existing _is_dark_solver_input), beside the unchanged solver_resolve_skips.
• /api/test/solver-stats gains resolve_skips: {total, dark, attempts_ratio, window_effective_minutes}, windowed from the deque. attempts_ratio is all-lane skips over dark attempts — the shape the Phase 1 acceptance target is quoted in. window_effective_minutes is there because 500 entries against ~50 skips/min means a long window is truncated here even when the solve stores cover it.
• /api/test/mlat-history?kind=resolve_skips dumps the deque.
• Skips are still not counted as attempts or rejects in the funnel.

3. Live contamination metric

A DARK history record that matched ground truth now carries foreign_node_ids — the contributing nodes (including ones _trim_and_resolve dropped) for which the matched aircraft was not visible at solve time — plus contaminated: bool.

The verdict is the associator's own visibility predicate applied whole (retina_analytics.association._point_in_beam against the registered NodeGeometry), which is the gate known-lane claiming uses; claiming and the dark lane have to mean the same thing by "this node can see there", so no new geometric rule is introduced. Two documented consequences: it is a ground-projected test with no altitude term, and under FOV_MODE=active it is the learned FOV rather than the theoretical wedge — both being exactly what the rest of the pipeline believes about coverage.

Cost: no extra trail lookup (the GT stamp already resolved the truth point), one cone test per contributing node.

/api/test/solver-stats gains contamination: {records_with_gt, contaminated, pct, foreign_nodes_per_record}, dark-only and windowed. Records nothing could be asked about — no GT match, or no registered geometry for any contributing node — are left unstamped and out of the denominator, so an abstention never reads as innocence.

4. Frame drops

state.frames_dropped is already exposed (/api/admin/metrics), so it is not re-exposed here. But it counts frame_queue saturation and reads zero throughout — the frames NODE_FRAME_MIN_INTERVAL_S refuses per node were counted nowhere at all, so "the tracker sees what this node sent" looked true from every published metric. New node_frames_rate_limited counter at that gate, surfaced in solver-stats' counters block.

5. docs/solverflow.md

Every file.py:NNN reference replaced with file + symbol; all ~85 had drifted (frame_processor.py:294 for "process_one_frame entry" now lands mid-confirmed_track_views, for instance). New §7 documents the endpoints above and how to read the new blocks.

How verified

• tests/test_mlat_history.py tests/test_solver_stats.py tests/test_solver_worker.py: 204 passed (new: contamination stamp incl. the one-node-in-cone/one-out unit case, trimmed nodes judged, unstamped-abstention; lane filter and per-lane cap incl. the known-evicts-dark regression at 1/500 scale; skip deque, lane split, and "a skip is not an attempt or a reject").
• Full backend suite: 2805 passed, 2 skipped in 226.59s (-n 2 -m "not external").
• pre-commit run --all-files: all five hooks Passed.

Not deployed.

🤖 Generated with Claude Code

[claude]

Claude finished @jehanazad's task in 3m 11s —— View job

---

Claude's review of PR #293

Todo list

• Gather context (PR body, diff, CLAUDE.md)
• Understand request (re-review after synchronize)
• Review the diff across all changed files
• Post final review findings

---

Summary

This PR's own contribution — the dark-lane observability work described in the PR body — is unchanged in substance since the last review and still holds up: per-lane cap/filter in _cap_per_lane (backend/routes/test.py), the separate solver_resolve_skips_recent deque and _record_resolve_skip/_resolve_slot_blockers (backend/services/tasks/solver.py), the contamination stamp _stamp_foreign_nodes reusing _point_in_beam/NodeGeometry, node_frames_rate_limited at the correct gate in tcp_handler.py, and the docs/solverflow.md file:line → file:symbol rewrite. No new issues found in that code.

Note on this diff vs. the PR description

The PR is now 16 files / 1538 additions, up from the 8 files the previous review covered — but that growth isn't new instrumentation work. feat/dark-solver-observability is stacked on PR #292 (feat/free-altitude-solve, still open against main) and, transitively, #291 (feat/assoc-world-gate-overlap). Since neither has merged to main yet, GitHub's diff for #293 (computed against main) necessarily includes their commits too: SOLVER_ALT_MODE/SOLVER_FREE_ALT_STARTS, the free-altitude multi-start solve path, configs_for_solver_input (config trimming for the pool), and backend/tests/test_solver_alt_mode.py all belong to #292, not to this PR — I confirmed none of it exists on origin/main yet. That's expected stacked-PR behavior, not scope creep in this branch, but it's worth flagging so reviewers don't read the "no behaviour change" claim in this PR's body against code that #293 didn't actually introduce. Once #292/#291 merge and this branch rebases onto main, the diff should shrink back to the instrumentation-only set described above.

Still open (non-blocking, from previous review)

backend/services/tasks/solver.py — _record_resolve_skip (and the pre-existing _record_solve_history) still use ig.get("lat")/ig.get("lon") truthiness, so a guess sitting exactly on the equator or prime meridian (0.0) records as None instead of 0.0. Not a new bug, very unlikely to matter for this network's coverage area, not fixed in this push — still just a nit.

No behavior-affecting bugs found in this PR's own code.