Hold established dark tracks against outlier solves (TRACK_KF_OUTLIER_MODE, TRACK_KF_R_SOURCE)

[jehanazad]

Why

Dark-lane feed entries more than 5 km from any aircraft ("ghosts") are 10–14% of dark entries
(test droplet, 20-min captures 15 and 17, 2026-09-05). Tracing each ghost frame to the solve that
governs it: 58% sit on keys that had already accumulated four-or-more-node solves, and what
moved them was a two- or three-node solve joining by proximity. Those joins are wrong far more
often than their node count suggests:

nodes	joins landing >3 km off
2	36–62%
3	25% (44% of mints)
4	5%
5+	2%

Nothing at solve time separates a good one from a bad one: a three-node free-altitude solve is
exactly determined (six measurements, six unknowns), so its rms is ~0 whether it is right or
wrong. What can separate them is the track — a key with several four-plus-node solves behind it
has a Kalman state good to a few hundred metres, and a 4 km innovation is not a manoeuvre
(250 m/s × 5 s = 1.25 km). Today services/track_filter.py lets it through anyway: _measurement_R
trusts every solve at a flat 1,200 m base regardless of node count, and a gate breach re-anchors
the whole filter at the outlier. Measured: a bad three-node join onto a four-plus key has raw
error 4.1 km and published (smoothed) error 3.3 km.

What

Two env-flagged behaviours in services/track_filter.py, both defaulting to today's behaviour,
both read per call like TRACK_SMOOTHER:

• TRACK_KF_R_SOURCE = flat (default) | uncertainty. uncertainty takes _measurement_R's
  additive base term from solve_uncertainty.solve_sigma_m(result, dark=…) — the same calibrated
  number the map draws its uncertainty disc from (floors 650/210/180 m at n=2/3/≥4, ×1.5 dark) —
  instead of the flat 1,200 m. The inflated formal cov_en_km2 term, the PSD rejection and the
  final clamp are untouched; only base changes.
• TRACK_KF_OUTLIER_MODE = reanchor (default) | hold. In hold, on dark keys only, a
  gate breach (χ² > 13.8) on an established entry (≥ TRACK_KF_HOLD_MIN_UPDATES = 2 accepted
  position updates) whose innovation exceeds TRACK_KF_HOLD_MIN_INNOV_M = 1500 m is held: the
  update is not applied, the anchor does not move, the state advances by the coast, and the solve
  is published at the predicted position — the key is refreshed but does not move.
  TRACK_KF_HOLD_MAX_STREAK = 3 consecutive holds concede and re-anchor (the new position keeps
  being confirmed, so the identity really changed). A good solve in between clears the streak.

A measured note that matters for how these ship: under the flat base an innovation must reach
~5 km before it breaches χ² at all, which is already outside the solver's 6 km proximity gate — so
with a flat R there is almost no in-gate bad join for hold to act on (which is also why today's
bad joins arrive as smoothed 3.3 km errors rather than re-anchors). Tightening the base to the
calibrated per-solve sigma is what lets the gate see the 4 km population in the first place. The
test overlay will therefore run hold + uncertainty together for a capture.

Measurement

Every _smooth_kf return path now stamps kf_action
(init/passthrough/smoothed/held/reanchored), kf_d2 (2 dp, None when no innovation was
computed) and kf_innov_m (1 dp). solver.py's _record_solve_history copies all three onto the
history record beside kf_pos_sigma_m, and dark-lane kf_held/kf_reanchored counters are bumped
in the publish path and exposed in /api/test/solver-stats's fragmentation block. With both
flags at their defaults nothing changes but those three diagnostic fields, so a capture can say
what each policy would have shown for the same solves before anything is switched on.

Verified

• tests/test_track_filter.py — new TestRSource (8), TestHoldOutliers (9), TestHoldThroughTheSolver (2);
  all 29 pre-existing tests unchanged and green: 52 passed, 1 skipped.
• tests/test_solver_stats.py — 56 passed (two key-set pins updated for the new fragmentation
  fields, plus a new counter test).
• tests/test_mn_lifetime.py 23 passed, tests/test_solver_anchor.py 30 passed.
• Full backend suite: 2803 passed, 2 skipped in 144.46s.
• pre-commit run --all-files: all hooks pass (ruff check / format / vulture / config x2).

Nothing is deployed and no docker-compose file is touched. Stacked on #298
(fix/kf-covariance-psd).

🤖 Generated with Claude Code

[claude]

Claude finished @jehanazad's task in 8m 47s —— View job

---

Code Review

Tasks

• Gather context (PR body, changed files, diff vs origin/fix/kf-covariance-psd)
• Understand the request (custom instruction: run /code-review skill on this PR)
• Run code review skill
• Post findings

---

Findings

1. A "held" result still ships the outlier's altitude, velocity, and node-count-derived fields — backend/services/track_filter.py:762-767

held = dict(result)
held["lat"] = round(lat_h, 6)
held["lon"] = round(lon_h, 6)
held["smoother"] = "kf"
held["kf_pos_sigma_m"] = round(...)
return _stamp(held, "held", d2, innov_m)

held starts as a copy of the rejected outlier solve (result), and only lat/lon/smoother/kf_pos_sigma_m are overwritten with the track's own predicted values. alt_m, vel_east/vel_north, n_nodes, rms_*, and cov_en_km2 all pass through untouched from the discarded solve.

The whole premise of hold is that this solve is a wrong-aircraft proximity join — that's the failure mode the PR measures (58% of ghost frames sit on established keys, moved by a bad 2–3 node join). If that's true, the held entry publishes the established track's correct position paired with the wrong aircraft's altitude and solved velocity. Downstream, aircraft_feed.py:130-131 takes alt_m straight from r, and aircraft_feed.py:173 sizes the map's uncertainty disc via solve_sigma_m(r, dark=...), which reads r["n_nodes"] from the outlier solve rather than anything derived from kf_pos_sigma_m/kf_action. A track cruising at 9,000 m could show a held (correct) position but a spurious 3,000 m or 15,000 m altitude on the same frame.

None of the new tests (TestHoldOutliers, TestHoldThroughTheSolver) vary alt_m/vel_east/vel_north/n_nodes between the established line and the injected outlier, so this gap isn't exercised.

Fix this →

2. New env knobs aren't added to backend/.env.example — backend/services/track_filter.py:258-260, 364, 370

TRACK_KF_R_SOURCE, TRACK_KF_OUTLIER_MODE, TRACK_KF_HOLD_MIN_UPDATES, TRACK_KF_HOLD_MIN_INNOV_M, and TRACK_KF_HOLD_MAX_STREAK are new os.getenv-backed configuration, but none appear in backend/.env.example. CLAUDE.md is explicit: "Configuration lives in backend/.env... Add new keys to backend/.env.example so the list stays current." .env.example does document other tuning knobs elsewhere (e.g. solve_uncertainty.py's SOLVE_SIGMA_* block around line 195-201) with the same "commented out with inline docs" style — worth noting this file's existing TRACK_KF_*/TRACK_SMOOTHER knobs also aren't listed there, so this PR is following a pre-existing gap in this module rather than introducing a new pattern, but it's still a growing list of undocumented operational levers for a behavior this PR itself says will be switched on for a test capture.

3. Sigma formula and ENU→latlon conversion duplicated between held and smoothed branches — backend/services/track_filter.py:766, 810

held["kf_pos_sigma_m"] = round(math.sqrt(max(0.0, 0.5 * (p_pred[0, 0] + p_pred[2, 2]))), 1)
...
smoothed["kf_pos_sigma_m"] = round(math.sqrt(max(0.0, 0.5 * (p_upd[0, 0] + p_upd[2, 2]))), 1)

plus the matching offset_latlon_m(entry.ref_lat, entry.ref_lon, east_m=..., north_m=...) call in both branches. Minor, but a future change to the marginal-sigma formula (e.g. clamping behavior) applied to one copy and not the other would silently desync held and smoothed semantics. Worth factoring into a small helper, not blocking.

---

Everything else — the R-source calibration wiring, the gate/hold/streak state machine, the lock discipline, and the diagnostic stamping (kf_action/kf_d2/kf_innov_m) — reads as carefully reasoned and is well covered by the new TestRSource/TestHoldOutliers/TestHoldThroughTheSolver suites. Finding #1 is the one I'd want addressed before this ships with TRACK_KF_OUTLIER_MODE=hold turned on anywhere real.

[jehanazad]

Live measurement (cap20, test droplet, stack a351bdd, 00:32–00:52Z 2026-09-06, TRACK_KF_R_SOURCE=uncertainty + TRACK_KF_OUTLIER_MODE=hold) — do not flip these defaults.

Thin fleet (4–7 dark aircraft vs 11–12 in cap17), 600 frames, 1141 GT-matched dark records.

	cap17 (flat/reanchor, dense)	cap19 (flat/reanchor, thin)	cap20 (uncertainty/hold, thin)
shown 6+ / 4–5 / 2–3	98 / 87 / 39 %	100 / 89 / 43 %	96 / 87 / 49 %
ghost rate (dark feed entries >5 km from any aircraft)	13.9 %	3.3 %	22.8 %

KF action audit (per published solve, raw solve error vs published error):

action	count	of which bad raw (>3 km)	bad: raw → published p50	good: raw → published p50
held	31	12	5.5 → 6.6 km	0.13 → 4.0 km (N=9)
reanchored	6	3	5.1 → 5.1	0.18 → 0.18
smoothed	395	30	5.6 → 5.6	0.22 → 0.19

The hold fires on the wrong side of the ledger: 9 of the 21 held solves with ground truth were correct solves whose published position became the coasted prediction 4 km off, and the 12 bad ones were published at the prediction, which was itself 6.6 km off (the "established" track that earned the hold was already wrong). Uncertainty-weighted R did not stop bad n=3 joins on established keys either (30 smoothed bad solves published at 5.6 km, unchanged from raw). Ghost causes are unchanged in shape: 73 % bad bottom-up solves (err>3 km), 13 % bad follow solves.

Verdict: flags stay at their defaults (flat / reanchor); this PR is not in the merge set. Test overlay reverted.

🤖 Generated with Claude Code

[jehanazad]

Closing per the 2026-09-10 review against main 886a342. The cap20 measurement above stands (ghost rate 3.3% -> 22.8%, hold firing on correct solves). The premise is also gone: #310 retries a gate breach with manoeuvre process noise before re-anchoring (a 4 km innovation scores d2~4 against the 13.8 gate and is rescued, not re-anchored), #329 already prices dark n=2 solves at 2500 m by default, and the dark sigma floors this PR's calibration cites are now 2100/850/240 m. Merging the hold would either pre-empt #310 on real turns or sit downstream where it never fires. The per-record kf_action / kf_d2 / kf_innov_m diagnostics are being carried forward on a fresh branch.