Skip to content

Calibrate empirical coverage from the claim lane, not the emit loop - #368

Open
jehanazad wants to merge 4 commits into
feat/synthetic-nodes-declared-wedgefrom
feat/calibration-from-claims
Open

Calibrate empirical coverage from the claim lane, not the emit loop#368
jehanazad wants to merge 4 commits into
feat/synthetic-nodes-declared-wedgefrom
feat/calibration-from-claims

Conversation

@jehanazad

@jehanazad jehanazad commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Empirical-coverage calibration has been effectively dead since #240 (2026-08-25) made KNOWN_LANE_MODE=binding the default. The only recorder was the tracker's emit loop, which needs an ADS-B-tagged detection to reach the tracker, and binding strips every claimed detection before it gets there. Measured on the test deployment's persisted bins on 2026-09-13: every synthetic node's newest calibration point is dated exactly 2026-08-25; radar3/radar3a get a trickle (newest 3–6 days old) made only of detections the claim lane refused but the greedy tagger accepted; the ret* TCP nodes have no calibration file at all. Every coverage polygon on every environment is a frozen pre-#240 picture drawn by the old greedy binder, and that binder was wrong about the aircraft roughly a third of the time (47 % of one 42° synthetic node's points lay outside a wedge the simulator never emits detections outside of).

This moves calibration into the claim lane under a rule much stricter than claiming, and silences the emit-loop path unless the lane is off. Real and synthetic nodes are treated identically here; which polygon is published is #367's business.

Stacked on #367 (base branch = its head); merge order: retina-analytics #28, #32, then #367, then this.

The rule (services/known_claiming.py::_calibration_from_claim)

A claim records a calibration point only if all five hold:

  1. it has a fresh transponder fix behind it: a node tag (path 1), a cached-fix assignment (path 2), or a hold whose fix was refreshed this frame (path H after _claim_holds' consistency check against a live fix); an unrefreshed hold or a follow claim never records;
  2. that fix is at most CAL_MAX_ADSB_AGE_S (10 s) old at the frame instant;
  3. the residual against the fix's prediction is within CAL_CLAIM_DELAY_US (3 µs) and CAL_CLAIM_DOPPLER_HZ (8 Hz), unscaled by age — about a tenth of the claim gate's area;
  4. the detection is uncontested: no dark projection inside the claim gate, and no other known hex whose prediction lies inside the gate (the candidate list is now built for the whole frame, not just the unclaimed detections, so every claim can be checked for rivals; claiming results are byte-identical);
  5. the link is mature: at least CAL_CLAIM_MIN_CLAIMS (3) claims of this hex by this node with no gap over CAL_CLAIM_STREAK_GAP_S (10 s).

The position recorded is the fix dead-reckoned to the frame instant, so the old fix-vs-detection skew rule does not apply; services/calibration.py::record_claim_calibration documents that as the module's fourth rule. Per-reason counters (calibration_points_recorded, calibration_claims_rejected_{hold,stale_fix,residual,contested,immature}) are in the stats payload beside the known_claims block, and the runbook says how to read them.

Measured offline (backend/scripts/calibration_attribution_bench.py)

Simulator world, 20 nodes, truth = the aircraft that actually produced each detection. points/node-min · wrong-hex · out-of-wedge, seed 7 / 240 s (seeds 42–44 in the commit body agree):

source blind node (path 2 + holds, the hardware case) tagged node (path 1)
old greedy tagger (the literal old recorder) 82.0 · 9.5 % · 9.3 % same
every claim, ungated 70.0 · 0.5 % · 0.1 % 74.5 · 0.4 % · 0.1 %
this rule 63.9 · 0.1 % · 0.0 % 68.2 · 0.1 % · 0.0 %

The first cut of rule 1 refused all holds and starved blind nodes to ~0.1 points/node-min because path H outranks path 2 from the second frame of every link; judging refreshed holds on their fresh fix (commit 2242780) restored the yield without loosening any constant.

Changes

  • services/known_claiming.py: candidate list built per frame for every claim's rival check; _fresh_fix_prediction returns the DR position; refreshed holds carry the fresh-fix prediction and DR position on extra (filtered out of the registry entry, which stays byte-identical); _calibration_from_claim + streak store; per-frame counter flush.
  • services/calibration.py: record_claim_calibration, fourth rule in the docstring.
  • services/track_gates.py: emit-loop recording only when KNOWN_LANE_MODE == "off".
  • config/constants.py: the four CAL_CLAIM_* constants with derivations. core/state.py, routes/test.py, vulture_whitelist.py: counters.
  • scripts/calibration_attribution_bench.py: the measurement above, --both, --hold-gap.
  • docs/pipeline.md §7 rewritten for the claim-lane source; docs/runbook.md funnel entry.

Test coverage

  • tests/test_calibration_from_claims.py (20 new): each rule's reject, the third-claim record, DR position, rival candidate, dark contention, node tags, streak reset, refreshed hold records / unrefreshed does not / refreshed judged on the fresh residual / registry entry carries no calibration keys; lane on ⇒ emit loop records nothing, lane off ⇒ unchanged.
  • Existing claiming, hold, follow, TCP-claim and track-gate tests pass unchanged; test_solver_stats.py updated for the two new keys in the known_claims stats block.
  • Full backend suite exit 0 (3536 passed, 1 skipped); pre-commit gate and identity guard clean.

Review notes

  • Yield is deliberately ~15 % below "every claim": rules 3–5 are what buy the 0.1 % wrong-hex rate. Tune constants only with the bench.
  • After deploy, calibration_points_recorded should climb within minutes on every claiming node, and the newest bin_pos_ts in coverage_data/empirical_<id>.json should move; live check on test owed.
  • Existing persisted bins are still the old greedy picture. They will be diluted, not replaced (200-point FIFO per bin). A one-off reset of coverage_data on deploy would give clean polygons faster; not done here — operator's call.

🤖 Generated with Claude Code

jehanazad and others added 4 commits September 13, 2026 23:43
@jehanazad @claude
Add a claim-lane recorder for calibration points, and its four gates
8241ad6 
services/calibration.py has held the one age rule and the one skew rule since
the frame and solve paths disagreed on them. It gains a third call site's
entry point — record_claim_calibration — and a fourth rule in the docstring
saying which call site may record at all.

The claim lane's recorder shares the age rule (at 250 m/s a 10 s fix is 2.5 km
stale against a 5-degree grid) and deliberately does not share the skew rule.
That is not an omission: its caller dead-reckons the fix to the frame instant,
so the fix and the detection it describes are the same instant by construction
and there is no skew left to bound. The test says so explicitly, because a
later reader would otherwise "fix" the missing gate.

The four constants are the gates the caller applies. 3 µs / 8 Hz against the
claim gate's 10 µs / 25 Hz: the simulator's measurement noise is sigma 0.1-0.2
µs and 2-4 Hz, so this is still over 5 sigma at the noisy end while shrinking
the delay x Doppler area a wrong aircraft can land in by ~10x. Three claims
with no gap over 10 s is the maturity bar, the counterpart of the emit path's
n_detections >= 3 — a wrong bind can be a coincidence once, not frame after
frame at a consistent (delay, Doppler).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jehanazad @claude
Calibrate empirical coverage from the claim lane, not the emit loop
b061050 
Empirical-coverage calibration has been dead since 2026-08-25 and nobody
noticed, because the thing that killed it was a default flip. KNOWN_LANE_MODE
went to "binding" in #240, and binding mode strips every detection the claim
lane binds out of the frame before the tracker sees it. The emit loop records
only when track.last_detection_adsb_hex is set, and that is set only from a
tagged detection the tracker actually saw — so for a synthetic node, whose
every detection is claimed, it is never set at all.

Measured on the test deployment's persisted bins 2026-09-13: every synthetic
node's newest calibration point is dated exactly 2026-08-25, 19 days old. Real
nodes get a trickle (3-6 days) because a few tagged detections survive
claiming — but those are precisely the binds claiming refused, which is
adverse selection, and it shows: across eight nodes 5-41 % of the points sit
outside the node's own declared wedge, and the simulator only ever emits a
detection inside it. synth-GVL-SCAT-0032, a 42-degree beam, held 3,037 points
of which 47 % were out of wedge.

So the claim lane becomes the one source whenever it runs, and the emit-loop
block is gated on mode "off" (where it is unchanged, rules and all). In shadow
mode the reason is different and just as decisive: nothing is stripped, so
both paths would record the same detection twice and the bin-count gates would
be reading a doubled denominator.

A claim is not automatically a calibration point. Claiming binds what it can
explain; calibration records what it could not have explained any other way,
and _calibration_from_claim charges every claim to exactly one of six
outcomes: recorded, or one of the five rules, in order. No hold or follow
claim (neither has a fresh transponder fix behind it). Fresh fix at the frame
instant. Tight residual, unscaled by fix age. Uncontested — no dark projection
inside the claim gate, and no OTHER known hex's prediction inside it either.
Mature link — three claims, no gap over 10 s.

Rule 4's second half is what the path-2 restructure is for. The candidate list
is now built for every frame carrying detections, and without skipping the
hexes paths 1 and H already took; the claimed_hexes skip moved to the
assignment's column build, so the cost matrix and every claim are exactly what
they were, while a path-1 tag's detection can finally be judged against the
rest of the cache. The published reject tallies are pinned to what the
ASSIGNMENT rejected, so widening the population does not inflate them.

The streak store is deliberately not the hold store: KNOWN_HOLD_MAX_GAP_S <= 0
is a supported rollback and calibration must not switch off with it. Every
non-hold claim advances the streak, including ones that fail rules 3-4 — those
are still evidence of the link, just not clean samples, and withholding them
would make a node's first clean sample its third, which on a noisy link may
never arrive.

The recorded position is the fix dead-reckoned to the frame instant, carried
from the assignment that computed it rather than re-derived, on a private
`extra` key filtered back out before the claim record is written — so
state.known_claims is byte-identical.

KNOWN GAP, pinned by a test rather than left to be rediscovered: path H
outranks path 2 and every claim creates a hold, so a node that sends no
frame["adsb"] is claiming through path H from the second frame of each link
onwards, and rule 1 refuses those. Such a link never matures. The whole
synthetic fleet sends tags (path 1 outranks path H) so it is unaffected;
KNOWN_HOLD_MAX_GAP_S=0 isolates it. See docs/pipeline.md for the numbers and
the fix.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jehanazad @claude
Measure claim-lane calibration attribution against simulator truth
69e43b0 
The five rules are only worth their cost if they buy attribution, so this
measures it rather than asserting it. Truth is exact, not inferred: the
simulator staples the producing aircraft's hex to each detection index, so the
frame read BEFORE the tags are stripped says which aircraft every detection
belongs to, and None means no transponder produced it — clutter or a dark
target — against which any hex is wrong by construction.

Three rules are scored side by side on the same frames. "greedy" is the
literal old source: associate_detections_to_adsb's tags, which is what set
track.last_detection_adsb_hex, recorded at the reported position, with no
one-to-one, no world gate and no visibility gate. "claims" is every claim the
lane makes recorded ungated — the fairer baseline, since the new rule is a
filter on exactly that population. "new" is the five rules, scored through
_calibration_from_claim itself so the bench reads the hex and the position the
rule actually accepted rather than re-deriving either.

20 synthetic nodes, 3 seeds, 5 simulated minutes each, ~26-30 aircraft:

                     wrong-hex    out-of-wedge   points/node-min
  greedy             8.9-11.0 %    8.1-10.0 %      69-100
  claims             0.9-3.5  %    0-1.3    %      57-92
  new                0.0-0.1  %    0.0      %      51-82

...except blind with path H enabled, where the yield collapses to 0.11-0.13
points per node-minute with 97 % of claims charged to rejected_hold. That is
the gap the previous commit documents, and --hold-gap is the lever that
isolates it; the 51-82 figures above are the tagged legs and the blind legs
with KNOWN_HOLD_MAX_GAP_S=0.

The bench swaps in its own in-memory NodeAnalyticsManager: the module-level
one is wired to backend/coverage_data, so registering there would load a
deployment's persisted bins into the run and write the run's points back out.

docs/pipeline.md section 7 is rewritten around the new source — the five
rules, the counters, the dead-since-#240 finding and the path-H gap — keeping
the emit-loop rules for mode off. The runbook gains the funnel and how to read
a flat `recorded`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jehanazad @claude
Calibrate from refreshed holds, so a blind node calibrates at all
2242780 
The previous commit shipped the claim-lane calibration rule with a measured
hole in it: a node that sends no frame["adsb"] yielded 0.11-0.39 calibration
points per node-minute against the 51-68 a tagged node gets, with ~97% of its
claims charged to rejected_hold.  That is every hardware receiver without its
own ADS-B correlation, which is the population the empirical polygon exists
for.

The cause was an ordering, not a judgement.  Path H runs before path 2 and
every claim creates a hold, so from the second frame of a link onwards the
claim is a hold claim; rule 1 refused it AND returned before the maturity
streak was touched, so the link could never reach CAL_CLAIM_MIN_CLAIMS.  Rule
1 was written for a hold with nothing behind it, and it was catching holds
with a live transponder behind them.

A hold carrying extra["fix_refreshed"] is not that hold.  The consistency rule
in _claim_holds has already compared this very detection against the LIVE
cached fix inside path 2's own age-scaled gate, and the claim carries that fix
rather than the hold's frozen one.  It is a path-2 claim in everything but
which path found the detection, so calibration now judges it as one: rule 2
against the fresh fix's fix_ts_ms (which is what the claim already carries),
rule 3 against the FRESH FIX's prediction rather than the hold's propagated
one, rule 4 against the same candidate population as any other claim, rule 5
advancing on it, and the position recorded is that fresh fix dead-reckoned to
the frame instant.

Rule 3's reference is the part that is not cosmetic.  A hold predicts this
node's next measurement from this node's last measurement, so its residual is
near zero whatever aircraft is actually out there — judging a calibration
point on it would be circular, and the 3 us gate would be measuring the
propagator rather than the attribution.  _fresh_fix_prediction therefore hands
back the dead-reckoned position it already computes alongside the prediction,
for the same reason path 2's candidate loop does: re-deriving it at the
recording site would be a second offset_latlon_m free to disagree with the one
the prediction was built from.  Both ride on the claim's private `extra` and
are filtered back out before the registry record is written, now through
_CAL_EXTRA_KEYS rather than a single-key comparison, so state.known_claims is
byte-identical and a test pins its exact key set.

A hold whose fix is gone, or has aged past KNOWN_CLAIM_MAX_FIX_AGE_S so
_fresh_fix_prediction stops answering, carries no fix_refreshed and is still
refused by rule 1 — both halves are tested, the second through the window
where the fix is merely stale (rejected_stale_fix) before it ages out
entirely (rejected_hold).

Re-measured with calibration_attribution_bench.py, 20 synthetic nodes, seeds
42/43/44, 5 simulated minutes, same seeds old and new (the old module run from
a shadow tree so both legs see identical traffic):

                          old rule (new)      this commit
  blind, holds ON      0.11-0.39 /node-min   54.5-61.3 /node-min
  blind, holds OFF     51.2-60.6             51.2-60.6  (unchanged: no holds)
  tagged (path 1)      57.6-67.5             57.6-67.5  (+0.1, 39-78 hold
                                                        rejects become
                                                        residual rejects)

Attribution did not pay for the yield: wrong-hex 0.0-0.3% and out-of-wedge
0.0% on the blind+holds leg, against 7.3-10.1% / 5.5-8.6% for the greedy tags
that fed the old emit path and 1.1-3.5% / 0-1.3% for every claim recorded
ungated, on the same frames.  The blind+holds leg's residual rejects rise
(363-394 vs 186-296 with holds off), which is exactly the fresh-fix residual
doing the work the hold's own residual could not.

docs/pipeline.md section 7 gains the refreshed-hold rule and loses the "known
gap" note; the runbook's funnel entry for a flat `recorded` with a large
`hold` now points at the transponder rather than at path H.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@claude

claude Bot commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude finished @jehanazad's task in 12m 7s —— View job

Claude finished the review

Todo list

  • Gather context (PR description, changed files, repo instructions)
  • Run /code-review (removed-behavior audit, cross-file trace, cleanup/efficiency/conventions pass)
  • Verify the strongest findings directly against the diff
  • Post review findings

Review — PR #368 (base feat/synthetic-nodes-declared-wedge)

Solid piece of work — the five-rule design and the offline bench are a real improvement over the frozen greedy-tagger polygons, and the yield/accuracy tradeoff is well-measured. Two things below look like genuine gaps worth a look before merge; the rest are lower-severity or optional.

1. Detection-range recording silently dies fleet-wide under the new default (services/track_gates.py:758)

if nid and _detection_fresh and state.KNOWN_LANE_MODE == "off":
    _n_recorded = record_adsb_calibration(...)
    if _n_recorded > 0:
        area.record_verified_detection(adsb_lat, adsb_lon, ac_hex)

record_verified_detection (feeds "detection range") lived in the same gated block as the calibration recorder. Adding state.KNOWN_LANE_MODE == "off" to the outer condition means this call no longer fires under binding (the default since #240) or shadow. No caller in known_claiming.py/calibration.py replaces it — the only other call site (services/tasks/analytics_refresh.py:1125) is a separate periodic ground-truth job, not a per-frame substitute. The new test TestKnownLaneSilencesThisPath confirms this is intentional, but neither docs/pipeline.md nor docs/runbook.md mention detection-range dying alongside the calibration-point migration — worth a documentation line at minimum, since it's a named operator-facing concept going silently to zero. (Impact is softened since routes/test.py's node_detection_range already doesn't publish furthest_detections today.)

Fix this →

2. Path-1 (node-tagged) calibration points can never fail the freshness rule (services/known_claiming.py:1073)

# taken as current — no dead-reckoning, fix_ts_ms = frame time.
...
fix = {..., "fix_ts_ms": ts_ms, ...}

For path 1, fix_ts_ms is always set to the current frame's own timestamp rather than any real transponder-report time (a node tag carries no last_seen_ms). That makes fix_age_s in _calibration_from_claim (known_claiming.py:368) exactly 0 for every path-1 claim, so rule 2 (CAL_MAX_ADSB_AGE_S, the staleness gate) is structurally a no-op for this path — it can only ever reject path 2/hold claims. If a node's onboard ADS-B correlation goes stale (keeps re-tagging a last-known position after losing the transponder) there's nothing in the five rules that catches it for path 1; that's exactly the "exit-smear" failure mode the PR's own docstring describes fixing elsewhere. No test in test_calibration_from_claims.py exercises a stale/incorrect path-1 tag. This may be an accepted tradeoff (the node already did its own correlation "this frame"), but it's worth confirming that's deliberate rather than an oversight, since it means rule 2 protects two of the three claim paths only.

Fix this →

3. Unconditional per-frame candidate build (services/known_claiming.py, claim_known_targets) — efficiency, unmeasured

The if free:-gated candidate-building loop (haversine prescreen, offset_latlon_m, _point_in_beam, predict_observation per cached aircraft) was widened to run whenever delays is non-empty, not just when there's an unclaimed detection — needed to build rival candidates for the new exclusivity check (rule 4). For a frame where every detection is claimed via path 1/holds (free empty), this goes from zero work to a full scan every frame, for every node. col_cands re-applies claimed_hexes correctly so no correctness issue was found, but there's no benchmark or perf test measuring the added cost, and no short-circuit for frames where no claim this cycle is even calibration-eligible. Worth a quick sanity check against production frame-rate budgets, especially for real (non-blind, fully-tagged) nodes.

Minor / optional

  • _CAL_DR_KEY/_CAL_PRED_KEY smuggle DR position and prediction through the claim's extra dict and then filter them back out before the registry write (known_claiming.py ~1326). Works, and keeps claims' tuple shape unchanged, but it's a round-trip whose only purpose is avoiding a signature change — an explicit side-channel (e.g. extending the claims tuple, or a parallel dict keyed by claim index) would remove the "future extra key collides with _CAL_EXTRA_KEYS" risk entirely.
  • backend/scripts/calibration_attribution_bench.py's _spy monkeypatches the module-global kc._calibration_from_claim to observe accepted points — fragile to a future rename/refactor and invisible to type checkers. An optional on_recorded callback threaded through claim_known_targets would give the bench the same visibility without patching internals. Non-blocking since it's a standalone measurement script, not production code.
  • Soft observation: _claim_streaks/_touch_claim_streak/_prune_claim_streaks is now the third hand-rolled "streak dict + gap-based expiry + prune" pattern in this area (alongside known_track_holds and dark_follow.py's reject/cooldown dicts). The PR's comment already gives a defensible reason not to reuse the hold store here (rollback via KNOWN_HOLD_MAX_GAP_S <= 0 must not also disable calibration maturity) — just flagging for a future shared helper, not something this PR needs to fix.

Checked, no issue found

  • _fresh_fix_prediction's 4→6 tuple change: single real call site (_claim_holds), correctly unpacked; no other caller depends on the old shape.
  • Counter names (calibration_points_recorded, calibration_claims_rejected_{hold,stale_fix,residual,contested,immature}): consistent across state.py, known_claiming.py, routes/test.py, vulture_whitelist.py — no typos.
  • counts_as_reject reproduces the pre-diff known_claims_visibility_rejects/world_rejects semantics exactly despite the widened candidate population; path-1/H hexes can't be double-claimed by path 2 (also covered by test_a_path1_hex_is_never_assigned_by_path2).
  • No CLAUDE.md rule implicated: no /v1/nodes route/model/node_config.py change (contract regen n/a), no new env keys (constants are plain module constants like their neighbors), no host/address references added.
    · branch feat/calibration-from-claims

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jehanazad