Dark solver phase 5: stale-track exclusion and measurement epoch alignment - #296
Conversation
Bumps libs/retina-analytics to the branch commit that stops register_node and rebuild_zones_for building an overlap zone between two nodes in different worlds (offworldlabs/retina-analytics#25). This fleet is the reason: 50 synthetic nodes and 8 receivers share one footprint over Greenville, so the associator held a zone for all 400 sim/real pairs — 39 of them a live adjacency edge — and a pairing drawn from one of those grids can only ever match a simulated tracklet against a real echo. Real node ids showed up in 48 of 156 synthetic dark solver records over a 7-minute window. node_world is already the single authority for the question and is already injected as node_world_provider, so nothing new is wired here. /api/radar/association/status gains assoc_world_skipped_pairs next to overlap_zones: without it a fleet whose cross-world pairs are being refused reads exactly like a fleet whose pairs never overlapped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ghost rate cannot see the failure the cluster-merge rework targets. A solver input that merges two aircraft usually still solves within MATCH_KM of one of them, so it counts as *matched* while carrying 4-5 km of position error and a node that never saw the aircraft it was published as. Nothing in the bench reported that, so the fix had no acceptance criterion. --mode track now scores every solver input association emits, ahead of the solve and ahead of every gate below it, against a truth side-channel: which aircraft actually produced each detection. That channel cannot come from the frame's own ADS-B list -- the simulator appends None there for every aircraft with has_adsb False, and dark aircraft are exactly the population under study -- so each detection is matched back to the aircraft whose noiseless (delay, doppler) it is nearest, at a gate ~5 sigma wider than the simulator's own measurement noise. It is built before _strip_adsb and never reaches association, so the blind discipline is intact. Reports contaminated_inputs_pct and foreign_nodes_per_input, per seed and pooled, alongside the ghost rate and matched error already there. Also adds the cluster-merge knobs (--merge-dist-km, --pair-vel-exclusive, --merge-vel-consistent and the two velocity thresholds) so each sub-step of the rework can be swept on its own; each defaults to None, meaning "leave the library's default alone", so a plain run measures what the library currently ships rather than freezing today's values into the bench. Pins retina-analytics at fix/cluster-contamination. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two independent changes to the same enqueue-and-solve path. 1. SOLVER_ALT_MODE (core/state.py, default "sweep" — nothing changes until it is set). The n>=3 altitude sweep solves once per fixed layer and keeps the lowest rms_delay. Its layers are 2 km apart, so the altitude it pins is systematically up to 1 km wrong, and on noise-free replay of this fleet's geometry that quantisation ALONE left rms_delay at a 1.76 us median against the 3.0 us reject gate — most of the gate's budget spent on an altitude the measurements themselves determine, and the residual left over then blamed on nodes, so _trim_and_resolve drops measurements that were never the problem. In "free" mode _solve_best_altitude makes ONE pool call to the geolocator's solve_multinode_multistart with three start layers (the one nearest the association guess and its two neighbours), altitude solved as a sixth unknown — against the sweep's six round trips, each of which pickles the node configs for a child. n=2 is untouched: altitude is unobservable there, so _solve_best_altitude_n2 keeps its single pinned solve and the geolocator pins regardless of the flag. Trimming re-solves through _solve_best_altitude, so a trim round inherits the mode its first solve used — mixing a swept altitude with a free one would make the rms it compares between rounds a different quantity. Both modes stamp altitude_mode on every history record, published or rejected, and free adds the per-start residuals and z_saturated. That is the whole comparison channel: one deploy of each, read off /api/test/mlat-history. There is deliberately no shadow mode — the two produce the same shape of result, so running both would double the solver's cost to learn what a deploy of each already says. 2. The frame path shipped ALL 58 node configs with every solver input. The pool is a spawn pool, so that whole set is pickled per solve while a candidate carries 2-8 measurements. configs_for_solver_input restricts it to the measurement node ids; nothing downstream needs the rest — the solver builds NodeSetups from the measurements, trimming and consensus only ever narrow that set, the beam gate iterates contributing_node_ids, and cv_epochs is built from the same matched nodes in all three input shapes association emits. The known lane fetches its own configs and is unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ruff format wrapped the truth-channel lookup and SIM910 wanted the redundant None default off dict.get. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three of the signals the dark-lane recovery work is judged on could not be read from a running server, so every judgement about it has been offline replay. This adds no behaviour: nothing about which candidates solve, which solves publish, or what reaches the map changes. 1. /api/test/mlat-history caps per lane. The known lane writes ~16x the dark lane's volume, so the flat records[:1000] answered a 30 min request with the newest ~6 min of dark records and the rest of the window read as a quiet period — the same failure the shared deque had before #289 split it, moved into the reader. ?lane= narrows to one lane, ?limit= (max 5 000) is applied per lane, and lane_counts is reported pre-cap so truncation stays legible. 2. Resolve-slot skips are recorded, not just counted. Live the rule refuses ~1 537 candidates per 646 dark attempts per 30 min, and nothing said WHICH claim did it — so a skip that suppressed a duplicate and one that suppressed a different aircraft (tracker track ids are shared across candidates) were indistinguishable. A separate 500-entry deque carries the blocking claims; solver-stats windows it as resolve_skips, and ?kind=resolve_skips dumps it. Its own deque and not the solve history: skips outrun dark records two to one and would evict them. 3. Contamination is measured live. A dark record matched to ground truth now says which of its own contributing nodes could not see that aircraft (foreign_node_ids/contaminated), using the associator's own visibility predicate — the same gate known-lane claiming uses, so the two cannot drift apart. The GT trail lookup already happened; this costs one cone test per node. Records nothing could be asked about stay out of the denominator rather than counting as clean. 4. NODE_FRAME_MIN_INTERVAL_S drops are counted. frames_dropped is published but is the queue-saturation counter and reads zero throughout; the frames the per-node rate limiter refuses were uncounted entirely, so "the tracker sees what this node sent" looked true from every metric. docs/solverflow.md's file:line references are replaced with file + symbol — every one of them had drifted — and gain a section on reading these endpoints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The suppression rule reads "this aircraft is already on the map at this width, at every track it is built from". _claim_resolve_slot recorded the claim BEFORE the solve, so a candidate that never reached the map still made that statement — and never released it. Two consequences, both live: * A rejected candidate blacked out its own identical twin for the full 12 s. 24% of dark attempts are rejected, and the retry that would have published was suppressed by the failure. * Tracker track ids are shared across the association candidates of DIFFERENT aircraft (74 of 178 ids in a 6 min window appeared in solves of more than one ground-truth aircraft — the finding that forced _supersession_match's spatial guard in #290). So a contaminated superset that the gates sank also suppressed the clean subsets behind it, including its neighbour's only candidate. Measured on the test droplet: ~1,537 skips against 646 dark attempts per 30 min. The mechanism refused more than twice as many candidates as it solved, for aircraft it had put nowhere. _claim_resolve_slot splits into a pure _resolve_slot_covered (read-only, run before the solve, also returning the blocking claims for the skip record) and _record_resolve_slot, called only on the publish path with the POST-TRIM survivors — result["source_track_ids"], which _filter_s_in_to_nodes rebuilds from the surviving track_ids_by_node. A trimmed node's track is deliberately left unclaimed: it contributed nothing to the published position and was probably another aircraft's, so claiming it would suppress that aircraft on the strength of a measurement this solve threw away. The rule itself is unchanged: every track covered at no fewer nodes within _SOLVER_RESOLVE_INTERVAL_S, widest claim wins, same pruning sweep. No negative claim for rejects — measure first. The cost is that the check no longer claims under the same lock, so two workers can both solve duplicates that arrived together. That is one extra solve, arbitrated downstream by keying and supersession, against the starvation above. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The candidate-level rate has a denominator association itself moves. Splitting one contaminated cluster into two clean sub-clusters plus the false pairing it was hiding emits three inputs where there was one, and the false pairing on its own scores as 100% contaminated -- so the rate rises while what reaches the map gets cleaner. Measured on the 50-node scatter scene: 20.3% -> 25.7% of candidates, with the ghost rate by solve falling 3.2% -> 2.4% over the same change. So the same score is now also taken at the publish point, over solves that cleared every gate and bound to a real aircraft. That is the population the live audit sampled (45% of published dark solves carried a foreign node), and its denominator is fixed by the aircraft in the sky rather than by how many hypotheses association chose to emit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/api/radar/association/status filed track_pairs_superseded under "track_pairs_inline_only", documented as permanently zero in production because the only exclusivity stage ran on a chi2 that cv_fit=None never computes. The deferred path now has an exclusivity stage of its own, so the counter moves on a live fleet and that grouping would be a lie. Moved to the live block alongside the new cluster_splits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Picks up offworldlabs/retina-analytics@7e5414a, which extends the cluster partition from same-node conflicts only to every group, bounding each emitted solver input at the merge distance. Foreign nodes per published solve 0.80 -> 0.32 on the 15-node ring bench scene. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two independent corrections to the same failure: legitimately in-cone nodes being trimmed out of dark solves because their measurement did not describe the moment the solve claims to. Stale-track exclusion (TRACK_MAX_STALE_S, default 3.0 s). A COASTING track survives N_DELETE=10 frames past its last association, and at the fleet's 0.74-1 Hz per-node cadence that is up to ~13 s of dead reckoning. confirmed_track_views hands association the track's newest REAL sample and association hands the solver that sample as the node's current measurement, so an aircraft that has flown out of a node's beam keeps contributing seconds-old delay to n>=3 solves. Measured on the test droplet: 230 out-of-cone nodes reached published dark solves in 20 min, median 4 deg outside the beam edge (p90 22 deg) and 1.8 km beyond max range. The freshness test reads the newest entry from get_recent_detections, which returns only ASSOCIATED samples, so it is the last real detection's time rather than a coast count — and it is compared against the frame timestamp being processed, never wall clock, so replays and backfills are unaffected. Epoch alignment (SOLVER_EPOCH_ALIGN, default on). The solver's residual model evaluates every measurement against one target state; nodes sample at independent phases, so it is fitting a set that spans up to a frame interval. align_measurement_epochs dead-reckons each delay onto the newest sample's time along its own measured Doppler, using d(delay_us)/dt = -doppler_hz * 1e6 / fc_hz — the rate implied by the geolocator's own residual model, in which positive Doppler is a closing target whose delay is decreasing. All-or-nothing per input: a partially aligned set just relocates the error, so a missing t_s, doppler_hz or fc_hz skips the input and counts solver_epoch_align_skipped. Both flags exist to be turned off live rather than rolled back. The known lane reuses the same helper: it does NOT have one epoch either (_CLAIM_SPREAD_S admits claims up to 5 s apart, ~1.5 km of motion at 300 m/s), and it is the lane whose residual is the fleet's accuracy measurement. Pins retina-analytics feat/measurement-epochs (PR #27), which carries t_s on each solver-input measurement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r more The free mode shipped with three start altitudes on the reasoning that freeing z removes the ladder's quantisation but not the LM's locality, so several starts are still what keeps a solve off the wrong side of a bistatic ellipse. On this fleet's geometry there was almost nothing for them to keep it off. Over a 20-minute window on test, 1019 free-mode solves: the three starts' rms_delay differed by more than 0.1 us in 13 of them (1.3%), and the nearest-layer start — the one the sweep would have pinned at — was more than 0.5 us worse than the best start in 2. So the extra two starts bought ~0.2% of solves a rounding error at three times the solver CPU, and solver CPU is now the constraint: ~1.7 attempts/s against a 2.0 s average latency on two pool workers. The count becomes SOLVER_FREE_ALT_STARTS, default 1, read per call beside the mode flag and clamped into [1, len(layers)] by _free_alt_starts against the ladder that module owns. One start is the layer nearest the association guess, or the guess altitude itself when the input carries a non-layer one (ADS-B) — the same splice the sweep does, so the one exact altitude a candidate has is still what it starts from. Above 1 the window is unchanged, so 3 reproduces what was measured; the knob stays because locality is a property of the geometry, not of this fleet, and a deployment whose nodes sit nearer an ellipse should not need a code change to buy the starts back. alt_starts_km / alt_start_rms_us keep being recorded — a one-element list rather than three — so the live comparison channel is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The sign of the Doppler-to-delay-rate conversion is the part of this that cannot be checked by reading it, so it is checked against the simulator's own _bistatic_delay / _bistatic_doppler: a target is flown in a straight line, sampled at two times, and the correction applied to the older sample must land on the newer sample's true delay. Three velocities (inbound, outbound, crossing-with-climb), plus a direct assertion that the correction and the true delay change share a sign, so a failure says "the sign is wrong" rather than "the error did not shrink enough". The staleness tests express the coast as a gap between the newest ASSOCIATED sample and the frame time, which is exactly what the tracker produces — get_recent_detections skips the None measurements mark_missed appends — and pin that TENTATIVE exclusion is unchanged and does not feed the stale counter, so the counter keeps meaning "an aircraft left this node's cone". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
Picks up offworldlabs/retina-analytics@2b83871. The 6.0 / 4.5 / 3.0 km sweep on the association bench is monotone with the real-track count flat at every point, so 3.0 is taken: published contamination 32.0% -> 25.1%, foreign nodes per published solve 0.76 -> 0.52. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Picks up offworldlabs/retina-analytics@67f1488 (ruff-format only, no behaviour change). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r-observability docs/solverflow.md: keep this branch's symbol-named 'Defined in' column and add #292's SOLVER_ALT_MODE / SOLVER_FREE_ALT_STARTS rows and section in it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…/resolve-slot-claim-on-publish
…is merged there Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r-observability
…/resolve-slot-claim-on-publish
…ix/cluster-contamination test_analytics_routes.py: keep both new status tests (live counters from this branch, world-skipped pairs from #291). libs/retina-analytics: point at analytics main (#25 world gate + #26 cluster split + #27 epochs merged). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…urement-epochs state.py / routes/test.py / solver.py / test_solver_stats.py: adjacent additions from both sides kept (epoch-align flag + counters next to the free-altitude flags, resolve_skips_dark and altitude_mode stamps). libs/retina-analytics: analytics main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Claude finished @jehanazad's task in 9m 50s —— View job Review: PR #296 — stale-track exclusion and measurement epoch alignment
SummaryThis is a re-review after the branch's rebase/sync; the substance matches the prior pass on this PR (both mechanisms are narrowly scoped, default-on with a kill switch, and well-instrumented). I ran a fresh high-effort pass over the full diff and verified two concrete, previously-unflagged issues directly against the code: Findings1. "guess_lat": round(float(ig["lat"]), 6) if ig.get("lat") else None,
"guess_lon": round(float(ig["lon"]), 6) if ig.get("lon") else None,
2. New env vars missing from Everything else checks out
Neither finding blocks the core correctness of the stale-track or epoch-alignment logic — they're a diagnostic-data gap and a documentation gap, respectively. |
Two independent corrections to the same failure mode: legitimately in-cone nodes being trimmed out of dark solves because their measurement does not describe the moment the solve claims to. Both default on, both with an env override so they can be turned off live rather than rolled back.
Motivation (live measurements, retina-test, this morning, post cluster-contamination + free-altitude fixes)
N_DELETE=10frames andconfirmed_track_viewskeeps COASTING, so a last real detection several seconds old (0.74–1 Hz per-node cadence) is handed to the solver as current.rms_delaygate.Part A —
TRACK_MAX_STALE_S(default 3.0 s, 0 disables)confirmed_track_viewsnow skips a track whose newest real detection is older thanTRACK_MAX_STALE_Srelative to the frame timestamp being processed (never wall clock — the fleet replays and backfills, and a wall-clock filter would silently empty every view in those runs). TENTATIVE exclusion is unchanged.Which signal, and why:
get_recent_detectionsreturns only ASSOCIATED samples —mark_missedappendsNoneintohistory["measurements"]and the reverse scan skips those — sohist[-1]["timestamp"]is the last real detection's time. That is the honest signal, and strictly better than the coast count:n_missedcounts only frames this node happened to process, so the same n_missed means different wall-time on a node that stalled. No tracker change was needed.Skips count
state.tracks_stale_skipped.scripts/association_bench.pypasses the frame time too, so the bench keeps feeding association exactly what production does.Part B —
SOLVER_EPOCH_ALIGN(on|off, default on)align_measurement_epochs(s_in, node_cfgs) -> (s_in, meta)— a pure helper inservices/tasks/solver.py— dead-reckons every delay ontot0 = max(t_s)along its own measured Doppler, before consensus and before_solve_best_altitude, so every gate below judges the numbers the published solve is fitted to. It restampstimestamp_msto t0.All-or-nothing per input: a partially aligned set has no marker saying which measurements share an epoch, so it just relocates the error. A missing
t_s,doppler_hz, or nodefc_hzskips the input and countssolver_epoch_align_skipped._extraon every history record now carriesepoch_aligned: booland, when it ran,epoch_skew_s— so the live effect is readable straight from/api/test/mlat-history. Both counters are also on/api/test/solver-statsand the admin stats payload.Sign convention — verified, not asserted
Writing
d_tx/d_rxfor the TX→target and target→RX ranges, the geolocator's_residual_functionhaspred_delay = (d_tx + d_rx - baseline)/candpred_doppler = (fc/c)(v_tx + v_rx), wherev_tx/v_rxare velocity components along unit vectors pointing from the target toward each site. Moving toward a site shortens that leg, sod(d_tx)/dt = -v_tx, givingi.e. positive Doppler = closing = delay decreasing.
retina_simulation.world's_bistatic_delay/_bistatic_doppleruse the identical convention.The derivation is not what the tests check.
tests/test_epoch_alignment.py::TestSignAgainstSimulatorGeometryflies a target in a straight line through the simulator's own geometry helpers, samples the true delay at two times and the Doppler at the first, and requires the correction applied to the older sample to land within 20% of the true error on the newer sample's delay — for inbound, outbound, and crossing-with-climb velocities. A sign error doubles that error instead of shrinking it. A separate test asserts directly that the correction and the true delay change share a sign.fc_hzis read with the samefc_hz→FCfallback chain the geolocator uses to build itsNodeSetup, so a node aligns on exactly the carrier its solve predicts against.Known lane
It is not already single-epoch and it does reuse the helper.
_CLAIM_SPREAD_Sadmits per-node claims up to 5 s apart — ~1.5 km of target motion at 300 m/s — and this is the lane whose residual is the fleet's accuracy measurement (truth_match vs ghost). Its measurements now carryt_sfrom each claim's ownts_ms, and_attemptcalls the same helper behind the same flag. Itst0is by construction the newest claim's epoch, which is exactly the epoch its initial guess was already dead-reckoned to, so the accuracy classification stays self-consistent.Submodule
retina-analyticsPR (draft): offworldlabs/retina-analytics#27 — carriest_son each solver-input measurement from all three emitters (format_track_pairs_for_solvervia newTrackPairCandidate.t_s_a/t_s_b,_adsb_seed_round,_claim_round).t_swas already present on every history sample those builders read; the change only stops discarding it. Pinned here on branchfeat/measurement-epochs.Verified
tests/test_epoch_alignment.py(new, 17 tests) +test_frame_processor.py+test_solver_worker.py+test_known_lane.py+test_adsb_seed_backend.py+test_solver_stats.py: 260 passed-n 2 -m "not external")retina-analyticslib suite: 411 passedpre-commit run --all-files: all hooks pass (ruff check, ruff format, vulture, both ruff-config checks)Not verified: live effect on the droplet — nothing here has been deployed. The numbers above are the pre-change measurements that motivated it.
🤖 Generated with Claude Code