Reject non-PSD solve covariance before it poisons the display filter - #298
Merged
Merged
Conversation
Bumps libs/retina-analytics to the branch commit that stops register_node and rebuild_zones_for building an overlap zone between two nodes in different worlds (offworldlabs/retina-analytics#25). This fleet is the reason: 50 synthetic nodes and 8 receivers share one footprint over Greenville, so the associator held a zone for all 400 sim/real pairs — 39 of them a live adjacency edge — and a pairing drawn from one of those grids can only ever match a simulated tracklet against a real echo. Real node ids showed up in 48 of 156 synthetic dark solver records over a 7-minute window. node_world is already the single authority for the question and is already injected as node_world_provider, so nothing new is wired here. /api/radar/association/status gains assoc_world_skipped_pairs next to overlap_zones: without it a fleet whose cross-world pairs are being refused reads exactly like a fleet whose pairs never overlapped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ghost rate cannot see the failure the cluster-merge rework targets. A solver input that merges two aircraft usually still solves within MATCH_KM of one of them, so it counts as *matched* while carrying 4-5 km of position error and a node that never saw the aircraft it was published as. Nothing in the bench reported that, so the fix had no acceptance criterion. --mode track now scores every solver input association emits, ahead of the solve and ahead of every gate below it, against a truth side-channel: which aircraft actually produced each detection. That channel cannot come from the frame's own ADS-B list -- the simulator appends None there for every aircraft with has_adsb False, and dark aircraft are exactly the population under study -- so each detection is matched back to the aircraft whose noiseless (delay, doppler) it is nearest, at a gate ~5 sigma wider than the simulator's own measurement noise. It is built before _strip_adsb and never reaches association, so the blind discipline is intact. Reports contaminated_inputs_pct and foreign_nodes_per_input, per seed and pooled, alongside the ghost rate and matched error already there. Also adds the cluster-merge knobs (--merge-dist-km, --pair-vel-exclusive, --merge-vel-consistent and the two velocity thresholds) so each sub-step of the rework can be swept on its own; each defaults to None, meaning "leave the library's default alone", so a plain run measures what the library currently ships rather than freezing today's values into the bench. Pins retina-analytics at fix/cluster-contamination. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two independent changes to the same enqueue-and-solve path. 1. SOLVER_ALT_MODE (core/state.py, default "sweep" — nothing changes until it is set). The n>=3 altitude sweep solves once per fixed layer and keeps the lowest rms_delay. Its layers are 2 km apart, so the altitude it pins is systematically up to 1 km wrong, and on noise-free replay of this fleet's geometry that quantisation ALONE left rms_delay at a 1.76 us median against the 3.0 us reject gate — most of the gate's budget spent on an altitude the measurements themselves determine, and the residual left over then blamed on nodes, so _trim_and_resolve drops measurements that were never the problem. In "free" mode _solve_best_altitude makes ONE pool call to the geolocator's solve_multinode_multistart with three start layers (the one nearest the association guess and its two neighbours), altitude solved as a sixth unknown — against the sweep's six round trips, each of which pickles the node configs for a child. n=2 is untouched: altitude is unobservable there, so _solve_best_altitude_n2 keeps its single pinned solve and the geolocator pins regardless of the flag. Trimming re-solves through _solve_best_altitude, so a trim round inherits the mode its first solve used — mixing a swept altitude with a free one would make the rms it compares between rounds a different quantity. Both modes stamp altitude_mode on every history record, published or rejected, and free adds the per-start residuals and z_saturated. That is the whole comparison channel: one deploy of each, read off /api/test/mlat-history. There is deliberately no shadow mode — the two produce the same shape of result, so running both would double the solver's cost to learn what a deploy of each already says. 2. The frame path shipped ALL 58 node configs with every solver input. The pool is a spawn pool, so that whole set is pickled per solve while a candidate carries 2-8 measurements. configs_for_solver_input restricts it to the measurement node ids; nothing downstream needs the rest — the solver builds NodeSetups from the measurements, trimming and consensus only ever narrow that set, the beam gate iterates contributing_node_ids, and cv_epochs is built from the same matched nodes in all three input shapes association emits. The known lane fetches its own configs and is unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ruff format wrapped the truth-channel lookup and SIM910 wanted the redundant None default off dict.get. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three of the signals the dark-lane recovery work is judged on could not be read from a running server, so every judgement about it has been offline replay. This adds no behaviour: nothing about which candidates solve, which solves publish, or what reaches the map changes. 1. /api/test/mlat-history caps per lane. The known lane writes ~16x the dark lane's volume, so the flat records[:1000] answered a 30 min request with the newest ~6 min of dark records and the rest of the window read as a quiet period — the same failure the shared deque had before #289 split it, moved into the reader. ?lane= narrows to one lane, ?limit= (max 5 000) is applied per lane, and lane_counts is reported pre-cap so truncation stays legible. 2. Resolve-slot skips are recorded, not just counted. Live the rule refuses ~1 537 candidates per 646 dark attempts per 30 min, and nothing said WHICH claim did it — so a skip that suppressed a duplicate and one that suppressed a different aircraft (tracker track ids are shared across candidates) were indistinguishable. A separate 500-entry deque carries the blocking claims; solver-stats windows it as resolve_skips, and ?kind=resolve_skips dumps it. Its own deque and not the solve history: skips outrun dark records two to one and would evict them. 3. Contamination is measured live. A dark record matched to ground truth now says which of its own contributing nodes could not see that aircraft (foreign_node_ids/contaminated), using the associator's own visibility predicate — the same gate known-lane claiming uses, so the two cannot drift apart. The GT trail lookup already happened; this costs one cone test per node. Records nothing could be asked about stay out of the denominator rather than counting as clean. 4. NODE_FRAME_MIN_INTERVAL_S drops are counted. frames_dropped is published but is the queue-saturation counter and reads zero throughout; the frames the per-node rate limiter refuses were uncounted entirely, so "the tracker sees what this node sent" looked true from every metric. docs/solverflow.md's file:line references are replaced with file + symbol — every one of them had drifted — and gain a section on reading these endpoints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The suppression rule reads "this aircraft is already on the map at this width, at every track it is built from". _claim_resolve_slot recorded the claim BEFORE the solve, so a candidate that never reached the map still made that statement — and never released it. Two consequences, both live: * A rejected candidate blacked out its own identical twin for the full 12 s. 24% of dark attempts are rejected, and the retry that would have published was suppressed by the failure. * Tracker track ids are shared across the association candidates of DIFFERENT aircraft (74 of 178 ids in a 6 min window appeared in solves of more than one ground-truth aircraft — the finding that forced _supersession_match's spatial guard in #290). So a contaminated superset that the gates sank also suppressed the clean subsets behind it, including its neighbour's only candidate. Measured on the test droplet: ~1,537 skips against 646 dark attempts per 30 min. The mechanism refused more than twice as many candidates as it solved, for aircraft it had put nowhere. _claim_resolve_slot splits into a pure _resolve_slot_covered (read-only, run before the solve, also returning the blocking claims for the skip record) and _record_resolve_slot, called only on the publish path with the POST-TRIM survivors — result["source_track_ids"], which _filter_s_in_to_nodes rebuilds from the surviving track_ids_by_node. A trimmed node's track is deliberately left unclaimed: it contributed nothing to the published position and was probably another aircraft's, so claiming it would suppress that aircraft on the strength of a measurement this solve threw away. The rule itself is unchanged: every track covered at no fewer nodes within _SOLVER_RESOLVE_INTERVAL_S, widest claim wins, same pruning sweep. No negative claim for rejects — measure first. The cost is that the check no longer claims under the same lock, so two workers can both solve duplicates that arrived together. That is one extra solve, arbitrated downstream by keying and supersession, against the starvation above. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The candidate-level rate has a denominator association itself moves. Splitting one contaminated cluster into two clean sub-clusters plus the false pairing it was hiding emits three inputs where there was one, and the false pairing on its own scores as 100% contaminated -- so the rate rises while what reaches the map gets cleaner. Measured on the 50-node scatter scene: 20.3% -> 25.7% of candidates, with the ghost rate by solve falling 3.2% -> 2.4% over the same change. So the same score is now also taken at the publish point, over solves that cleared every gate and bound to a real aircraft. That is the population the live audit sampled (45% of published dark solves carried a foreign node), and its denominator is fixed by the aircraft in the sky rather than by how many hypotheses association chose to emit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/api/radar/association/status filed track_pairs_superseded under "track_pairs_inline_only", documented as permanently zero in production because the only exclusivity stage ran on a chi2 that cv_fit=None never computes. The deferred path now has an exclusivity stage of its own, so the counter moves on a live fleet and that grouping would be a lie. Moved to the live block alongside the new cluster_splits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Picks up offworldlabs/retina-analytics@7e5414a, which extends the cluster partition from same-node conflicts only to every group, bounding each emitted solver input at the merge distance. Foreign nodes per published solve 0.80 -> 0.32 on the 15-node ring bench scene. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two independent corrections to the same failure: legitimately in-cone nodes being trimmed out of dark solves because their measurement did not describe the moment the solve claims to. Stale-track exclusion (TRACK_MAX_STALE_S, default 3.0 s). A COASTING track survives N_DELETE=10 frames past its last association, and at the fleet's 0.74-1 Hz per-node cadence that is up to ~13 s of dead reckoning. confirmed_track_views hands association the track's newest REAL sample and association hands the solver that sample as the node's current measurement, so an aircraft that has flown out of a node's beam keeps contributing seconds-old delay to n>=3 solves. Measured on the test droplet: 230 out-of-cone nodes reached published dark solves in 20 min, median 4 deg outside the beam edge (p90 22 deg) and 1.8 km beyond max range. The freshness test reads the newest entry from get_recent_detections, which returns only ASSOCIATED samples, so it is the last real detection's time rather than a coast count — and it is compared against the frame timestamp being processed, never wall clock, so replays and backfills are unaffected. Epoch alignment (SOLVER_EPOCH_ALIGN, default on). The solver's residual model evaluates every measurement against one target state; nodes sample at independent phases, so it is fitting a set that spans up to a frame interval. align_measurement_epochs dead-reckons each delay onto the newest sample's time along its own measured Doppler, using d(delay_us)/dt = -doppler_hz * 1e6 / fc_hz — the rate implied by the geolocator's own residual model, in which positive Doppler is a closing target whose delay is decreasing. All-or-nothing per input: a partially aligned set just relocates the error, so a missing t_s, doppler_hz or fc_hz skips the input and counts solver_epoch_align_skipped. Both flags exist to be turned off live rather than rolled back. The known lane reuses the same helper: it does NOT have one epoch either (_CLAIM_SPREAD_S admits claims up to 5 s apart, ~1.5 km of motion at 300 m/s), and it is the lane whose residual is the fleet's accuracy measurement. Pins retina-analytics feat/measurement-epochs (PR #27), which carries t_s on each solver-input measurement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r more The free mode shipped with three start altitudes on the reasoning that freeing z removes the ladder's quantisation but not the LM's locality, so several starts are still what keeps a solve off the wrong side of a bistatic ellipse. On this fleet's geometry there was almost nothing for them to keep it off. Over a 20-minute window on test, 1019 free-mode solves: the three starts' rms_delay differed by more than 0.1 us in 13 of them (1.3%), and the nearest-layer start — the one the sweep would have pinned at — was more than 0.5 us worse than the best start in 2. So the extra two starts bought ~0.2% of solves a rounding error at three times the solver CPU, and solver CPU is now the constraint: ~1.7 attempts/s against a 2.0 s average latency on two pool workers. The count becomes SOLVER_FREE_ALT_STARTS, default 1, read per call beside the mode flag and clamped into [1, len(layers)] by _free_alt_starts against the ladder that module owns. One start is the layer nearest the association guess, or the guess altitude itself when the input carries a non-layer one (ADS-B) — the same splice the sweep does, so the one exact altitude a candidate has is still what it starts from. Above 1 the window is unchanged, so 3 reproduces what was measured; the knob stays because locality is a property of the geometry, not of this fleet, and a deployment whose nodes sit nearer an ellipse should not need a code change to buy the starts back. alt_starts_km / alt_start_rms_us keep being recorded — a one-element list rather than three — so the live comparison channel is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The sign of the Doppler-to-delay-rate conversion is the part of this that cannot be checked by reading it, so it is checked against the simulator's own _bistatic_delay / _bistatic_doppler: a target is flown in a straight line, sampled at two times, and the correction applied to the older sample must land on the newer sample's true delay. Three velocities (inbound, outbound, crossing-with-climb), plus a direct assertion that the correction and the true delay change share a sign, so a failure says "the sign is wrong" rather than "the error did not shrink enough". The staleness tests express the coast as a gap between the newest ASSOCIATED sample and the frame time, which is exactly what the tracker produces — get_recent_detections skips the None measurements mark_missed appends — and pin that TENTATIVE exclusion is unchanged and does not feed the stale counter, so the counter keeps meaning "an aircraft left this node's cone". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Picks up offworldlabs/retina-analytics@2b83871. The 6.0 / 4.5 / 3.0 km sweep on the association bench is monotone with the real-track count flat at every point, so 3.0 is taken: published contamination 32.0% -> 25.1%, foreign nodes per published solve 0.76 -> 0.52. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Picks up offworldlabs/retina-analytics@67f1488 (ruff-format only, no behaviour change). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two map complaints, both traceable to numbers chosen when a dark aircraft was re-solved every ~12 s. Dark solves now land every 1-3 s while a track is held, which changes what a gap means: it is a lost track, not a cadence gap, and every budget sized as an allowance for the old cadence now buys wrong pixels instead of coverage. Measured on the test droplet over 20 minutes (dark multinode feed entries against ground truth), median position error by solve age: 1.05 km under 3 s, 1.21 km at 3-8 s, 1.50 km at 8-15 s, 2.02 km at 15-30 s (7% more than 5 km off), 3.99 km at 30-60 s (12% of all displayed dark entries, 32% more than 5 km off). Dead reckoning is cut at that curve's two knees. MN_DR_CAP_S (15 s, was a hardcoded 30) is how far an entry is extrapolated before it holds its last point; MN_DARK_EXPIRY_S (30 s) is when a dark entry leaves the feed entirely. mn-adsb-* entries keep the 60 s expiry -- a transponder hex anchors them, so the same gap there is the ADS-B feed breathing. The frontend's matching budgets move with them: the dark icon-hide distance 6 km -> 3 km (6 km was 90%-drawable at the 12 s cadence; now it is the width of the error it hides) and the uncertainty disc's growth cap 60 s -> 30 s, which is the age past which a dark entry cannot exist. The selected track's per-solve dots came from a 30 s poll, so at a 1-3 s solve cadence they showed a decomposition of the track that was mostly missing. The poll is now MLAT_HISTORY_REFRESH_MS (3 s), and a fall in the selected entry's `seen` -- the feed announcing a fresh solve -- refetches inside it. newSolveArrived is the pure predicate for that, unit-tested. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The test droplet threw 91 ValueError: math domain error tracebacks in 40 minutes out of learned_velocity's sqrt, from two callers that each lose real work: solver.py's multinode_key_decision (that solve is dropped) and aircraft_feed's multinode_to_aircraft (the whole feed broadcast for the tick is dropped). One sick track was taking down the map. Root cause is upstream of the sqrt. cov_en_km2 is the top-left 2x2 of s2 * inv(JtJ) for the solver's 5-state fit, and the solver falls back to pinv only on an outright LinAlgError -- an ill-conditioned-but-singular JtJ (near-parallel baselines, the degenerate tail that puts the formal sigma's p99 at 3.8e6 km) inverts to numerical garbage that is INDEFINITE while still having both diagonals positive. _measurement_R checked only shape, finiteness and positive diagonals, so that matrix became R. An indefinite R is not survivable: _kf_correct's Joseph form preserves PSD for any gain, but only GIVEN PSD P and R -- its K R K^T term inherits R's negative eigenvalue -- and _init_entry seeds P's position block straight from R, so a sick covariance poisons the filter at birth as well as on every update. Once P's velocity diagonals go negative, learned_velocity's sqrt raises. Three changes, defence in depth: - _measurement_R now requires the cov to be positive-semidefinite (both diagonals > 0 AND determinant >= 0), falling back to the base floor otherwise. That is already this function's documented answer for a degenerate cov, and a covariance this broken carries no trustworthy relative weighting worth repairing by eigenvalue clipping. - The predict step re-symmetrises P, as _kf_correct already did. - learned_velocity clamps its sqrt at 0, so a pathological state degrades to "sigma 0" rather than taking out a solve or a broadcast. The feed's per-entry build is now failure-isolated for the same reason: one entry raising logs once a minute and skips that aircraft instead of aborting the whole tick. Measured on origin/main across a 180-point sweep of indefinite covariances (correlation 1.2-10, formal variance 0.01-100 km^2, 1.4/5/20 s cadence, 0/100/250 m/s): 89 of 180 reproduce the exact math domain error through the public API, the fastest in 3 solves. All 180 pass after this change, with the worst P diagonal over the sweep at +71. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
solve_uncertainty.grown_sigma_m is the backend twin of the frontend's uncertainty.ts and exists so the two can be pinned to the same shape. Its horizon stays with UNCERTAINTY_DR_CAP_S: 60 -> 30 s. Its docstring claimed the frontend stops dead-reckoning at 60 s, which stopped being true in the previous commit, and a dark entry no longer reaches 60 s of age at all. No production caller -- the feed ships pos_sigma_m at the solve epoch and the frontend does the growing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every test here fails on origin/main and passes on this branch.
test_track_filter.TestIndefiniteCovariance:
- the indefinite cov is rejected and R falls back to the base floor, while
a strongly-correlated-but-valid cov (corr 0.9, det > 0) still gets
through — the check must reject only the impossible matrices, or it
silently discards the relative weighting _KF_R_INFLATE exists for
- 200 solves carrying an indefinite cov at both a 1.4 s and a 20 s
cadence keep P PSD and symmetric and learned_velocity answering. On
origin/main the 20 s case raises at solve 16 with the droplet's exact
traceback, track_filter.py:278 ValueError: math domain error
- a 1 m and a 1e-6 m formal sigma both compose to the 1200 m floor, so a
metre-scale R never reaches the update
- 200 updates at an effectively-zero cov with alternating positions stay
PSD
- a hand-poisoned negative P[1,1]/P[3,3] returns sigma 0.0 rather than
raising
test_feed_multinode.TestMultinodeEntryFailureIsolation:
- three multinode keys, the middle one raising ValueError("math domain
error") from learned_velocity: the other two are still served, only the
sick one is missing, and one log line names it. On origin/main the
exception escapes build_combined_aircraft_json and the whole tick is
lost, which is what "Aircraft flush failed" was
- 20 consecutive failures produce one log line, not 20
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…r-observability docs/solverflow.md: keep this branch's symbol-named 'Defined in' column and add #292's SOLVER_ALT_MODE / SOLVER_FREE_ALT_STARTS rows and section in it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…/resolve-slot-claim-on-publish
…is merged there Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r-observability
…/resolve-slot-claim-on-publish
…ix/cluster-contamination test_analytics_routes.py: keep both new status tests (live counters from this branch, world-skipped pairs from #291). libs/retina-analytics: point at analytics main (#25 world gate + #26 cluster split + #27 epochs merged). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…urement-epochs state.py / routes/test.py / solver.py / test_solver_stats.py: adjacent additions from both sides kept (epoch-align flag + counters next to the free-altitude flags, resolve_skips_dark and altitude_mode stamps). libs/retina-analytics: analytics main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ts-and-dr-budget .env.example: both new comment blocks, the shared commented knobs once. docs/solverflow.md: symbol-named constants table from the stack plus this branch's MN_DR_CAP_S / MN_DARK_EXPIRY_S rows; section 7 kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Claude finished @jehanazad's task in 3m 15s —— View job Claude finished reviewingTodo list:
Scope noteThis branch carries several already-merged stacked PRs (#291–#296) on top of it, which is why the file list is 41 files / +3447/-286. The actual new work in this PR is three commits: ReviewThe diagnosis and fix hold up. I re-verified independently:
One new finding from the merge, otherwise only previously-flagged nits:
No correctness issues in the core PSD/Joseph-form fix, the predict-step symmetrization, or the failure-isolation logic. The regression tests are well-targeted at the actual production failure mode. |
jehanazad
marked this pull request as ready for review
September 6, 2026 02:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
Test droplet, 91 tracebacks in 40 minutes:
Two callers, both of which lose real work:
solver.py multinode_key_decision -> _entry_dr_velocity -> learned_velocity, logged as "Solver worker: unhandled error for one item" — that solve is dropped.aircraft_feed.multinode_to_aircraft -> learned_velocity, logged as "Aircraft flush failed" — the whole feed broadcast for that tick is dropped.One KF track with a negative velocity variance was taking down the map.
Root cause — not the covariance update
The brief expected the culprit to be a simple-form
P = (I - KH) Pupdate. It isn't:_kf_correcthas been Joseph-form since 2026-08-26, and I confirmed the deployed container is running the Joseph code, withlearned_velocityat exactly line 278. So the negative diagonal is not roundoff in the update.The real path is upstream, in the measurement noise:
cov_en_km2is the top-left 2x2 block ofs² · inv(JᵀJ)for the solver's 5-state fit (libs/retina-geolocator/retina_geolocator/multinode_solver.py). The solver falls back topinvonly on an outrightLinAlgError, so an ill-conditioned-but-not-singularJᵀJ— near-parallel baselines, the same degenerate tail that puts the formal sigma's p99 at 3.8e6 km — inverts to numerical garbage that is indefinite while both diagonals stay positive.Both validation sites checked only shape, finiteness and positive diagonals. Neither checked the determinant, so that matrix became
R.An indefinite
Ris not survivable downstream:K R Kᵀterm inherits R's negative eigenvalue directly._init_entryseeds P's position block straight from R, so a sick covariance poisons the filter at birth as well as on every update.Once P's velocity diagonals go negative,
learned_velocity's sqrt raises.In a Monte Carlo over near-rank-deficient
JᵀJmatrices, 47% of the covariances that pass the existingexx > 0 and eyy > 0check are indefinite.The fix
backend/services/track_filter.py_measurement_Rnow requires the covariance to be genuinely PSD — both diagonals> 0and determinant>= 0— falling back to the base floor otherwise. Rejecting rather than repairing by eigenvalue clipping: a covariance this degenerate carries no trustworthy relative weighting to preserve, andR = baseis already this function's documented answer for a degenerate cov. A strongly-correlated but valid cov still gets through, so the relative weighting_KF_R_INFLATEexists for is not discarded.P, as_kf_correctalready did after the update.learned_velocityclamps its sqrt at 0, so a pathological state degrades to "sigma 0" instead of taking out a solve or a broadcast. (kf_pos_sigma_mwas already clamped;solve_uncertainty.velocity_sigma_msalready guards non-finite.)backend/services/aircraft_feed.py_multinode_entry(key, r, now)and the call sits behind onetry/exceptinbuild_combined_aircraft_json: a raising entry logs once a minute (rate-limited, naming the key) and is skipped, instead of aborting the tick. A skipped entry ages out ofstate.multinode_trackswithin 60 s on its own.Numbers
Sweep of indefinite covariances (correlation 1.2–10, formal variance 0.01–100 km², 1.4/5/20 s cadence, 0/100/250 m/s), driven through the public
smooth_solve+learned_velocityAPI:origin/mainAt the 20 s cadence real solves arrive on,
origin/mainraises within 16 solves.Verification
origin/mainwith the reported traceback verbatim (services/track_filter.py:278: ValueError: math domain error) and pass here.tests/test_track_filter.py tests/test_feed_multinode.py tests/test_solve_uncertainty.py tests/test_mn_lifetime.py: 113 passed, 1 skipped.-n 2 -m "not external"): 2789 passed, 3 skipped, exit 0.pre-commit run --all-files: all 5 hooks passed (ruff check, ruff format, vulture, both ruff-config checks).Notes for the integration branch
aircraft_feedchange is a pure code move plus the try/except, so it should merge cleanly alongside othersolver.py/aircraft_feed.pywork._KF_MIN_POS_SIGMA_Mwas not added. That floor already exists at 500 m, and_KF_DEFAULT_POS_SIGMA_M(1200 m) is added toRunconditionally, so R sigma cannot fall below 1200 m — a few-metre or zero R never reaches the filter. Adding a 25 m constant would be dead code and would weaken a documented model. A test pins the actual behaviour instead.multinode_solver.pyshould either usepinvon a condition-number test rather than only onLinAlgError, or apply the same determinant check before settingcov_en_km2/pos_sigma_km. Not done here — it is a submodule pin change, and out of scope for an urgent fix.🤖 Generated with Claude Code