20260909 - Track bistatic range, and measure the tracker against truth - #27
Merged
Merged
Conversation
Live data cannot tell you whether two detections were the same aircraft, so it cannot tell you whether a track was right. Tracker changes have been judged by eye against a plot ever since, which is why the association parameters have never had a defensible baseline. synthetic/ generates a blah2-shaped detection stream with a truth sidecar recording which aircraft produced each detection, and scores tracker output against it: recall, fragmentation, purity, id_switches, false_track_rate and clutter_absorbed. Every constant in profile.py marked as measured came from a live capture of the Atlanta reference node, and tests/test_synthetic.py asserts the generator still reproduces them so the fitted knobs cannot drift. Three properties of the reference are easy to get wrong and are modelled explicitly: dropouts are bursty (a two-state Markov chain, continuity 0.37), clutter is 90% one-shot, and ADS-B freshness has a heavy tail. The eleven profile constants added to vulture_whitelist.py are recorded reference measurements the test does not yet assert. They are a backlog entry, not an exemption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
23ae236 dropped MAX_COMPLETED_TRACKS from 5000 to 50 to bound a long-running server's memory, reasoning that "get_confirmed_tracks and completed_tracks have zero call sites in the server". True, but the CLI is the other consumer, and the same commit message names it: "to_dict serializes each confirmed track's full history, and that is the CLI's entire batch output". A batch run therefore kept only the 50 most recently retired tracks and silently discarded the rest. Measured on a 937-frame synthetic capture: 435 tracks written before, 79 after. Scored against truth, track_recall read 0.449 and detection_recall 0.185, purely because the tracks were missing from the file. The tracker itself was unchanged, which is what made this invisible. The escape hatch already existed and was already tested; nothing used it. The batch path's archive is bounded by the input file, so it takes maxlen=None. The server default is untouched at 50, and no caller passed max_completed_tracks explicitly, so giving None its ordinary deque meaning breaks nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The filter carried [delay, delay_rate, doppler, doppler_rate] with H = [[1,0,0,0],[0,0,1,0]]: two independent constant-velocity chains. Doppler is not independent of delay, it is its derivative. range_rate = -wavelength * doppler, exactly, so the old model spent a free parameter estimating a quantity every detection already measured. It measured it badly. On the reference capture the delay residual is 0.14 km median, so fitting a rate from successive delays needs 20.4 s of track history to reach 9.7 m/s. One Doppler measurement gives 1.07 m/s. The state now carries [range_km, range_rate_km_s, range_accel_km_s2] and a detection arrives as [delay, -wavelength * doppler], so H stays [[1,0,0],[0,1,0]] and the filter stays linear: no EKF, no Jacobians. This is what breaks the trade-off the parameters could not. Twelve config sweeps over gates, coasting and deletion moved along one curve: every setting that cut clutter absorption raised fragmentation and vice versa. Coupling moves off it. On 937 synthetic frames, 89 aircraft: clutter absorbed 0.253 -> 0.101, purity 0.799 -> 0.943, id switches 470 -> 184, fragmentation 4.87 -> 4.45, detection recall 0.902 -> 0.907, false tracks 2 -> 1. Fragmentation and clutter improve together for the first time. Live on a node it removes a track population that was never physical. Of the tracks that node's previous build emitted over two days, 867 of 1796 (48%) had a delay slope inconsistent with their own Doppler, the worst by a factor of 25. Every track the coupled build has emitted agrees with its Doppler to within 17%. Supporting changes, each forced by the coupling: Wavelength is now load-bearing, so radar.center_frequency has to be right; WAVELENGTH_KM() derives from it and get_mach1_doppler_threshold() shares the accessor. Deployments already pass --blah2-config, which reads capture.fc from the node. Without it the fallback is 200 MHz, and a wrong value biases every track's velocity by the same ratio, silently. Process noise becomes one jerk PSD on the range axis, process_noise.range_jerk, replacing the two per-chain knobs which no longer have chains to drive. Swept over six decades on the reference capture: 1e-7 is the optimum and every metric degrades monotonically above it. It scales as wavelength squared, so a node on a different carrier wants a different value. The covariance is symmetrised after predict and update. Round-off in (I - KH)P leaves the off-diagonals differing in the last bit; that was invisible while they were exactly zero, but a constant-acceleration model correlates range with its rate and the association gate applies Sylvester's criterion to S as a symmetric matrix. TestGateStructuralInvariant pinned block-diagonality, which the coupling deliberately breaks, so it now pins symmetry and definiteness, which is what the gate actually depends on. _initiate_tracklets no longer overwrites the velocity state with a slope fitted to three delays. That fit is the 20-second estimate the Doppler measurement beats ninefold. It keeps the residual check as a promotion gate and now seeds range acceleration from the Doppler slope, which is otherwise unobserved. _init_from_adsb loses its velocity seed: it computed vel_horiz / 299792.458, dividing metres per second by c in kilometres per second, roughly 600x too small and therefore always about zero. Doppler supplies that state properly now. Several test fixtures described motion that cannot occur, holding Doppler constant while delay drifted, or ramping it against the delay trend: a target closing and receding at once. The decoupled model never checked. They now derive Doppler from the delay step they describe. test_intermittent_adsb is xfailed strictly rather than fixed, because its fixture pairs a monostatic Doppler with a bistatic range and reconciling them changes what every supersonic test in that file asserts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…shadow A large aircraft casts multipath and sidelobe replicas of itself at longer bistatic range. They move with the target they came from, so they obey range_rate = -wavelength * doppler exactly, and the coupled filter has no reason to reject them. One banking aircraft on a live node produced thirteen of them. Measured over 93 same-frame pairs on that node: 89% of the spurious detections around an aircraft sat at longer delay than it and 99% were weaker, a median of 6.9 dB down. Doppler is deliberately not compared - the replica travels a different path, so its Doppler genuinely differs, a median of 35 Hz even for copies within 1.5 km. Longer delay and lower SNR is the whole signature. A detection behind a brighter one is flagged rather than dropped, and the flag blocks track initiation rather than association. A real aircraft passing behind something brighter still updates its own track; it just cannot found a new one while most of its history sits in another return's shadow. Thresholds come from the same measurement rather than from taste. Across whole tracks the two populations barely overlap: every ADS-B-matched aircraft stayed at or below a shadowed fraction of 0.10, thirteen of the fourteen at exactly 0.00, while replica tracks ran a median of 0.64. The 0.5 threshold sits in that gap with a fivefold margin, and blocks 60.9% of replica tracks against 0.0% of aircraft. The thresholds are fitted to one site, so the failure mode matters more than the values. Sweeping the window over 1-6 km and the margin over 1-8 dB - six- and eightfold ranges - moves how many replicas are caught between 0% and 61% while the aircraft-blocked rate stays at 0.0% throughout. A badly set window on another site therefore loses benefit rather than causing harm. That is an argument, not a guarantee, so the fraction is reported alongside each track and each event. A site where these thresholds do not fit shows up as identified aircraft drifting off zero, without anyone having to go looking. The whole test is behind shadow.enabled. On the synthetic harness this costs detection recall 0.907 -> 0.901 and fragmentation 4.45 -> 4.58, with purity and clutter unchanged. That generator models no multipath, so there are no replicas to catch and only the false positive rate shows; it is also far denser than the live node, at 11 detections per frame against one to four, which inflates that rate. A dense site can lower shadow.delay_km to 2.0 and still catch 47.8%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every tracker change so far has been judged against synthetic/score.py, which is exact because the generator knows which aircraft produced each detection. It is also a single site: every constant in synthetic/profile.py was fitted to one 118 s capture, and the fitting has now been wrong twice. It records clutter as 90% one-shot, which sent me past a persistent road-traffic population on a live node; and it records a Doppler slew of 0.57 Hz/s median where live aircraft run 3.7, a factor of five in the quantity the process noise exists to cover. That is not a reason to distrust the harness, which measures things nothing else can. It is a reason not to let it be the only yardstick. This scores the events.jsonl a node already writes, against the ADS-B annotation blah2 attaches to each detection. The label is much weaker: cooperative traffic only, matched within a tolerance, missing roughly a quarter of real target detections. The numbers are therefore not comparable to the synthetic ones. What they are is producible on any node, which is the only way to tell an improvement from something fitted to one site. Six measures, each one an analysis that had to be written by hand to answer a question during this work: kinematic consistency (does a track's range move the way its own Doppler demands), fragmentation per identified aircraft, the shadow guardrail, the near-origin share where ground traffic lives, replica offsets in delay and SNR, and the observed Doppler slew - which is what the harness's own manoeuvre model has to be held to. Stdlib only, and inside the package rather than beside it, so it can be run in the container or copied onto a node and run under whatever python3 is there. On the node it was written against, over its own two files: old build 904/1796 tracks kinematically impossible (50.3%) coupled 4/46 (8.7%) with fragmentation per identified aircraft unchanged at 1.37 -> 1.38, which is the result the synthetic harness could not have told us about that node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…try a knob The tracker's process noise is sized against how fast Doppler slews, and profile.py recorded 0.570 Hz/s median for that. Nothing asserted it, which is why it was on the vulture whitelist, and it turns out to match no estimator of the generator's own output: the same frames read 0.971 Hz/s differenced between adjacent frames and 0.344 regressed over ten seconds. The gap between those two is not subtle and is not noise in the usual sense. It is noise being measured as signal: 0.63 Hz of Doppler residual across a one second frame is 0.9 Hz/s of apparent slew, the same order as the thing being measured. generate.doppler_slew() fixes the estimator as the regressed one and is the same measurement retina_tracker.live_score makes on a node, so the two are finally comparable. The constants become 0.408 and 1.820, pooled over the reference seeds and asserted, and come off the whitelist. The reason for going near this was a live node slewing 2.9x faster than the harness in carrier-independent terms, which looked like the generator modelling implausibly placid aircraft. It is not. Turning is nearly irrelevant here: putting 60% of the fleet into a near-standard-rate turn, twenty times the current setting, moves the median from 0.57 to only 0.57 against the 1.01 that site implies. What differs is where the traffic is. The harness flies aircraft at a median bistatic range of 29.8 km and that node sees 5.8 km, and an aircraft crossing near the baseline sweeps Doppler far faster than the same aircraft further out. So slew is a property of a site, not of aircraft, and there is no single correct value to write down. Refitting these constants to the second node would only have moved the overfit rather than removed it. Instead the traffic disc becomes a parameter: build_fleet already took spawn_radius_km and nothing threaded it through, so generate() and the CLI now do, and pulling the disc from 46 km to 10 km takes the median range to 8.9 km and the slew to 0.872 Hz/s. A capture can be generated against the site it is meant to represent. The second site's figures are recorded in profile.py as prose rather than as constants, because nothing reads them and an observation is not code. The turn knobs, which were bare literals inside build_fleet, become profile constants with a test that they are actually wired - an unreferenced constant being precisely how the slew figure drifted in the first place. The tracker is untouched and its synthetic score is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
range_jerk was the most overfit number in the tracker: swept to an optimum against one generated dataset, and that dataset turned out to represent one site's traffic geometry. Bistatic range acceleration is mostly geometry rather than aircraft - it depends on how close a target passes - so it varies more between sites than between flight phases. Two nodes measured 2.9x apart, driven by traffic at 29.8 km against 5.8 km. There is no single right value to tune to, which is a reason to remove the tuning rather than repeat it. A matched filter produces a normalised innovation squared averaging the measurement dimension. The ratio to that is how badly the constant-acceleration assumption is failing right now, measured by the filter on itself, so each track carries a running average of it and widens its own process noise by that ratio. range_jerk becomes a floor rather than an optimum. A banking aircraft widens for as long as it banks; a cruising one never leaves the floor. Floored at 1 rather than allowed to shrink, so a well-modelled target behaves exactly as it did before this existed, and capped, so one wild innovation cannot open a gate indefinitely. Neither bound is doing much work: on close traffic the average engages on 18% of updates and peaks around 3x, and raising the cap from 25 to 100 changes no scored metric at all. Scored on the default geometry it is a wash against the value it replaces (fragmentation 4.58 -> 4.57, purity 0.941 -> 0.942, id switches 198 -> 196), and on close traffic where slew is 2.2x higher it recovers a little track recall (0.944 -> 0.952). What matters more is what it avoids: reaching the same following behaviour by raising the fixed jerk costs dearly, with 1e-5 dropping purity to 0.876 and doubling clutter absorption on the default geometry, and to 0.760 on close traffic. Adaptation buys the manoeuvre response without paying for it on every quiet track. It is also currently limited by something else, which is worth recording. The running average sits at a median of 1.08 where a matched filter would sit at 2.0, so the innovation covariance is about twice what the residuals warrant. That is R: MEASUREMENT_NOISE_DELAY is 1.0 km^2 against a measured delay residual of 0.14 km, and it was never fitted to anything. Too loose an R both widens every gate and masks the model error this reads, so calibrating it against the ADS-B residuals would tighten association and let this see more than it can now. The two are coupled and R is the one still unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 1s —— View job I'll analyze this and get back to you. |
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seven commits on the tracker's association quality, plus the two things needed to judge it: a synthetic harness with ground truth, and a scorer that works on a real node without any.
The reasoning and the numbers are in the commit messages, which are detailed. This is a map of them.
Being able to measure it at all
ff1b315— synthetic detection harness. Live data cannot tell you whether two detections were the same aircraft, so it cannot tell you whether a track was right. Association parameters had never had a defensible baseline.synthetic/generates a blah2-shaped stream with a truth sidecar and scores recall, fragmentation, purity, id_switches, false_track_rate and clutter_absorbed. Constants marked measured came from a live Atlanta capture, and the tests assert the generator still reproduces them so the fitted knobs cannot drift.f430d3d— score a node's own output without truth data. The harness is one site, and its fitting has been wrong twice. This scores theevents.jsonla node already writes against blah2's ADS-B annotation: kinematic consistency, fragmentation per identified aircraft, the shadow guardrail, near-origin share, replica offsets and observed Doppler slew. A much weaker label, and not comparable to the synthetic numbers, but producible on any node — which is the only way to tell an improvement from something fitted to one site.331846c— pin the harness's Doppler slew.profile.pyrecorded 0.570 Hz/s and nothing asserted it. The same frames read 0.971 differenced and 0.344 regressed: noise being measured as signal, since 0.63 Hz of residual across a one-second frame is 0.9 Hz/s of apparent slew. The estimator is fixed as the regressed one, matching whatlive_scoremeasures on a node, so the two are finally comparable.The model change
8c85fa0— track bistatic range, not delay and Doppler separately. The filter carried two independent constant-velocity chains, spending a free parameter estimating a quantity every detection already measures:range_rate = -wavelength * doppler, exactly. The state becomes[range_km, range_rate_km_s, range_accel_km_s2]and stays linear.This is what breaks a trade-off twelve config sweeps could not: every setting that cut clutter absorption raised fragmentation and vice versa. On 937 synthetic frames, 89 aircraft — clutter absorbed 0.253 → 0.101, purity 0.799 → 0.943, id switches 470 → 184, fragmentation 4.87 → 4.45, false tracks 2 → 1. Live, it removes a track population that was never physical: of the tracks the previous build emitted over two days, 867 of 1796 had a delay slope inconsistent with their own Doppler.
17a64f4— widen the motion model on the filter's own residuals.range_jerkwas the most overfit number in the tracker, and bistatic range acceleration is mostly geometry, so it varies more between sites than between flight phases — two nodes measured 2.9x apart. Each track now carries a running average of its normalised innovation squared and widens its own process noise by that ratio, makingrange_jerka floor rather than an optimum.Two corrections
32bf051— refuse to promote tracks in a brighter return's shadow. Replicas move with the target they came from, so they obey the range/Doppler relation exactly and the coupled filter has no reason to reject them; one banking aircraft produced thirteen. Over 93 same-frame pairs, 89% of spurious detections sat at longer delay and 99% were weaker, a median 6.9 dB down. Flagged rather than dropped, blocking initiation rather than association.632883d— restore the CLI's completed-track archive. DroppingMAX_COMPLETED_TRACKSto 50 bounded a server's memory but silently truncated batch output: 435 tracks written before, 79 after, readingtrack_recall0.449 anddetection_recall0.185 purely because the tracks were missing from the file. The server default is untouched.Verification
181 tests pass on this branch alone, run in a clean worktree. ~4,600 lines added, over half of them tests.
Running live on jonathan-node-1 since 10:16 UTC on 10 Sep, verified by introspection of the loaded modules rather than by file path.
Note on stacking
retina-tracker's
20260909-tracker-control-resetis based on this branch and should merge after it. That work is the control surface, detection classification and the four-hour record; it is plumbing and does not touch the filter.🤖 Generated with Claude Code