Rank illuminators by expected bistatic detection area, with diversity, sweep calibration and fleet feedback

[jehanazad]

Implements the ranking redesign from the design note (https://claude.ai/code/artifact/ce5a71b5-0be4-4084-8052-af60ccfff2a8). Four commits, each independently reviewable.

Compatibility with the nodes

retina-gui and retina-spectrum read this API. No existing field is renamed, retyped or removed; rank stays contiguous 1..n; query and count keep every existing key. A contract test pins the pre-existing field set and types on both routes. Everything below is additive.

New per-tower fields: expected_area_km2, best_azimuth_deg, horizon_km, feedback_factor, feedback_n, site_id, site_channels, diversity_penalty, direct_power_source, direct_power_dbm_override, measurement_quality.
New query keys: ranking (both routes), calibration_offset_db and calibrated_towers (POST).

What changes

1. Score by expected detection area (services/tower_scoring.py). Bistatic radar equation over a 2 km grid of the receiver's 80 km disk at 3 km target altitude: echo against thermal noise plus the direct-path residual after 50 dB cancellation, 13 dB margin after coherent processing (6 MHz × 0.5 s for ATSC, 100 kHz × 1 s for FM), baseline region cut at 150° bistatic angle, radio-horizon loss from the FCC antenna height, best surveillance azimuth over a 42° Yagi with 20 dB front-to-back. numpy-vectorised: 200 towers in ~10 ms. Every knob is in a new scoring config section.
2. Band tier becomes a soft prior. ranking.band_offset_db (shipped as zeros) is added to EIRP inside the model. The physical differences between bands (path loss, processing gain, resolution) are already in the model; the offsets are for what is left and should be fitted from fleet data.
3. Site-aware diversity. Maximal-marginal-relevance ordering with ranking.diversity config (λ = 0.7 shipped). Same-mast channels move down the list rather than being removed.
4. Sweep calibration on POST. Per-sweep dBFS→dBm offset from the median measured-minus-modelled gap over matched TV towers; measured direct-path power replaces the model for those towers. The analyser score (different scales for FM and TV upstream) becomes a quality multiplier instead of a sort key.
5. Fleet feedback. POST /api/feedback/tower-outcome behind a fail-closed TOWER_FINDER_FEEDBACK_TOKEN (separate from the admin token because every node will hold it), SQLite store on the runtime volume, shrinkage model exp(n/(n+5)·r̄) over rows from receivers within 30 km, applied before the sort. GET /api/feedback/summary for admins. Never raises into the request path.

Default sort and migration

Shipped sort_order is now [expected_area_km2 desc, received_power_dbm desc]. A runtime overlay still carrying one of the old shipped defaults is upgraded in memory with a warning and the file left untouched, following the distance-class precedent; a deliberately different overlay is left alone. Deployed volumes therefore pick up the new ranking on deploy without a config PUT, unless an operator had changed the sort.

Verification

• Backend: 572 passed including integration marks; ruff check and format clean. Frontend: tsc clean, 51 vitest tests, build emits no image assets.
• Live query against this branch, lat=37.78&lon=-122.41&radius_km=120&limit=200, 2.9 s end to end (FCC fetch dominates):

rank	tower	km	P_rx dBm	area km²	point
1	KTXL, Sacramento (UHF)	96	−28	16,920	240°
2	KSTS, San Jose (UHF)	57	−26	16,540	300°
3	KVIE, Sacramento (VHF)	96	−34	14,208	240°
4	KNTV, San Bruno Mtn (VHF)	11	−12	7,232	30°
5	KSVY, Sonoma (FM)	51	−39	6,036	60°

The 19 Sutro Tower channels, previously ranks 1 to 19 at −3 dBm, now start at rank 27: too strong for the front end, too close for geometry. Ordering is stable across 40 to 60 dB of cancellation and 1 to 100 m² targets (sensitivity runs in the design note).

Follow-ups, not in this PR

• retina-gui: post Auto-Calibrate outcomes to the new endpoint; retina-server: nightly join of the detection archive with config history, and reset the learned FOV on retune.
• Fit band_offset_db and per-region residuals once feedback data exists; the archive residual is provisional and marked so.
• Terrain line of sight along the path is the largest remaining model gap; the horizon term is a blunt proxy.
• Frontend for alternates was deliberately not built: no rows are removed, so there is nothing to show yet.

🤖 Generated with Claude Code

[claude]

Claude finished @jehanazad's task in 3m 55s —— View job

---

Claude finished the review

Todo list

• Gather context (PR description, changed files)
• Read CLAUDE.md and repo conventions (none present)
• Review diff against origin/main across backend scoring/ranking/feedback and frontend
• Post review feedback

Summary

This is a large, carefully documented PR, and the vast majority of it is solid: the config validation, fail-soft wiring around scoring/coverage/feedback, the MMR diversity pass, the SQLite feedback store, and the frontend additions all look correct and well-tested. I found one real correctness bug in the sweep-calibration path that's worth fixing before merge, plus a couple of smaller notes.

---

🐛 Receiver antenna gain is double-counted for calibrated (measured) towers

backend/services/tower_scoring.py:262-294 (expected_detection_area) and backend/services/tower_ranking.py:1071-1103 (_calibrate_direct_power)

The docstring for direct_power_dbm_override says it "replaces the modelled direct-path power at the receiver before the antenna pattern is applied" — i.e. it should be on the same isotropic scale as the non-override branch:

if direct_power_dbm_override is None:
    direct_iso_dbm = eirp_eff - fspl_db(...) - hloss_db   # isotropic, no rx gain yet
else:
    direct_iso_dbm = float(direct_power_dbm_override)     # expected to also be isotropic
...
dpi_dbm = (
    direct_iso_dbm
    + params.rx_gain_dbi          # rx antenna gain applied here, once
    + _yagi_pattern_db(...)
    - params.cancellation_db
)

But the override actually supplied by _calibrate_direct_power is derived from received_power_dbm, which already includes RX_ANTENNA_GAIN_DBI:

# tower_ranking.py
def received_power(eirp_dbm, distance_km, freq_mhz):
    return eirp_dbm + RX_ANTENNA_GAIN_DBI - fspl(distance_km, freq_mhz)   # gain included

residuals = [t["power_db"] - t["received_power_dbm"] for t in towers ...]   # offset relative to a gain-included baseline
offset = statistics.median(residuals)
...
t["direct_power_dbm_override"] = t["power_db"] - offset   # lands on the received_power_dbm scale (gain included)

So for every calibrated tower, dpi_dbm ends up as (gain-included measured power) + rx_gain_dbi (again) + pattern - cancellation, i.e. the noise floor is inflated by an extra RX_ANTENNA_GAIN_DBI (6 dB by default, whatever receiver.rx_antenna_gain_dbi is configured to). That systematically understates expected_area_km2 specifically for towers that did get a real sweep measurement — the exact set the calibration feature (commit 4, "Sweep calibration on POST") is supposed to make more trustworthy than the model.

Failure scenario: POST /api/towers with a spectrum sweep that calibrates ≥2 TV towers (as in test_towers_routes.py::test_post_names_the_ordering_and_the_calibration). Those towers' expected_area_km2 comes out too low by a factor of 10^(rx_gain_dbi/10) relative to what the model intends, which can visibly reorder the ranking versus uncalibrated towers or versus what the design note's sensitivity analysis assumed. No existing test catches this because the calibration tests only assert the offset's type/count (test_towers_routes.py:648-649) and the test_tower_scoring.py:240 override test only checks the direction of the effect (swamped < base), not the magnitude — so a constant 6 dB bias in n_eff_dbm passes silently.

Suggested fix: either subtract RX_ANTENNA_GAIN_DBI when building the override in _calibrate_direct_power (so it lands back on the isotropic scale expected_detection_area expects), or have expected_detection_area skip adding params.rx_gain_dbi when an override is supplied. The former is more consistent with the model's existing isotropic/gain-applied-once convention. Fix this →

---

Minor notes (not blocking)

• backend/services/tower_ranking.py:1098-1103 — the loop applying direct_power_dbm_override checks only t.get("measured") and _is_number(t.get("power_db")), without the band in _CALIBRATED_BANDS filter that the residual computation above it uses. Today this is harmless because retina-spectrum only ever sends power_db for VHF/UHF (FM is always None, per the model docstring in backend/models/measurements.py), but it's an implicit cross-file contract rather than something this module enforces itself — worth a defensive band in _CALIBRATED_BANDS check or a comment noting the reliance.
• backend/services/tower_scoring.py (horizon_loss_db) — the horizon-loss penalty is computed once per tower from d_tower_km (tower-to-receiver ground distance) and applied uniformly to every cell in the grid, including the bistatic echo path through a 3 km-altitude target. Physically the target-altitude paths shouldn't be gated by a ground-level horizon check in the same way the direct path is. This looks like a deliberate simplification — the PR description explicitly calls out "Terrain line of sight along the path is the largest remaining model gap; the horizon term is a blunt proxy" — so flagging only for awareness, not as a bug.

What I did not find issues with

• Config validation (validate_config/_validate_scoring/_validate_diversity) is thorough and consistent with what apply_config actually consumes.
• The legacy sort-order upgrade/distance-rule-drop logic (_upgrade_legacy_default_sort, _drop_legacy_distance_rules) is careful about only touching overlays that are byte-for-byte a previously-shipped default, and doesn't rewrite the file on disk.
• tower_feedback.py's SQLite store: locking, pruning, bounding-box + haversine filtering, and the fail-soft wrapping in apply_feedback all look correct.
• core/auth.py's new feedback token reuses the same constant-time, fail-closed comparison as the admin token — no issues.
• Frontend changes (format.ts, ResultsTable.tsx, TowerMap.tsx, types.ts) are additive, null-safe, and match the backend contract; format.test.ts covers the edge cases (out-of-range bearings, missing values).

[claude]

Claude finished @jehanazad's task in 3m 46s —— View job

---

Claude finished the review

Todo list

• Gather context (PR description, changed files, prior review comment)
• Check repo CLAUDE.md for conventions (none present)
• Run code review across the diff (origin/main...HEAD)
• Post review feedback

Summary

This sync only merged main (PRs #29/#31/#32/#33) into the branch — git diff origin/main...HEAD is byte-for-byte the same 28 files at the same sizes as the previous review, and there are no merge-conflict markers or edits to this PR's own commits. The bug flagged in the previous review is still present and unaddressed.

---

🐛 (still open) Receiver antenna gain is double-counted for calibrated (measured) towers

backend/services/tower_scoring.py:262-294 (expected_detection_area) and backend/services/tower_ranking.py:1250-1265 (_calibrate_direct_power)

direct_power_dbm_override is documented as replacing the direct-path power before the antenna pattern/gain is applied — i.e. it should be isotropic, matching the non-override branch (eirp_eff - fspl_db(...) - hloss_db, no rx_gain_dbi yet):

# tower_scoring.py:262-294
if direct_power_dbm_override is None:
    direct_iso_dbm = eirp_eff - fspl_db(...) - hloss_db      # isotropic
else:
    direct_iso_dbm = float(direct_power_dbm_override)        # assumed isotropic too
...
dpi_dbm = direct_iso_dbm + params.rx_gain_dbi + _yagi_pattern_db(...) - params.cancellation_db

But _calibrate_direct_power derives the override from received_power_dbm, which already includes RX_ANTENNA_GAIN_DBI:

# tower_ranking.py:1250-1265
residuals = [t["power_db"] - t["received_power_dbm"] for t in towers ...]   # received_power_dbm has rx gain baked in
offset = statistics.median(residuals)
...
t["direct_power_dbm_override"] = t["power_db"] - offset   # lands on the received_power_dbm (gain-included) scale

So every calibrated tower's dpi_dbm gets rx_gain_dbi added twice, inflating the noise floor and understating expected_area_km2 by a factor of 10^(rx_gain_dbi/10) (6 dB by default) — specifically for the towers the sweep-calibration feature is meant to make more trustworthy than the model. This can visibly reorder calibrated towers below where they belong relative to uncalibrated ones.

Worth noting, and it widens the bias further: since this branch merged in #33 (terrestrial path loss), received_power_dbm is now computed via path_loss() → Okumura-Hata (PROPAGATION_MODEL defaults to "hata" in the shipped config, backend/config/tower_config.json:91), which adds terrain/clutter excess loss on top of free space. expected_detection_area's own direct-path term still deliberately uses plain fspl_db + the blunt horizon_loss_db proxy (self-consistent by design, per the PR's own "terrain is the largest remaining model gap" note). That divergence is a defensible modeling choice on its own, but it means the "modelled" baseline used to compute the calibration offset (Hata-based) is on a different footing than the isotropic scale expected_detection_area expects for the override — on top of, not instead of, the antenna-gain double-count above.

Suggested fix: subtract RX_ANTENNA_GAIN_DBI when building direct_power_dbm_override in _calibrate_direct_power so it lands back on the isotropic scale expected_detection_area assumes. Add a test asserting the numeric value of expected_area_km2 for a calibrated tower against the uncalibrated prediction with a known offset (not just direction — test_tower_scoring.py:240's swamped < base check would pass even with this bug).

Fix this →

---

Minor notes (unchanged, not blocking)

• backend/services/tower_ranking.py:1260-1264 — the override-assignment loop still checks only t.get("measured") and _is_number(t.get("power_db")), without the band in _CALIBRATED_BANDS filter the residual computation above it uses. Harmless today since retina-spectrum only sends power_db for VHF/UHF, but it's an implicit cross-module contract rather than something this function enforces itself.
• backend/services/tower_scoring.py (horizon_loss_db) — still a ground-distance-only proxy applied uniformly across the grid, including bistatic echo paths through a 3 km-altitude target. Called out in the PR description as a known, deliberate simplification — flagging only for awareness.

What I did not find issues with

Everything else previously reviewed is unchanged in this sync: config validation (_validate_scoring/_validate_diversity), the legacy sort-order upgrade logic, tower_feedback.py's SQLite store, core/auth.py's fail-closed feedback token, and the frontend additions all still look correct. I did not find any new issues introduced by merging main (#29/#31/#32/#33) into this branch — no conflict markers, and the propagation-model change from #33 is additive to tower_ranking.py without touching tower_scoring.py.

I was not able to run the backend test suite or linters in this session (command execution for pytest/ruff required approval that wasn't available), so this review is based on static reading of the diff plus the merged-in path_loss/Hata code.
|