Skip to content

Bench-ready measured-input chain, radio packet format, and overnight side runs - #7

Merged
Whyater merged 107 commits into
mainfrom
overnight/2026-11-04-sim-hardening
Nov 4, 2026
Merged

Whyater merged 107 commits into
mainfrom
overnight/2026-11-04-sim-hardening

Conversation

@Whyater

@Whyater Whyater commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Overnight work on making the sim usable with real measured inputs. No registered result folder or hash in tools/reproduce_registry.json was changed. Most of this is new side-run tools, docs and tests; existing input tools also gain guards that can now refuse or relabel an input (listed below).

What changed

Bench data and measured-input chain

  • tools/preflight_bench_data.py: checks a bench folder before any intake runs.
  • tools/bench_verdict.py: one command from preflight through intakes, combined rerun and latency budget.
  • tools/propose_measured_entry.py: drafts a registry entry for a measured run, with strict run-dir checks.
  • tools/measured_provenance.py and edits to the link, combined and measured reruns, latency budget and fit writers: a "measured" label now needs provenance, otherwise it becomes "unverified".
  • Camera log quirk checks in sim/vision_noise_fit.py, sim/detection_log.py and tools/intake_vision.py, documented in docs/DETECTION_LOG.md.
  • Combined S6 rerun gains default-off noise shape, bias and dropout options.
  • Input guards: tools/latency_budget.py refuses a typed --processing-median-ms outside 0.5 to 500 ms and reports "not measured" when no input is measured; tools/vision_latency.py checks --refresh-hz against the recorded display_refresh_hz; sim/vision_noise_fit.py refuses positions that are not in meters and leaves latency unfitted when any frame is over 5 s.

Radio

  • sim/radio/packet.py: fixed 161-byte ESP-NOW state packet with version byte and CRC-32, under the 250-byte v1.0 limit.
  • tools/espnow_airtime_budget.py: airtime and channel use per swarm size and send rate.

Side runs (all simulated; outputs stay out of sim/runs/)

  • Grid extension past the old delay and loss ceilings, and follow-ups.
  • Outlier robustness with an NIS gate, turn-aware CT and IMM trackers, switching targets, and gated constant acceleration.
  • Vision error shape and a camera-shaped bias model.
  • Slow-motor gains, drag long check, send-period hold correction, monitor mission noise sweep.
  • Late-packet re-filtering at 2x and 3x jitter, with an independent capture-order check.
  • Full-chain closed-loop pilot, plus an opt-in lead goal rule (--lead: a constant-turn prediction aimed the 0.75 s controller lag plus the setpoint radio delay ahead; the old velocity rule stays the default and its rows are unchanged) and a pure delay sweep (--delay-sweep). One hand check (circle radius) failed against the sim and is kept as a strict xfail with its reason.
  • Tracker float32 and fixed-tick numeric readiness, plus runs explaining why snapping a capture to the next 100 Hz tick looked about 1 % better than fusing at the exact time: it comes from scoring at the 10 ms ticks (on a 1 ms scoring grid the sign flips) and from a filter q larger than a straight line needs. No bug in the exact-time path; exact-time fusion stays the reference.

Platform and test hygiene

  • LF line endings for result writers so Windows reruns match registered hashes.
  • Off-macOS float tolerance in the negative-control comparisons of the vision error shape and grid-extension side runs.
  • Platform tolerance test with planted bugs; corrected comms hand checks (four stale xfails removed).
  • tools/tests/test_reproduce_registry_coverage.py: byte-checks every committed registered output and fails if a new unregistered result folder appears. Six existing unregistered folders are allowlisted for review (burst-bridge-radio, d4-ellipsoid-diagnosis, e-exp035-logs, missions-atlas, retry-fades-exp038, s6-interaction).
  • Opt-in ground plane (default off).

Testing

  • Each merge was tested on Windows with its own test files; the measured-input chain (10 files) passed 373 tests on the merged branch.
  • CI (windows-2022, full suite) passed on 547d340 (both runs). On 304e929 the push run passed and the pull-request run failed one exact-hash check in sim/tests/test_ground.py: the same code gave different bits on a runner with a different CPU. That check now compares within a relative 1e-9 (09ce9e0). The run on the current head is in the checks below.
  • A final whole-PR review read every file touched by more than one merge and found no broken logic; it asked for description fixes and a naming cleanup. A re-review found four more internal labels in side-run tools and the ground test; those are replaced in ae3d33d and 3d55938 (the final ground test profile name gives a derived profile identical to the old one in every field).

Not in this PR

  • Echo firmware for the radio bench (not started).
  • Branches that still need a decision: radio log quirks, frame and time layer, public-number linter, camera bias scorer, radio airtime model fixes.

…ned S6 rerun

The options read the vision fit (per_axis.std_m, sensor_config.bias_m and its dropout settings) and map camera axes onto world axes with --axis-map. The shaped path reuses sim/sensor.py with the registered sensor seed and sim/s6_recovery.deliver for the link, so equal per-axis noise, zero bias and zero dropout give the registered packets exactly. With every option off the outputs are unchanged.
… (process pools fail from detached shells on Windows)
tools/preflight_bench_data.py reads a bench folder and fails, with a one-line
reason and fix, on anything the intake would refuse or that would give
misleading numbers: unreadable or unprovenanced logs, low detection rate,
truth that disagrees with the file name or the camera, sidecar and
calibration sha256 mismatches, frame size mismatches, mixed statuses or
sessions, colliding names and an unusable intake output folder. It reuses
the intake's own readers and checks.
… and loss 0.9, with an exact control against the registered rows
GroundPlane clamps the body at the floor after each RK4 step, records the in-step crossing speed, and flags touchdowns above a placeholder 2.0 m/s. SixDOFInterceptor and ActuatedQuadrotor take ground=None by default, so the default path is unchanged (saved 77be94f history hashes still match).
The fit now adds WARNING lines to its assumptions for dropped frames,
duplicate or backward capture stamps, frames seen only under another
subject id, pose flips behind the camera, outliers, frozen frames and very
large distances. It refuses positions that are clearly not in meters and
refuses the latency fit when any latency is over 5 s. Clean logs fit to the
same bytes as before. The reader says when a last line was cut off
mid-write, and the intake summary points at fit warnings.
A noise_std_m that is text, true/false, NaN, infinite or below zero now stops the lookup with a plain message. A summary cell on the no-loss main row whose noise_std_m is not a number is named in the error; before, it was reported as a bad delay_s. The registered S6 summary has only numeric noise levels, so its lookups are unchanged.
In the S6 link rerun, a small link JSON that says measured must list its source files under inputs, each with a name and a lowercase sha256 (the block the vision link converter writes), and a fit whose trace says measured must keep its trace_id. Unlabelled, synthetic, fixture and wrong-case inputs are unchanged. The one existing test that fed a bare measured JSON now adds an inputs block; its assertions are the same.
synth --vsync-hz now also writes display_refresh_hz at the top of run_meta.json. analyze refuses a --refresh-hz that disagrees with a recorded rate or a recorded rate that is not a positive number, and prints a warning when a rate is recorded but the flag was not given. No default changes: the refresh-phase term is still added only with --refresh-hz, and runs without the field give the same output.
…tests and output keys with plain descriptions
…ndows CI runners with different CPUs give different bits
…d test with plain names (profile name keeps the same derived numbers)
…one in every field (the previous rename changed pitch, which the dynamics do not read)
@Whyater
Whyater merged commit 96fc86c into main Nov 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant