Experiment: replay verification at scale + FP perturbation sensitivity - #47
Experiment: replay verification at scale + FP perturbation sensitivity#47gmliao wants to merge 8 commits into
Conversation
…ation - ReevaluationRunner --record: headless batch recording with deterministic MoveTo injection; seed via landID - MovementSystem: env-driven perturbation hook (HERO_PERTURB_*) for the replay sensitivity experiment; inert when unset - Fix order-dependent game logic: sorted-key iteration over players/ monsters/turrets and lowest-id tie-break in nearest-target selection. Dictionary iteration order made 1,200-tick replays diverge (28/30 failed); 30/30 verify cleanly after the fix - Runner: re-check server-event mismatches by content, reporting pure sequence-numbering differences (known replay accounting gap) as a warning instead of a failure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sm3UKGwyUw6RyELpBMFd3H
30 recordings x 1,200 ticks: 0 hash mismatches, run1 == run2 for all. Perturbation at tick 600: sub-LSB float noise absorbed 10/10 by fixed-point quantization; >=1 LSB shifts detected 10/10 with 0-tick latency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sm3UKGwyUw6RyELpBMFd3H
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3181997b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // Experiment hook: shift the quantized x coordinate by eps raw LSB units (1 LSB = 0.001) | ||
| if let p = MovementPerturbation.current, p.mode == "fixed", ctx.tickId == p.tick { | ||
| // eps is in raw LSB units (1 LSB = 0.001 world units); eps/1000 quantizes back to exactly eps LSB | ||
| player.position = Position2(v: player.position.v + IVec2(x: p.eps / 1000.0, y: 0.0)) |
There was a problem hiding this comment.
When HERO_PERTURB_MODE=fixed, this converts a raw-LSB count to world-space Float by manually dividing by 1000 and then requantizing it. For fractional, sufficiently large, or precision-losing HERO_PERTURB_EPS values, the applied delta can differ from the requested LSB count, trap during Int32 conversion, or wrap when added to the coordinate, invalidating the experiment rather than producing a controlled perturbation. Parse a bounded integer LSB value and add a semantic DeterministicMath helper instead of manipulating the scale directly.
AGENTS.md reference: AGENTS.md:L130-L138
Useful? React with 👍 / 👎.
| @@ -0,0 +1,117 @@ | |||
| # Replay verification at scale + FP perturbation sensitivity | |||
There was a problem hiding this comment.
This new README.md is predominantly Traditional Chinese, but no README.zh-TW.md counterpart or language-switching links are added; the new English design note likewise has no synchronized Chinese version. Consequently English readers do not receive the default-language experiment report, and future regeneration can update only one language. Add paired English/Traditional Chinese documents using the required filenames and switcher links.
AGENTS.md reference: AGENTS.md:L248-L254
Useful? React with 👍 / 👎.
Records and logs are deterministically reproducible via --record/--verify.
9 variant recordings (players 2/10, MoveTo cadence 5) and two 12,000-tick long-horizon recordings, all 0 mismatches; totals now 41 recordings / 70,800 ticks. crossarch-verify.sh replays the same arm64-recorded batch on an x86_64 machine.
…; Float exists only at boundaries
1 participant
RQ3 reinforcement experiment (design note:
Notes/plans/2026-08-31-replay-scale-fp-perturbation-experiment-design.md).Results
Part A — replay verification at scale (aggregate)
Core matrix: 30 seeds × 1,200 ticks (5 players, MoveTo every 20 ticks). Robustness variants: players 2 and 10 (3 seeds each), MoveTo cadence 5 (3 seeds), plus two 12,000-tick (~10 min) long-horizon recordings. 0 mismatches everywhere → 95% rule-of-three upper bound on per-tick mismatch probability ≈ 4.2e-5.
Part B — perturbation sensitivity (injected at tick 600, 10 recordings each)
Conclusion
Per-tick hash verification stays at 0 mismatches over 41 recordings / 70,800 ticks across player-count, input-cadence, and duration variations; sub-quantization floating-point noise cannot enter the fixed-point state (quantization barrier), and once state actually shifts by even 1 LSB (= one fixed-point quantization step, 0.001 world units — not a float ULP), audit-time replay verification exposes it in the same tick with no blind spots. This is an offline audit/CI mechanism, not runtime monitoring.
Notable findings (in this PR's code commits)
sequenceonly. Runner now re-checks by content and warns instead of failing.Cross-architecture follow-up
crossarch-verify.shreplays the same arm64-recorded batch on an x86_64 machine (records tarball prepared separately); results to be appended.Axes NOT varied
perturbation tick (600, applied to the first 10 core seeds only), turrets (0). Same-arch (arm64) in this PR; cross-arch run pending per above.
Environment:
d23d66d/fcdd41e, Apple Swift 6.3.2, release, Apple M2 (arm64).regenerate.py --check= OK.🤖 Generated with Claude Code
https://claude.ai/code/session_01Sm3UKGwyUw6RyELpBMFd3H