Skip to content

Add an independent IK baseline and restart ablation - #9

Merged
Dylan-Gallagher merged 4 commits into
mainfrom
evidence/ik-independent-baseline
Aug 18, 2026
Merged

Dylan-Gallagher merged 4 commits into
mainfrom
evidence/ik-independent-baseline

Conversation

@Dylan-Gallagher

Copy link
Copy Markdown
Owner

Summary

  • add a predeclared, deterministic comparison of arm-lab IK with independently maintained k 0.32.0 on the existing 1,000-target full-pose cohort
  • retain all 1,000 target rows and all 4,000 solver-result rows, with every success rechecked through arm-lab FK under one pose-and-limit rule
  • separate cold-start performance from eight fixed restart starts and replay every artifact in CI
  • correct the README's old description of the 97.8% result: it is home plus up to eight seeded restarts, not a cold-start result

Results

Variant Common FK-checked success
arm-lab, home only 713/1,000 (71.3%)
arm-lab, home + up to 8 restarts 978/1,000 (97.8%)
k, home only 328/1,000 (32.8%)
k, home + the same 8 restart vectors 718/1,000 (71.8%)

For the restart-enabled variants, both solve 714 targets, arm-lab alone solves 264, k alone solves four, and neither solves 18. Their union solves 982/1,000. The exact paired sign/McNemar value is 8.998e-73, reported only as a descriptive statistic for this frozen cohort.

The pre-existing 97.8% arm-lab control was known before the protocol was frozen and is explicitly labeled non-blind. The cold-start, independent-solver, paired-cell, and union outcomes were not known before the evaluator ran.

Reproducibility

  • protocol committed before evaluator and new outcomes: d9078f1b95a159cc09acf2995f895cd91c23aa6e
  • evaluator committed before the first new result-bearing run: 6b3320a5e73406eaa9c497eeba3bc96960322e62
  • a post-outcome corrective commit changes only tiny-probability formatting, not any target, result row, success rule, or analysis
  • two consecutive full runs reproduced every result byte
  • CI now replays the 4,000 rows in addition to the existing multi-scene, randomized, and independent-planner artifacts

Validation

  • cargo fmt --all -- --check
  • strict workspace Clippy across all targets and features
  • 68/68 release tests
  • full deterministic checks for the multi-scene, randomized, OxMPL, and new IK artifacts
  • fresh pre-publication IK regeneration: every generated byte matched

Claim boundaries

Targets are generated by FK from in-limit configurations, so they are reachable under this exact model but are not Cartesian-uniform or task-representative. Evaluation is collision-unaware and simulation-only. Maximum starts and iteration caps are matched, but per-iteration work, update rules, damping, and early stopping differ; this is not a runtime, compute, hardware, sim-to-real, workspace-wide, or universal solver ranking.

Generated-by: OpenAI Codex (GPT-5; accessed 2026-08-18)

Freeze the target cohort, solver configurations, paired analyses, common success rule, and outcome-retaining verification gates before implementing or running the comparison.

Generated-by: OpenAI Codex (GPT-5; accessed 2026-08-18)

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Implement the frozen four-variant comparison, common FK-based outcome rule, full CSV/report rendering, paired statistics, and byte-exact artifact verification without changing the library solver.

Generated-by: OpenAI Codex (GPT-5; accessed 2026-08-18)

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Render the predeclared exact paired-test probability in scientific notation instead of rounding a nonzero value to 0.000000. Solver configurations, targets, outcome rules, and result rows are unchanged.

Generated-by: OpenAI Codex (GPT-5; accessed 2026-08-18)

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Retain all 1,000 targets and 4,000 solver outcomes, add the restart ablations and independent k baseline to the evidence index, record exact validation provenance, and replay the artifacts in CI.

Generated-by: OpenAI Codex (GPT-5; accessed 2026-08-18)

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
@Dylan-Gallagher
Dylan-Gallagher merged commit 4e36ab5 into main Aug 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant