Skip to content

Add an independent OxMPL planner comparison - #8

Merged
Dylan-Gallagher merged 5 commits into
mainfrom
evidence/oxmpl-independent-baseline
Aug 18, 2026
Merged

Dylan-Gallagher merged 5 commits into
mainfrom
evidence/oxmpl-independent-baseline

Conversation

@Dylan-Gallagher

Copy link
Copy Markdown
Owner

Summary

  • add a predeclared comparison against independently maintained OxMPL 0.6.0
  • retain all 1,085 paired seed trials over the 217 frozen blocked-direct queries
  • independently revalidate every external path with arm-lab's MuJoCo checker
  • commit the full CSV, generated report, exact-host two-run record, and fast CI artifact verifier

Result

  • in-repository RRT-Connect: 271/1,085 successes
  • OxMPL RRT-Connect: 290/1,085 successes
  • paired cells: 271 both, 0 in-repository only, 19 OxMPL only, 795 neither
  • independently valid OxMPL paths: 290/290; invalid paths: 0
  • blocked queries with at least one successful seed: 56/217 in-repository, 58/217 OxMPL

The protocol was committed before the external dependency, evaluator, or result-bearing run. A write followed by an immediate full check matched every status and exact path digest after normalizing wall time.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --release --workspace --all-targets
  • multi_query_bench --check
  • randomized_eval --check
  • oxmpl_baseline --check
  • oxmpl_baseline --verify-artifacts
  • independent 1,085-row CSV key/seed/status invariant audit

Boundaries

This is a frozen-cohort cross-implementation comparison, not a universal or compute-matched planner ranking. The in-repository rows use a 2,000-iteration budget; OxMPL uses a predeclared 250-ms wall-time budget. The integer seeds are paired labels but feed different PRNGs. The seed-level McNemar/binomial calculation is retained as descriptive only because five seeds share each query. Collision checks remain sampled simulation checks, with no continuous-safety, hardware, or sim-to-real claim.

Development disclosure

OpenAI Codex assisted with implementation, testing, analysis, and documentation. Dylan Gallagher is responsible for reviewing the changes, evidence, and public claims.

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
@Dylan-Gallagher
Dylan-Gallagher merged commit a5321fe into main Aug 18, 2026
1 check passed
@Dylan-Gallagher
Dylan-Gallagher deleted the evidence/oxmpl-independent-baseline branch August 18, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant