Skip to content

Add predeclared randomized planning evaluation - #5

Merged
Dylan-Gallagher merged 3 commits into
mainfrom
evidence/randomized-planning-eval
Aug 17, 2026
Merged

Dylan-Gallagher merged 3 commits into
mainfrom
evidence/randomized-planning-eval

Conversation

@Dylan-Gallagher

Copy link
Copy Markdown
Owner

Summary

  • add a protocol frozen in its own commit before the evaluator and result-bearing runs
  • retain 300 independently sampled joint-space queries across the three shipped scenes
  • record all 2,253 planner outcomes and 276 nominal controller replays in raw CSV artifacts
  • add generated aggregate reporting, exact artifact verification, and a bounded CI reproduction

Results

  • direct interpolation: 83/300 sampled-clear queries (27.7%, 95% Wilson 22.9--33.0%)
  • canonical RRT-Connect: 138/300 successes (46.0%, 95% Wilson 40.4--51.7%), including 55/217 recoveries on blocked-direct queries
  • blocked-query goal-bias ablation: 271/1,085 default successes versus 270/1,085 with zero goal bias
  • position PD: 0/138 numeric and full tracking passes
  • PD + velocity feedforward: 104/138 numeric passes and 90/138 full zero-sampled-penetration passes
  • retained negatives: 162 canonical planning failures and 37/276 tracking cases with sampled penetration, maximum depth 3.2225 mm

Validation

  • cargo fmt --all -- --check
  • cargo clippy --locked --workspace --all-targets --all-features -- -D warnings
  • cargo test --locked --release --workspace --all-targets
  • cargo run --locked --release -p arm-lab-demo --bin multi_query_bench -- --check
  • cargo run --locked --release -p arm-lab-demo --bin randomized_eval -- --check
  • separate row-level audit of cohort sizes, unique queries, seed formulas, planner layouts, paired controller replays, gates, and reported aggregates

Scope

This changes evaluation and documentation only; the planner, controllers, collision predicate, trajectory limits, actuator model, and thresholds are unchanged. The evidence is deterministic simulation with sampled collision checks. It is not hardware, continuous-safety, grasping, workspace-uniform, or sim-to-real evidence.

Freeze the query distribution, planning ablation, nominal tracking comparison, artifact schema, and interpretation boundaries before generating any result-bearing randomized benchmark output.

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Implement the frozen 300-query multi-scene sample, blocked-path goal-bias ablation, paired nominal controller replay, raw artifact rendering, and deterministic artifact checker without changing production planning or control.

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
Retain all 300 predeclared queries, 2,253 planner outcomes, and 276 nominal controller replays, including every unconnected and sampled-penetration result. Add generated aggregate reporting and exact artifact verification without changing planner or controller behavior.

Signed-off-by: Dylan Gallagher <76841713+Dylan-Gallagher@users.noreply.github.com>
@Dylan-Gallagher
Dylan-Gallagher merged commit 31efe8c into main Aug 17, 2026
1 check passed
@Dylan-Gallagher
Dylan-Gallagher deleted the evidence/randomized-planning-eval branch August 17, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant