Skip to content

feat(evaluation): add agent-confirmed PointNav evaluation - #9

Merged
Staaaaaaaaar merged 4 commits into
mainfrom
feat/simple-navigation-evaluator
Aug 17, 2026
Merged

feat(evaluation): add agent-confirmed PointNav evaluation#9
Staaaaaaaaar merged 4 commits into
mainfrom
feat/simple-navigation-evaluator

Conversation

@Staaaaaaaaar

Copy link
Copy Markdown
Owner

Motivation

Implement roadmap PR 09 with a deterministic PointNav evaluator, while correcting two architecture gaps discovered during implementation:

  • navigation completion must be declared by the Agent instead of inferred from a transient goal crossing;
  • concrete Task/Evaluator selection needs one composition root in the Experiment container.

Scope

  • Add the AgentTaskState ROS contract with IDLE, RUNNING, SUCCEEDED, and FAILED states.
  • Require an Agent to stop producing non-zero commands before reporting SUCCEEDED.
  • Implement ordered 3D trajectory sampling and simple PointNav metrics in simulation time.
  • Confirm Agent success against the first ground-truth pose at or after the Agent result.
  • Add evaluator integration through the orchestrator's serialized termination path.
  • Detect a frozen simulation clock with a steady-time watchdog.
  • Add rh_bringup as the sole concrete Experiment-container composition root.
  • Remove Task, Evaluator, and generic lifecycle application entrypoints.

Out of scope

  • Result persistence and Experiment summaries.
  • Multi-Episode execution and retry policy.
  • Collision metrics, SPL, PoseNav, and automatic goal sampling.
  • Isaac Sim, GPU, and production three-container deployment validation.

Architecture changes

The final success decision now uses two signals:

  1. the Agent stops its navigation control and reports SUCCEEDED;
  2. the Evaluator confirms that ground-truth position remains inside the configured PointNav radius.

Passing through the goal without Agent completion no longer terminates the Episode. Agent failure, simulation-time timeout, infrastructure failure, and operator abort remain distinct termination paths.

rh_experiment remains implementation-independent and exposes Task/Evaluator factory boundaries. rh_bringup is the only package that imports concrete implementations and currently provides the reviewed pointnav_simple assembly.

Implementation

  • Add C++/Python typesupport and contract coverage for AgentTaskState.
  • Extend the Mock Agent with transient-local task-state publication and sequence checks.
  • Add pure trajectory and evaluation modules plus a TF2-backed ROS observer.
  • Add evaluator submission and clock-watchdog support to the single-Episode orchestrator.
  • Add ros2 run rh_bringup experiment --profile pointnav_simple.
  • Update repository lint discovery and architecture documentation.

Testing

  • Clean development-container build using new temporary build/install/log directories: 16 packages built.
  • Clean full test run: 192 tests, 0 errors, 0 failures, 0 skipped.
  • Repository structure, Markdown, YAML, Ruff, and git diff --check: passed.
  • Clean install entrypoint audit: only rh_bringup experiment is exposed for the central runtime.
  • Integration coverage verifies TF transformation, fixed trajectory metrics, goal pass-through without termination, Agent completion plus ground-truth confirmation, no evaluator control publisher, and frozen /clock handling.

Acceptance criteria

  • Scope and out-of-scope match the implementation.
  • Build, lint, and relevant tests pass.
  • Documentation is updated.
  • Acceptance evidence is included.

Known limitations

  • Metrics remain in memory until the Recorder PR.
  • The current orchestrator still intentionally supports one Episode.
  • Agent task state is an initial compact protocol; richer feedback and failure taxonomy remain extensible.
  • CPU mock integration is not evidence of Isaac Sim or robot-controller compatibility.

Follow-up

  • PR 10: durable result recorder.
  • PR 11: multi-Episode execution and failure policy.
  • Runtime deployment work will assign the bringup command to the Experiment service in the three-container stack.

@Staaaaaaaaar
Staaaaaaaaar merged commit a5a578f into main Aug 17, 2026
1 check passed
@Staaaaaaaaar
Staaaaaaaaar deleted the feat/simple-navigation-evaluator branch August 17, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant