Skip to content

feat(experiment): add multi-episode orchestration - #11

Merged
Staaaaaaaaar merged 1 commit into
mainfrom
feat/multi-episode-orchestrator
Aug 18, 2026
Merged

feat(experiment): add multi-episode orchestration#11
Staaaaaaaaar merged 1 commit into
mainfrom
feat/multi-episode-orchestrator

Conversation

@Staaaaaaaaar

Copy link
Copy Markdown
Owner

Motivation

PR 07 established the authoritative single-Episode lifecycle and PR 08-10 added PointNav, evaluation, and durable recording. PR 11 connects those boundaries into an ordered multi-Episode Experiment with explicit continuation and infrastructure-failure policies.

Scope

  • execute every configured Episode serially without restarting Env, Agent, or Experiment processes
  • recreate and dispose Episode-scoped Task and Evaluator resources between Episodes
  • reset Env before Agent, republish the immutable task, and reuse the existing manual/automatic start path
  • bind evaluator termination callbacks to the originating Episode ID and reject delayed callbacks
  • freeze evaluator metrics and trajectory during safe finalization
  • persist each Episode and the final Experiment summary through ResultRecorder
  • publish EpisodeResult after durable Episode artifacts are committed
  • preserve the SingleEpisodeOrchestratorNode compatibility import

Failure Policy

  • SUCCESS, TIMEOUT, ABORTED, navigation FAILURE, and INVALID_TASK finish the current Episode and continue when another Episode remains
  • component ERROR/staleness, clock freeze, and Env/Agent reset failures stop the Experiment
  • no process or container is automatically restarted

Architecture Changes

  • add an ROS-independent ExperimentController for ordered Episode lifecycles
  • add a recorder-ready EpisodeEvaluationResult boundary
  • make Task publishers and Evaluators explicitly Episode-scoped and closeable
  • keep Experiment lifecycle mutation serialized on the existing control thread
  • retain all existing ROS interfaces without wire changes

Testing

  • make dev-lint
  • make dev-check
  • 16 packages built
  • 207 tests passed, 0 errors, 0 failures, 0 skipped
  • protocol-stub tests cover three Episodes, unchanged process ID, reset ordering, independent artifacts, EpisodeResult publication, stale callback rejection, INVALID_TASK continuation, and reset-failure stop policy
  • a reference vertical test runs Mock Env + Mock Agent + PointNav + Simple Evaluator + Recorder for three consecutive simulation-time timeout Episodes without process restart

Acceptance Criteria

  • at least three Episodes execute without restarting services
  • Episode-scoped resources and state do not leak between runs
  • each Episode produces independent artifacts and aggregate counts are correct
  • success, timeout, abort, invalid task, and reset/component failure paths are covered
  • status and simulation-clock watchdog behavior remains active
  • interrupted Experiments retain parseable complete=false results

Known Limitations

  • Episodes remain serial; parallel execution is out of scope
  • no automatic process/container restart or distributed recovery
  • results_root must be configured to enable filesystem recording and experiment_id must be unique for immutable result directories
  • three-container Compose and CI black-box execution remain PR 12

Follow-up

PR 12 will run this vertical path across the production-like Env, Agent, and Experiment container boundaries.

Roadmap: PR 11 - Multi-Episode and Failure Policy

@Staaaaaaaaar
Staaaaaaaaar merged commit cab169f into main Aug 18, 2026
1 check passed
@Staaaaaaaaar
Staaaaaaaaar deleted the feat/multi-episode-orchestrator branch August 18, 2026 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant