Skip to content

Complete Rust orchestration migration - #7

Draft
areshand wants to merge 16 commits into
mainfrom
codex/rust-migration-tests
Draft

Complete Rust orchestration migration#7
areshand wants to merge 16 commits into
mainfrom
codex/rust-migration-tests

Conversation

@areshand

@areshand areshand commented Aug 13, 2026

Copy link
Copy Markdown
Owner

What changed

  • Move launch, tmux subprocess orchestration, status/watch, recovery, completion, and subagent gate behavior into the Rust multiagent binary.
  • Remove all production shell CLI wrappers. launch.sh is the only retained source-checkout bootstrap and immediately executes multiagent launch.
  • Keep tmux as the PTY owner; Rust invokes tmux without reimplementing terminal semantics.
  • Build and install the release Rust binary in native-solver container images.
  • Reduce the SWE-bench adapter to public-input setup, process lifecycle, and workspace handoff.
  • Wait only for the Rust orchestrator window to exit, then expose committed and untracked workspace changes to EvalScope.
  • Remove adapter status/evidence parsing, pane capture, Go validation singleflight, pre-run diff cleanup, file filtering, Git-diff diagnostics, and submission-rejection logic.
  • Leave patch extraction and correctness evaluation exclusively to EvalScope and the official SWE-bench verifier.
  • Update prompts, docs, evaluation reporting, and tests for the pass-through submission boundary.
  • Make parallel SWE aggregation read shard summaries from the configured report directory.

Why

The prior split across Rust, Python, and shell duplicated command routing and state transitions. The SWE adapter also behaved like a second verifier: a potentially scoreable patch could be discarded because an agent omitted adapter-specific evidence markers. This migration creates one locked production writer and one command surface while preserving the v1 durable-state format and tmux interactive behavior.

The adapter now organizes the submission only: it launches the production workflow, waits for the orchestrator lifecycle, materializes worker commits, marks all untracked files intent-to-add, and returns the workspace to EvalScope. It neither validates nor pre-accepts the patch.

Validation

  • cargo test --locked — 14 passed
  • bash tests/run.sh — full Rust, shell, lifecycle, evaluation, provenance, and migration suite passed
  • Focused native-solver outcome/import/provenance tests — 21 passed
  • Python compilation, shell syntax, and git diff --check passed

Regression tests verify that orchestrator exit prepares the workspace for the official scorer, committed worker changes are materialized, and every untracked source or test file is included in the EvalScope diff. Contract assertions also prevent reintroduction of adapter evidence parsing, rejection diagnostics, validation wrappers, and status files.

First-10 SWE-bench Pro regression

Exact-revision rerun: solver f15f225e05163bacf99296c83d64a75e5e99cd96, EvalScope 1.8.1 (fce1d21), SWE-bench Pro ca10a60, gpt-5.4, two shards of five.

  • 5/10 resolved (0.5), unchanged from the preceding first-10 migration run.
  • 10/10 solver handoffs reached EvalScope; 0 runner errors, 0 timeouts, and 0 adapter-side rejections.
  • 9/10 handoffs contained a non-empty diff; index 7 completed successfully but produced an empty patch and was scored unresolved by the official verifier.
  • Shards: indices 0–4 scored 0.6; indices 5–9 scored 0.4.
  • Resolved indices: 1, 2, 3, 5, and 6.
  • Both provenance bundles validated successfully.

The rerun confirms no aggregate score regression and preserves the pass-through boundary. It also exposed a packaging concern rather than an evaluation concern: under the intentional no-filter policy, repository-local runtime artifacts such as Redis AOF files and patch.txt are included in the submitted diff. Correctness remains exclusively the official verifier's responsibility; framework artifacts should preferably be written outside /app rather than introducing semantic validation into the adapter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant