Phase 7 Step 6: End-to-end validation soak (operator)

[aberson]

Build step 6 of 6 — Phase 7 (umbrella #162)

Build doc: documentation/plans/phase-7-build-plan.md §3 Step 6
Type: operator (live multi-hour soak — not a /build-step code change)

What to build

Run the full /improve-bot-advised loop unattended to confirm the staleness signal → soak scheduling → daemon recovery path works end-to-end against the live system. This was split out of the old Step 3 (which buried a 1h live soak inside a code step's Done-when, per plan-review §22).

Done when (operator runs + reports)

1. With reward rules freshly edited, run /improve-bot-advised --self-improve-code --hours 8.
2. Staleness detected on iteration 1 (reward edit newer than newest checkpoint).
3. A soak improvement is emitted and runs to completion (2–4h, clamped ≤ 50% of budget).
4. Post-soak deterministic eval shows a measurable Elo/WR delta (sign doesn't matter — proves the path works end-to-end).
5. Loop resumes and finishes remaining budget without starving other iterations.

Operator commands

# edit a reward rule first so the policy is provably stale, then:
uv run python -m orchestrator.staleness <current-version>   # sanity: is_stale should be true
uv run improve-bot-advised --self-improve-code --hours 8

What to look for

Check	Expected
Iteration 1 staleness	is_stale: true, reason cites reward-edit-after-checkpoint
Soak emitted + clamped	One type: soak, duration ≤ min(remaining/2, 4h)
Daemon recovery	API-only backend restarts cleanly after the soak; data/ intact
Budget	Soak-type time ≤ 50% of total --hours
End-to-end	Post-soak eval delta non-zero; loop completes remaining budget

Depends on

Steps 1–5 (#180–#184) all shipped + green.

Parallel-safe with

none — strictly sequential (this is the phase-end validation gate; it observes the behavior Steps 1–5 build).

Produces

A run-log / observation report (operator-authored). No code artifact.

Operator workflow notes

• This is a Type: operator step: /build-phase halts here and hands off to the operator; it does not spawn /build-step.
• It is the Phase 7 validation gate (plan §5/§6).

---

Synced from documentation/plans/phase-7-build-plan.md by /repo-sync at acd147f