Problem
factory outer-loop status only shows completed evaluations. While candidates are mid-run (improve cycle or gate scoring), the output looks like nothing is happening:
=== Outer Loop Status ===
Benchmark: forecastbench
Generation: 0
Total evaluations: 0
Best score: 0.0000
Budget remaining: 9
Meanwhile, a sub-CEO is actively running an improve cycle and the gate is scoring in a worktree — but the status command gives no indication of this. The only way to see progress is to watch the terminal where evaluate is running.
Observed behavior
Ran factory outer-loop calibrate followed by factory outer-loop evaluate. The evaluate command runs in the foreground and streams logs, but factory outer-loop status (from another terminal) shows 0 evaluations even though a candidate has completed its improve cycle and is being scored by the gate.
Expected behavior
factory outer-loop status should show:
- Which candidates are currently being evaluated
- What phase each is in (improve cycle / gate scoring / complete)
- Elapsed time per candidate
- Which worktree each candidate is running in
Something like:
=== Outer Loop Status ===
Benchmark: forecastbench
Generation: 0
Total evaluations: 0 (1 in progress)
Budget remaining: 9
In progress:
evolve-gen0-0d4cc86a [gate scoring] 3m12s wt-evolve-gen0--7a8111fd
Completed:
(none yet)
Context
This came up during the first real evaluation run with the forecastbench benchmark (PR #1332). Each candidate takes several minutes (improve cycle + 2-round gate with API calls), so there's a long window where status looks empty.
Problem
factory outer-loop statusonly shows completed evaluations. While candidates are mid-run (improve cycle or gate scoring), the output looks like nothing is happening:Meanwhile, a sub-CEO is actively running an improve cycle and the gate is scoring in a worktree — but the status command gives no indication of this. The only way to see progress is to watch the terminal where
evaluateis running.Observed behavior
Ran
factory outer-loop calibratefollowed byfactory outer-loop evaluate. The evaluate command runs in the foreground and streams logs, butfactory outer-loop status(from another terminal) shows 0 evaluations even though a candidate has completed its improve cycle and is being scored by the gate.Expected behavior
factory outer-loop statusshould show:Something like:
Context
This came up during the first real evaluation run with the forecastbench benchmark (PR #1332). Each candidate takes several minutes (improve cycle + 2-round gate with API calls), so there's a long window where status looks empty.