Skip to content

outer-loop status should show in-progress candidates #1360

Description

@lukeinglis

Problem

factory outer-loop status only shows completed evaluations. While candidates are mid-run (improve cycle or gate scoring), the output looks like nothing is happening:

=== Outer Loop Status ===
Benchmark: forecastbench
Generation: 0
Total evaluations: 0
Best score: 0.0000
Budget remaining: 9

Meanwhile, a sub-CEO is actively running an improve cycle and the gate is scoring in a worktree — but the status command gives no indication of this. The only way to see progress is to watch the terminal where evaluate is running.

Observed behavior

Ran factory outer-loop calibrate followed by factory outer-loop evaluate. The evaluate command runs in the foreground and streams logs, but factory outer-loop status (from another terminal) shows 0 evaluations even though a candidate has completed its improve cycle and is being scored by the gate.

Expected behavior

factory outer-loop status should show:

  • Which candidates are currently being evaluated
  • What phase each is in (improve cycle / gate scoring / complete)
  • Elapsed time per candidate
  • Which worktree each candidate is running in

Something like:

=== Outer Loop Status ===
Benchmark: forecastbench
Generation: 0
Total evaluations: 0 (1 in progress)
Budget remaining: 9

In progress:
  evolve-gen0-0d4cc86a  [gate scoring]  3m12s  wt-evolve-gen0--7a8111fd

Completed:
  (none yet)

Context

This came up during the first real evaluation run with the forecastbench benchmark (PR #1332). Each candidate takes several minutes (improve cycle + 2-round gate with API calls), so there's a long window where status looks empty.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions