Outer Loop v1 Fixes: Leftover Work — Phase 4 + Code Gaps

[akashgit]

Outer Loop v1 Fixes: Leftover Work

Parent: #1272 (Post-Mortem and Fix Plan)
PR: #1273 (code fixes — targets feat/outer-loop-phase1-foundation)

Context

Design mode session on 2026-08-15 implemented all 10 code fixes from #1272. However, Phase 4 (Operational Validation) was never attempted, and several code gaps were identified during review. This issue captures everything that needs to happen next.

What the CEO Fucked Up

1. Phase 4 Was Never Run — Despite Docker Being Available

Docker was available (docker --version → 29.7.2). A FeatureBench pydantic image existed (libercoders/featurebench-specs_pydantic-instance_78693714, 29.7GB). The user explicitly asked for Phase 4 (operational validation — actually running the outer loop on FeatureBench). The plan had concrete execution steps.

What happened: The first Builder invocation timed out at 10 minutes (Bash tool limit). When the CEO re-invoked, it scoped the task to "verify, test, lint, commit, PR" and silently dropped Phase 4 entirely. The CEO then declared the cycle "complete" without ever attempting the operational validation that the user specifically requested. This is exactly the failure pattern from CEO playbook item ceo-00014: "Writing code that runs pipelines is not the same as running pipelines."

2. PR Targeted Wrong Branch

The PR was opened targeting main instead of feat/outer-loop-phase1-foundation. Fixed during review, but the Builder should have gotten this right — the task said "create from feat/outer-loop-phase1-foundation" but never specified the PR base.

3. Fix #7 Uses ThreadPoolExecutor, Not ProcessPoolExecutor

The plan explicitly specified ProcessPoolExecutor (bypasses GIL for true parallelism). The research report recommended it. The code uses ThreadPoolExecutor. For Docker subprocess calls this probably doesn't matter (I/O bound, not CPU bound), but it's not what was specified. If the evaluation pipeline ever does CPU-heavy work (graph manipulation, scoring), ThreadPoolExecutor won't parallelize it.

4. Fix #6 Prompt Mutation Is Sentence-Shuffle, Not LLM Crossover

The plan said "EvoPrompt crossover style — LLM-driven prompt rewriting." The implementation is _crossover_prompts() which splits prompts on periods and randomly picks sentences from parent A or parent B. This is a random sentence interleaver, not an LLM call. It works (tests pass, frozen segments preserved), but it's a significant downgrade from the spec. A real EvoPrompt implementation would call the LLM to synthesize a new prompt combining ideas from both parents.

This was likely a pragmatic choice — LLM calls during mutation are expensive and slow, and the Builder might not have had access to call Claude from within the test environment. But the gap should be documented.

5. Only 1 FeatureBench Image Available

Calibration needs 20-30 instances to select 10 training + 5 holdout. Only 1 pydantic image exists. Either more images need to be pulled, or the calibration needs to work with fewer instances (and log a warning). The CEO never checked how many images were actually available.

6. CEO Trusted Builder Summary Without Verifying

The first Builder said "all 10 fixes implemented." The CEO wrote PROCEED. The code reviewer then found Fix #7 was dead code (evaluate_batch existed but engine never called it). The CEO should have caught this by reading the engine.py diff more carefully before spawning the QA pipeline.

Leftover Work

Phase 4: Operational Validation (MUST DO)

This is the whole point. Run the outer loop on FeatureBench.

Step 1: Pull more FeatureBench images

# Check what instances are available
# Pull enough for 20-30 candidate instances
# Current: only 1 pydantic image

Step 2: Calibration run

• Run bare builder (1 node) on available FeatureBench instances
• Record per-instance pass/fail
• Select 10 training (0.3-0.7 difficulty) + 5 holdout
• If fewer than 15 instances available, adjust and log

Step 3: Evolutionary run

• Configure SwarmConfig with calibrated split, parallelism=4, generations=2-3
• Run SwarmEngine.run() with the fixed code
• Validate each fix is working in practice:
  • No WebSearch/WebFetch in agent logs
  • Fitness scores are raw pass rate (no 0.185 offset)
  • Holdout scores tracked every generation
  • Designer workflows score > 0.0
  • PROMPT_MUTATE produces functional variants
  • Parallel eval is faster than sequential
  • Per-instance results recorded
  • INSERT_NODE creates functional agents

Step 4: Report

• Document results in results/outer_loop_v2_report.md
• Compare to first experiment (issue Outer Loop v1: Post-Mortem and Fix Plan #1272 data)

Code Gaps to Fix

Gap 1: ThreadPoolExecutor → ProcessPoolExecutor

• File: factory/outer_loop/evaluator.py:114
• Change ThreadPoolExecutor to ProcessPoolExecutor
• Make evaluate() args picklable (they should be already since Workflow serializes to dict)
• Test with parallelism=4

Gap 2: Real LLM Prompt Crossover (v2)

• File: factory/outer_loop/mutations.py — _crossover_prompts()
• Current: random sentence interleaving
• Target: LLM call that synthesizes a new prompt from two parents
• This requires access to Claude (via factory agent or direct API)
• Can be deferred to v2 if the sentence shuffle shows evolutionary signal in Phase 4
• If sentence shuffle works: keep it as the cheap default, add LLM crossover as an optional upgrade
• If sentence shuffle doesn't work: this becomes P0

Gap 3: CLI Entry Points for Phase 4

• The plan specified CLI commands for calibration and evolution
• These don't exist yet — the Builder was supposed to create thin wrappers
• Either create factory outer-loop calibrate and factory outer-loop evolve commands, or document the Python API for running Phase 4 manually

Run Instructions

# From the remote machine (has Docker + FeatureBench images)
tmux new -s outer-loop
factory ceo /path/to/project --focus "<this issue number>" --mode design --from-plan <this issue number>
# Detach: Ctrl-B D
# Reattach later: tmux attach -t outer-loop

Files

All changes are on branch fix/outer-loop-v1-postmortem (PR #1273):

• factory/outer_loop/engine.py — clean lifecycle + parallel wiring
• factory/outer_loop/evaluator.py — raw pass rate + evaluate_batch
• factory/outer_loop/direct_evaluator.py — --disallowedTools + --network none
• factory/outer_loop/overfit.py — audit_generation + should_early_stop
• factory/outer_loop/mutations.py — PROMPT_MUTATE + functional INSERT_NODE
• factory/outer_loop/designer.py — populate_prompt templates
• factory/outer_loop/models.py — new fields (instance_results, parallelism, etc.)
• factory/outer_loop/subset.py — CalibratedSubsetSelector
• tests/test_outer_loop/test_postmortem_fixes.py — 67 new tests