Outer Loop v2: Wire InnerLoop Abstraction + Exhaust-Driven Evolutionary Search

[akashgit]

Outer Loop v2: Wire InnerLoop Abstraction + Exhaust-Driven Evolutionary Search

Parent: #1274 (Phase 4 + Code Gaps)
Post-mortem: #1274 Phase 4 was attempted but architecturally wrong — the outer loop bypassed the InnerLoop abstraction, used binary scoring with no exhaust analysis, and delegated execution to a Builder agent instead of orchestrating directly. This issue fixes all of that.

---

Relationship to Create Mode

The outer loop mode is the benchmark-driven scientific evolution of create mode.

Create mode (factory ceo --mode create) takes a human description, has the Strategist design a workflow, the Builder implement it, and registers it as a new mode. It's one-shot, human-driven, and constrained by what the Strategist can imagine.

The outer loop mode does the same thing — creates workflow modes — but replaces human judgment with benchmark evaluation and iteration:

	Create Mode (v1)	Outer Loop Mode (v2)
Input	Human idea / spec	Benchmark + seed workflow
Output	One registered mode	Best-evolved registered mode
Candidates	1 (Strategist's design)	Population (N variants per generation)
Quality signal	Human approves	Benchmark score + CycleRecord exhaust
Iteration	One shot	Evolutionary loop (generations)
Design driver	Strategist imagination	Reflection on empirical failure/success
Constraint	Limited to 5 agent roles	Same roles, but discovers novel combinations

The outer loop reuses create mode's infrastructure — DesignerAgent for generating variants, workflow graph primitives for defining modes, the mode registration system — and wraps it in a search loop where the benchmark replaces the human and reflection replaces intuition.

---

The Problem

The current outer loop (factory/outer_loop/engine.py) uses DirectFeatureBenchEvaluator as its fitness function. This evaluator:

1. Runs workflow agents directly via subprocess (bypassing the factory's mode/CEO infrastructure)
2. Returns a binary 0.0 or 1.0 score — either all tests pass or the score is zero
3. Throws away all exhaust — agent outputs, failure reasons, what the builder tried, what errors it hit, how far it got, costs, durations, node traces
4. Doesn't use RELOOP — the FeatureBench workflow has a gate_verify → builder RELOOP (max 3 iterations on test failure), but DirectFeatureBenchEvaluator runs each agent exactly once
5. Treats candidates as raw Workflow objects — not as registered modes that the factory knows how to run

The result: the outer loop makes blind mutations with zero understanding of WHY candidates fail. It's like training a neural network with only the sign of the loss — you know the direction but not the magnitude or structure.

The Solution: InnerLoop as the Evaluation Function

The factory already has factory/inner_loop.py — a model-like wrapper that composes mode execution with structured exhaust collection:

class InnerLoop:
    def step(self, directives=None) -> CycleRecord:
        # 1. Write outer-loop directives (steering)
        # 2. Run `factory ceo --mode <mode> --no-worktree` as subprocess
        # 3. CycleAnalyzer reads ALL execution artifacts
        # 4. Evaluator parses eval-specific artifacts
        # 5. Return CycleRecord with full exhaust

    def history(self) -> list[CycleRecord]
    def score_trajectory() -> list[float]
    def mutable_nodes() -> set[str]
    def is_mutable(node_id) -> bool

The CycleRecord contains everything the outer loop needs:

@dataclass
class CycleRecord:
    score_start, score_end, score_delta     # Scores
    experiments: list[ExperimentRecord]       # What was tried
    steps: list[AgentStep]                   # Per-agent: role, duration, cost, succeeded, error, produced
    node_trace: dict[str, NodeTrace]         # Which DAG nodes fired, what artifacts they created
    eval_artifacts: list[str]                # Paths to eval output files
    kept, reverted, errored                  # Verdict counts
    total_cost_usd, cost_by_agent            # Cost breakdown
    consecutive_reverts, plateau_detected    # Stall signals
    frozen_nodes, mutable_node_ids           # What the outer loop can/cannot modify

This exhaust IS the gradient equivalent. It tells the outer loop not just "this candidate scored 0" but "the builder failed because of ImportError on line 42 of foo.py after 2 RELOOP retries, costing $0.85 over 340 seconds, with the researcher successfully identifying the right files but the builder hallucinating the interface."

Orchestration: The Outer Loop IS a Mode

The outer loop is itself a registered factory mode, just like design, improve, featurebench, or create. You run it with:

factory ceo /path --mode outer-loop --benchmark featurebench

The CEO reads skills/workflow-outer-loop/SKILL.md, follows the workflow graph, and orchestrates the evolutionary search using specialist agents and CLI tools — exactly the same pattern as every other mode.

Outer Loop Workflow Graph

seed ──▶ evaluate ──▶ reflect ──▶ evolve ──▶ gate_converge ─┐
              ▲                                              │
              └──────────── RELOOP ──────────────────────────┘
                                                  │
                                          PROCEED │
                                                  ▼
                                             finalize

5 nodes + 1 gate. Deliberately simple — the complexity lives inside the evaluate and reflect nodes, not in the graph structure.

Node	Type	What it does
seed	FnNode	SwarmEngine.seed() — creates initial population of mode variants, registers them as ephemeral modes
evaluate	FnNode	Runs InnerLoop.step() for each individual in the population. Each step spawns a sub-CEO that runs the candidate mode (e.g., featurebench variant). Returns CycleRecords with full exhaust.
reflect	AgentNode	Reflection agent (factory agent reflector) reads all CycleRecords, analyzes failure/success patterns, produces ReflectionReport with mutation suggestions
evolve	FnNode	Tournament selection using rich CycleRecord data (not just score). Applies informed mutations guided by ReflectionReport. Registers next generation as new ephemeral modes. Cleans up dead modes.
gate_converge	GateNode	Checks: budget exhausted? Score plateau? Target reached? RELOOP to evaluate if not converged. PROCEED to finalize if done.
finalize	FnNode	Cleanup all ephemeral modes except winner. Optionally promote winner to permanent mode. Archive results.

The Two CEOs

OUTER LOOP CEO                            INNER LOOP (sub-CEO, one per candidate)
──────────────                             ─────────────────────────────────────
Invoked by:                                Invoked by:
  factory ceo --mode outer-loop              InnerLoop.step() → factory ceo --mode evolve-gen0-{id}

Runs:                                      Runs:
  The evolutionary search loop               The candidate workflow on one benchmark instance

Workflow:                                  Workflow (e.g. featurebench):
  seed → evaluate → reflect                  study → builder → gate_verify
  → evolve → gate → RELOOP                   → RELOOP (max 3x) → auto_merge

Lifetime: hours (full evolution)           Lifetime: minutes (one evaluation)

Specialist agents:                         Specialist agents:
  reflector, evolver                         builder, researcher, health_checker

Produces:                                  Produces:
  Best evolved mode (promoted)               CycleRecord with full exhaust

The outer loop CEO calls InnerLoop.step() inside the evaluate node. InnerLoop.step() spawns the sub-CEO as a subprocess (factory ceo --mode <variant>). The sub-CEO executes the candidate workflow (with its own RELOOPs, agent calls, etc.), produces .factory/ artifacts, and exits. The outer loop CEO's CycleAnalyzer reads those artifacts into a CycleRecord.

Specialist Agents for the Outer Loop CEO

Same pattern as Builder/Researcher/etc. — spawned via factory agent <role>:

Agent	Purpose
reflector	Reads CycleRecords from all candidates in a generation. Analyzes failure patterns (e.g., "ImportError on cross-file refs in 3/4 candidates"), success patterns (e.g., "study phase correlates with passing tests"), and produces specific mutation suggestions.
evolver	Reads ReflectionReport + current population. Runs tournament selection with rich fitness (score + cost + retries + partial progress). Applies mutations informed by reflection — not random. Produces the next generation of workflow variants.

CLI Tools for the Outer Loop CEO

The CEO orchestrates directly via CLI — no Builder delegation:

factory outer-loop calibrate --project . --benchmark featurebench --parallelism 4
factory outer-loop evaluate --project . --generation 0
factory outer-loop reflect --project . --generation 0
factory outer-loop evolve --project . --generation 0
factory outer-loop status --project .
factory outer-loop promote --run-id <id>   # promote winner to permanent mode

Population Architecture

┌─────────────────────────────────────────────────────────────┐
│                    OUTER LOOP (SwarmEngine)                  │
│                                                             │
│  Population of MODE VARIANTS (ephemeral registered modes)   │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐      │
│  │ Mode A   │ │ Mode B   │ │ Mode C   │ │ Mode D   │      │
│  │(builder  │ │(R→B→HC   │ │(study→B  │ │(B with   │      │
│  │ only)    │ │ +RELOOP) │ │ →verify) │ │ better   │      │
│  │          │ │          │ │          │ │ prompt)  │      │
│  └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘      │
│       │             │             │             │            │
│       ▼             ▼             ▼             ▼            │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              INNER LOOP (per candidate)              │    │
│  │                                                      │    │
│  │  InnerLoop.step() runs:                             │    │
│  │    factory ceo --mode <variant> --no-worktree       │    │
│  │                                                      │    │
│  │  The mode itself has LOOPS:                          │    │
│  │    study → builder → gate_verify ──RELOOP──┐        │    │
│  │                         │                   │        │    │
│  │                         ▼                   │        │    │
│  │                    auto_merge          (max 3x)      │    │
│  │                                                      │    │
│  │  Returns: CycleRecord (scores + full exhaust)        │    │
│  └──────────────────────────────────────────────────────┘    │
│       │             │             │             │            │
│       ▼             ▼             ▼             ▼            │
│  ┌─────────────────────────────────────────────────────┐    │
│  │              REFLECTION AGENT                        │    │
│  │                                                      │    │
│  │  Reads ALL CycleRecords from this generation:        │    │
│  │  - Why did Mode A fail? (ImportError, line 42)       │    │
│  │  - Why did Mode B succeed? (researcher found deps)   │    │
│  │  - Pattern: modes with study phase succeed 2x more   │    │
│  │  - Pattern: RELOOP helps on multi-function tasks     │    │
│  │  - Recommendation: add dependency analysis node      │    │
│  │  - Recommendation: improve builder prompt re: imports│    │
│  │                                                      │    │
│  │  Produces: ReflectionReport (failure_patterns,        │    │
│  │            success_patterns, mutation_suggestions)    │    │
│  └──────────────────────────────────────────────────────┘    │
│       │                                                      │
│       ▼                                                      │
│  ┌─────────────────────────────────────────────────────┐    │
│  │         EVOLUTIONARY SEARCH (informed by reflection) │    │
│  │                                                      │    │
│  │  Tournament selection (using rich CycleRecord, not   │    │
│  │  just binary score — consider cost, retries, partial │    │
│  │  progress, error diversity)                          │    │
│  │                                                      │    │
│  │  Informed mutations:                                 │    │
│  │  - PROMPT_MUTATE: guided by failure analysis         │    │
│  │  - NODE_INSERT: address observed gaps                │    │
│  │  - PARALLELIZE: based on dependency analysis         │    │
│  │  - LLM crossover: synthesize from winning patterns   │    │
│  │                                                      │    │
│  │  Produces: next generation of mode variants           │    │
│  └──────────────────────────────────────────────────────┘    │
│                                                             │
│  REPEAT until convergence / budget exhausted                │
└─────────────────────────────────────────────────────────────┘

What Exists Today

Component	Status	Location
InnerLoop	EXISTS — needs FeatureBench evaluator	factory/inner_loop.py
CycleAnalyzer	EXISTS — reads .factory/ exhaust	factory/cycle_analyzer.py
CycleRecord / AgentStep / ExperimentRecord	EXISTS	factory/cycle_analyzer.py
SwarmEngine	EXISTS — needs to use InnerLoop instead of DirectEvaluator	factory/outer_loop/engine.py
Mutation operators (7 types)	EXISTS	factory/outer_loop/mutations.py
Population + MAPElitesArchive	EXISTS	factory/outer_loop/population.py
FeatureBench workflow with RELOOP	EXISTS	factory/workflow/contributed/featurebench/workflow.py
CompressOuterLoop → CompressInnerLoop	EXISTS — reference implementation	factory/compress/
skillopt/reflect.py (minibatch reflection)	EXISTS — reference for reflection	factory/skillopt/reflect.py
Checkpoint + Progress tracking	EXISTS	factory/outer_loop/checkpoint.py, progress.py
CLI entry points	EXISTS	factory/cli/outer_loop.py
FeatureBench evaluator for InnerLoop	MISSING	needs FeatureBenchEvaluator(Evaluator)
Reflection agent for outer loop	MISSING	needs OuterLoopReflector
Mode-as-individual registration	MISSING	SwarmEngine creates Workflow → needs to register as mode
Partial credit scoring	MISSING	needs per-test pass/fail, not binary
Informed mutation (reflection → mutation)	MISSING	mutations are currently random
CEO orchestration of outer loop	MISSING	CEO should run this, not Builder
Outer loop as a registered mode	MISSING	needs workflow graph + SKILL.md + trigger
Create mode (reference for mode generation)	EXISTS	factory/workflow/contributed/ + skills/workflow-create/
reflector agent role	MISSING	needs prompt + registration in agent runner
evolver agent role	MISSING	needs prompt + registration in agent runner

Implementation Plan

Phase 1: Wire InnerLoop into SwarmEngine

What: Replace DirectFeatureBenchEvaluator with InnerLoop.step() as the fitness function.

Details:

1. Create FeatureBenchEvaluator(Evaluator) that implements the parse() protocol for FeatureBench:

   • Reads Harbor/Docker verification artifacts (pytest output, pass/fail per test)
   • Returns EvalResult with partial credit: fraction of tests passing, not binary
   • Parses agent output artifacts for failure reasons

2. Create FeatureBenchInnerLoop(InnerLoop) (following CompressInnerLoop pattern):

   • __init__: takes project_dir, evaluator, workflow (the candidate mode variant), frozen_nodes
   • The mode parameter maps to the candidate's registered mode name
   • Inherits step(), collect(), history(), score_trajectory() from InnerLoop
   • Adds FeatureBench-specific tracking: per-instance results, failure categorization

3. Modify SwarmEvaluator.evaluate() to use FeatureBenchInnerLoop.step():

   • For each candidate workflow: create a FeatureBenchInnerLoop with that workflow
   • Call loop.step() → get CycleRecord with full exhaust
   • Extract score from CycleRecord.score_end
   • Store the full CycleRecord on the Individual for reflection

4. The InnerLoop runs factory ceo --mode <variant> which:

   • Executes the workflow graph (study → builder → gate_verify → RELOOP → auto_merge)
   • The RELOOP gives the builder up to 3 retries on test failure
   • Produces all .factory/ artifacts (events.jsonl, reviews/, experiments/)
   • CycleAnalyzer reads these into the CycleRecord

Files to modify:

• factory/outer_loop/evaluator.py — swap EvaluatorFn to use InnerLoop
• New: factory/outer_loop/featurebench_evaluator.py — FeatureBenchEvaluator(Evaluator)
• New: factory/outer_loop/featurebench_inner_loop.py — FeatureBenchInnerLoop(InnerLoop)

Phase 2: Candidates as Ephemeral Modes (with Lifecycle Management)

What: Each individual in the population is a registered workflow mode, not just a Workflow object. Modes are ephemeral — most don't survive selection — so the registry needs creation, cleanup, and optional promotion.

Details:

1. When SwarmEngine.seed() creates the initial population:

   • For each Workflow variant, register it as an ephemeral mode via EphemeralModeRegistry
   • Naming convention: evolve-gen{N}-{individual_id[:8]} (e.g., evolve-gen0-cfc66ffc)
   • Write the workflow definition as JSON to .factory/outer_loop/modes/evolve-gen0-cfc66ffc.json
   • The InnerLoop resolves the mode name → loads the workflow JSON → runs it
   • CRITICAL: These NEVER touch the global workflow registry (register_all() in definitions.py). Ephemeral modes live entirely in .factory/outer_loop/modes/ and are resolved at runtime by the InnerLoop. factory workflow list will not show dead experiment modes.

2. When mutations create new variants:

   • Register the mutated workflow as a new ephemeral mode: evolve-gen1-b208ca66
   • The old generation's non-surviving modes are cleaned up after selection

3. The InnerLoop for each candidate uses its registered mode name:

   loop = FeatureBenchInnerLoop(
       project_dir=testbed_path,
       mode=f"evolve-gen{gen}-{individual.id[:8]}",
       workflow=individual.workflow,
   )
   record = loop.step()

4. Each mode variant gets its own .factory/ directory (or subdirectory) so CycleAnalyzer can read its specific exhaust without cross-contamination.

5. Lifecycle management:

   class EphemeralModeRegistry:
       def __init__(self, run_dir: Path):
           self.modes_dir = run_dir / "modes"
           self.modes_dir.mkdir(parents=True, exist_ok=True)
           self._registered: dict[str, Path] = {}
   
       def register(self, individual_id: str, generation: int, workflow: Workflow) -> str:
           """Register a workflow as an ephemeral mode. Returns mode name."""
           mode_name = f"evolve-gen{generation}-{individual_id[:8]}"
           path = self.modes_dir / f"{mode_name}.json"
           path.write_text(json.dumps(workflow.to_dict(), indent=2))
           self._registered[mode_name] = path
           return mode_name
   
       def cleanup_generation(self, survivors: set[str]):
           """After selection: delete modes not in the survivor set."""
           for name, path in list(self._registered.items()):
               if name not in survivors:
                   path.unlink(missing_ok=True)
                   del self._registered[name]
   
       def cleanup_all(self, keep_best: str | None = None):
           """At convergence: delete all except the winner."""
           for name, path in list(self._registered.items()):
               if name != keep_best:
                   path.unlink(missing_ok=True)
           self._registered = {k: v for k, v in self._registered.items() if k == keep_best}
   
       def promote(self, mode_name: str, permanent_name: str):
           """Copy winning mode to factory/workflow/contributed/ as a permanent mode."""
           ...

6. Capacity invariant: At most population_size modes exist at any time per generation. After each generation's tournament, non-survivors are deleted immediately. After the run completes (convergence or budget), everything is cleaned up except the best individual's mode, which can optionally be promoted to a permanent contributed workflow via factory outer-loop promote <run_id>.

Files to modify:

• factory/outer_loop/engine.py — register/cleanup modes for each individual via EphemeralModeRegistry
• New: factory/outer_loop/mode_registry.py — EphemeralModeRegistry class

Phase 3: Reflection Agent

What: Between generations, a reflection agent analyzes all CycleRecords and produces actionable insights for mutation.

Details:

1. OuterLoopReflector (following skillopt/reflect.py pattern):

   • Input: list of (Individual, CycleRecord) pairs from this generation
   • Reads each CycleRecord's exhaust:
     • AgentStep — which agents succeeded/failed, what errors they hit
     • ExperimentRecord — what was tried, what worked
     • NodeTrace — which DAG nodes fired and what they produced
     • eval_artifacts — detailed test results (which tests passed/failed)
   • Calls an LLM (via claude -p -) with a structured prompt:
     • "Here are 4 workflow variants and their detailed execution traces on 7 FeatureBench instances..."
     • "Analyze failure patterns, success patterns, and recommend specific mutations"
   • Output: ReflectionReport:

     @dataclass
     class ReflectionReport:
         failure_patterns: list[FailurePattern]   # "ImportError on cross-file refs" 
         success_patterns: list[SuccessPattern]    # "study phase finds right files"
         mutation_suggestions: list[MutationSuggestion]  # "add dep-analysis node before builder"
         prompt_improvements: list[str]            # specific prompt text changes
         structural_recommendations: list[str]     # "add RELOOP from verify to builder"

2. The reflection output feeds directly into mutation selection:

   • Instead of random operator choice, weight operators by reflection recommendations
   • PROMPT_MUTATE uses reflection's prompt_improvements as the mutation source
   • NODE_INSERT uses reflection's structural_recommendations to choose what node to add
   • LLM crossover (via crossover_fn) uses reflection to synthesize prompts from winners

Files:

• New: factory/outer_loop/reflector.py — OuterLoopReflector
• New: factory/outer_loop/prompts/reflect.md — reflection prompt template
• Modify: factory/outer_loop/engine.py — call reflector between generations
• Modify: factory/outer_loop/mutations.py — accept reflection output to guide mutations

Phase 4: Partial Credit Scoring

What: Replace binary 0/1 scoring with fraction of tests passing.

Details:

The FeatureBench verification runs pytest, which outputs per-test pass/fail. Parse this output:

• 5 passed, 3 failed → score = 5/8 = 0.625
• This gives evolution a gradient: a candidate that passes 5/8 tests is better than one that passes 2/8
• The DirectFeatureBenchEvaluator's _verify_in_docker() already runs pytest — parse its output for individual test results
• Store per-test results on the Individual for reflection analysis

Files:

• Modify: factory/outer_loop/direct_evaluator.py — parse pytest output for per-test results
• New: factory/outer_loop/featurebench_evaluator.py — FeatureBenchEvaluator.parse() handles this

Phase 5: Outer Loop as a Registered Mode

What: The outer loop IS a factory mode — registered workflow, CEO runs it, has its own SKILL.md. This is the scientific evolution of create mode.

Details:

1. Register the outer loop workflow at factory/workflow/contributed/outer_loop/:

   • Workflow graph: seed → evaluate → reflect → evolve → gate_converge → RELOOP/finalize
   • Trigger: ctx.get("mode") == "outer-loop"
   • Terminal mode (doesn't chain to other modes)
   • Accepts --benchmark <name> to select which inner loop mode to optimize

2. Export SKILL.md via factory workflow export-skills → skills/workflow-outer-loop/SKILL.md

   • The CEO reads this skill and follows the workflow graph
   • Each node maps to a CLI command or agent invocation (same as all other modes)

3. Register specialist agents:

   • reflector agent: prompt at factory/agents/prompts/reflector.md
   • evolver agent: prompt at factory/agents/prompts/evolver.md
   • Both spawned via factory agent <role> (same as builder/researcher/etc.)

4. Invocation:

   factory ceo /path --mode outer-loop --benchmark featurebench
   # or for long runs:
   factory tmux /path --mode outer-loop --benchmark featurebench

5. The CEO orchestrates the evolutionary loop by following the SKILL.md:

   • Calls factory outer-loop calibrate for seed evaluation
   • Calls factory agent reflector to analyze exhaust
   • Calls factory agent evolver to produce next generation
   • Checks gate_converge (budget/plateau/target)
   • RELOOPs or finalizes

Files:

• New: factory/workflow/contributed/outer_loop/workflow.py — workflow graph definition
• New: factory/agents/prompts/reflector.md — reflection agent prompt
• New: factory/agents/prompts/evolver.md — evolution agent prompt
• Modify: factory/cli/outer_loop.py — add evaluate, reflect, evolve, status, promote subcommands
• Auto-generated: skills/workflow-outer-loop/SKILL.md via factory workflow export-skills

Phase 6: End-to-End Validation on FeatureBench

What: Actually run the outer loop on 2 FeatureBench instances and verify everything works.

Details:

This is not a unit test. This is running the entire pipeline:

1. Pick 2 FeatureBench instances: 1 lv1 (should pass with builder-only) + 1 lv2 (should fail with builder-only)
2. Create builder-only seed (1 node, no study, no RELOOP)
3. Run 1 generation of evolution with population=3:
   • Seed (builder-only)
   • Mutation: NODE_INSERT (add study node)
   • Mutation: add RELOOP (gate_verify → builder, max 2)
4. Verify:
   • InnerLoop.step() produces CycleRecord with populated AgentSteps and NodeTraces
   • CycleRecord.steps has per-agent success/failure with actual error messages
   • Partial credit scoring works (pytest output parsed)
   • Reflection agent produces a ReflectionReport with non-empty patterns
   • Mutations in Gen 1 are informed by reflection (not random)
   • The mode with RELOOP scores higher than the mode without it on lv2

This validation MUST run as part of the implementation, not after. The Builder implements Phase 1, then runs the validation to verify it works. Fixes what breaks. Then Phase 2, validate again. And so on. This is the design mode iteration loop: implement → run → observe → fix → repeat.

Important: Each instance evaluation takes 3-10 minutes (spawns Claude agents). Budget accordingly. Use --timeout 600 for agents. Run in tmux for long sessions.

Testing Strategy

Unit tests verify component contracts. The real test is the end-to-end validation (Phase 6).

Unit tests:

• FeatureBenchEvaluator.parse() correctly extracts partial credit from pytest output
• FeatureBenchInnerLoop.step() returns a CycleRecord with populated fields (mock the subprocess)
• OuterLoopReflector produces non-empty ReflectionReport from sample CycleRecords
• Mode registration and cleanup works (dynamic modes don't leak)

Integration test (Phase 6 — must actually run):

• 2 FeatureBench instances, 1 generation, population 3
• Verify exhaust is captured, reflection works, informed mutations produce different candidates than random mutations
• This takes ~30-60 minutes of wall clock (6-9 agent evaluations × 5-10 min each)
• Run in tmux, monitor progress.jsonl

Evolutionary Search Methods to Consider

The current implementation uses tournament selection with random mutations. Consider:

• OpenELM MAP-Elites: quality-diversity optimization — maintains an archive of diverse high-performing workflows, not just the single best
• EvoPrompt GA/DE: genetic algorithm and differential evolution for prompt optimization — proven on BBH benchmarks (25% improvement)
• Reflection-guided search: the reflection agent essentially provides a search direction, making this closer to gradient descent than random search
• LLM-as-crossover: the crossover_fn parameter already exists in mutations.py — wire it to use the reflection agent's insights

The choice of method should depend on the quality of the reflection signal. If reflection reliably identifies what to fix, use it as the primary search direction. If it's noisy, use it as a prior for random search (bias mutations toward reflection suggestions but don't eliminate randomness).

Files Summary

File	Action	Purpose
factory/workflow/contributed/outer_loop/workflow.py	NEW	Outer loop workflow graph (seed→evaluate→reflect→evolve→gate→finalize)
factory/outer_loop/featurebench_evaluator.py	NEW	Evaluator protocol impl for FeatureBench (partial credit)
factory/outer_loop/featurebench_inner_loop.py	NEW	InnerLoop subclass for FeatureBench
factory/outer_loop/reflector.py	NEW	Reflection agent — analyzes exhaust, produces MutationSuggestions
factory/outer_loop/mode_registry.py	NEW	EphemeralModeRegistry — register/cleanup/promote candidate modes
factory/agents/prompts/reflector.md	NEW	Reflector agent prompt
factory/agents/prompts/evolver.md	NEW	Evolver agent prompt
factory/outer_loop/prompts/reflect.md	NEW	LLM prompt template for reflection analysis
factory/outer_loop/engine.py	MODIFY	Use InnerLoop, call reflector, informed mutations, ephemeral modes
factory/outer_loop/evaluator.py	MODIFY	Swap to InnerLoop-based evaluation
factory/outer_loop/mutations.py	MODIFY	Accept ReflectionReport to guide mutations
factory/outer_loop/direct_evaluator.py	MODIFY	Parse pytest output for partial credit
factory/cli/outer_loop.py	MODIFY	Add evaluate/reflect/evolve/status/promote subcommands
skills/workflow-outer-loop/SKILL.md	AUTO-GEN	CEO playbook for outer loop mode (via export-skills)

Prior Art in This Codebase

• CompressOuterLoop → CompressInnerLoop → InnerLoop: exact pattern to follow
• skillopt/reflect.py: reflection that reads traces and produces structured patches
• skillopt/trainer.py: DL-style training loop with epochs, steps, eval splits
• CycleAnalyzer: reads .factory/ artifacts into structured records
• FeatureBench workflow: the inner loop template with RELOOP

Success Criteria

1. Running factory outer-loop evolve --project . --instances <2 instances> produces:

   • CycleRecords with populated AgentSteps (not empty)
   • A ReflectionReport with non-trivial failure/success patterns
   • Gen 1 mutations that differ from Gen 0 mutations (informed, not random)
   • A mode variant that scores higher than the builder-only seed on a mixed lv1+lv2 set

2. The score improvement is observable in the partial-credit metric (e.g., 3/8 tests → 6/8 tests), not just binary pass/fail.

3. The entire pipeline runs without CEO intervention (except monitoring) once started.