Introduce orchestration modes: constrained pipeline and free-flow agentic

[Colinho22]

Motivation

Every strategy in the harness shares one control flow: a predetermined sequence of agent calls, no revision step, no exchange of intermediate state between agents. That uniformity is a feature — token spend, latency, and cost are bounded and predictable before a run starts, and it is what makes strategies comparable to one another at all.

It is also a single point on a design axis. The other end — agents that critique, revise, and pass intermediate state until a stopping condition is met — trades predictability for the chance of self-correction. The harness cannot currently express that end, so it has never been measured against the constrained one.

Both belong in the tool. Callers who need a cost ceiling keep today's guarantees; callers who can spend more get an adaptive mode; and the trade-off between them becomes something the harness can run as an experiment rather than an assumption.

Proposal

Refactor the strategy layer into a shared orchestration contract that both modes implement.

• Define an Orchestrator interface exposing identical telemetry, cost accounting, and result schema regardless of mode.
• Extract existing strategies unchanged behind mode: constrained. Pure refactor — existing runs must reproduce bit-for-bit.
• Add a BudgetController both modes honour: caps on total token spend, wall-clock time, and agent turns. Effectively never hit in constrained mode; the safety rail in free-flow.
• Add mode: free_flow with configurable components:
  • reflection loop — generate, critique against source, revise, max-rounds cap
  • inter-agent feedback — entity and relation agents exchange intermediate output before merge, rather than merging blind
  • specialist roles — distinct prompts, optionally distinct models, per role
  • stopping condition — critic satisfied, no change between rounds, or budget exhausted
• Record which stopping condition fired plus the per-round trajectory, so cost and quality are traceable round by round rather than only at the end.

Done when:

• Both modes selectable by config, no code change to switch
• Constrained-mode runs reproduce existing results exactly post-refactor
• Budget caps enforced in both modes and present in telemetry
• Free-flow emits per-round cost, latency, and intermediate quality
• Stopping-condition distribution reportable across a run set
• README documents the trade-off and when each mode applies

Scope

• In scope: the Orchestrator interface, budget enforcement, lift-and-shift of existing strategies, the three free-flow components, per-round telemetry.
• Out of scope: tuning free-flow to beat constrained mode; adding new models or notations; changing any metric definition.

Open questions

• Does the critic in the reflection loop see the source text, the generated diagram, or both? Seeing both is stronger but risks the critic simply restating the generator.
• Should the critic share a model with the generator, or is an independent model required to avoid correlated blind spots?
• On budget exhaustion mid-round: return the last complete round's output, or mark the run invalid? Affects how free-flow enters the reliability metric.
• Is mode a third factor in the experimental design, or a property of each strategy? The former multiplies run cost; the latter keeps the design flat.

Related