Retry, fallback, and validation layer

[Colinho22]

Motivation

Any production use of the pipeline needs an answer to invalid generations, and the harness currently offers none. Adding recovery as an optional, instrumented layer serves real deployments and turns "recovery helps" into something measurable rather than assumed.

Proposal

• Retry with backoff on parse and schema failures, configurable cap.
• Fallback chain: on repeated failure, drop to a simpler strategy or a different model.
• Post-generation validation gate checking output against the target notation schema before scoring.
• Record retries and fallbacks separately so results are reportable with and without recovery — the layer must not silently inflate reliability figures.

Done when:

• Retry, fallback, and validation each toggleable, all off by default
• Telemetry distinguishes first-attempt success from recovered success
• With the layer disabled, results match current numbers exactly

Scope

• In scope: retry, fallback chain, validation gate, separate accounting for recovered runs.
• Out of scope: automatic repair of malformed output — recovery here means re-generation, not patching. Deciding the default configuration for published results.

Open questions

• Should a retry reuse the identical prompt, or append the parse error as feedback? The latter is more effective but blurs the line into free-flow reflection (Proof of Concept #1).
• Does a recovered run count as success in the headline reliability metric? Argues for reporting both, but one has to be the default.
• Fallback ordering: cheaper model first, or simpler strategy first? Different cost profiles, and it changes what the cost-frontier analysis sees.

Related

• Depends on Failure-mode analysis for invalid generations #94 — recovery should target the causes identified there.
• Depends on Introduce orchestration modes: constrained pipeline and free-flow agentic #91 — retry-with-feedback overlaps the reflection loop; avoid two implementations.