Failure-mode analysis for invalid generations

[Colinho22]

Motivation

Runs producing no valid output are counted, then set aside. Separating reliability from accuracy is a core design idea of the harness, and it only pays off if the reliability side is analysed as carefully as the accuracy side. At present the entire failure surface reduces to one number, and the failing outputs themselves are not retained for inspection.

Proposal

• Classify every invalid output by cause: parse or syntax error, schema violation, truncation or token limit, refusal, timeout, empty output, orchestration-level error.
• Persist the raw failing output alongside its classification so failures remain inspectable after the run.
• Break failure rate down by strategy × complexity × model, mirroring the accuracy tables.
• Test whether the surviving subset differs systematically from the full set — quantifying survivor bias rather than noting it.

Done when:

• Every invalid run carries a machine-readable failure cause
• Failure taxonomy documented with its classification rules
• Failure-rate breakdown generated as table and figure
• Survivor-bias check reported as a number

Scope

• In scope: classification logic, persistence of failing output, breakdown reporting, survivor-bias test.
• Out of scope: fixing or recovering from failures — that is Update openai requirement from >=1.0.0 to >=2.32.0 #6. Retroactive classification of runs whose raw output was not retained.

Open questions

• How much historical data can be classified retroactively? If failing outputs were discarded rather than stored, part of this needs a fresh run.
• Categories are not always mutually exclusive — a truncated output is usually also a parse error. Single primary cause with a precedence order, or multi-label?
• What is the right survivor-bias test given the task difficulty distribution is known? Comparing difficulty of surviving versus failing tasks may be more direct than comparing scores.

Related