feat: failure-mode analysis for invalid generations

docs/analysis.md

@@ -50,6 +50,8 @@ output/analysis/20260621T111935Z/
     effect_sizes.json
     error_taxonomy_by_strategy.json
     tradeoff_correctness_efficiency.json
+    failure_rates.json
+    survivor_bias.json
     figures/README.md
 ```
 
@@ -73,6 +75,10 @@ Every file has a top-level `status` string:
   A `reason` string explains which factor was underpopulated. Re-runs
   will populate the analysis automatically once the corpus grows; no
   code change is needed.
+- `"empty"`: there were no experimental rows to analyse at all, so the
+  analysis had nothing to run on rather than a factor too sparse to fit.
+  The distinction from `"skipped"` is deliberate: `"empty"` means no data,
+  `"skipped"` means data that will not support this particular test.
 
 ### 3.1 `descriptive.json`
 
@@ -140,6 +146,131 @@ Per-strategy medians on both dimensions: `entity_id_f1` (correctness)
 and `cost_usd` / `duration_ms` (efficiency). Consumed by the dashboard's
 Pareto view.
 
+### 3.9 `failure_rates.json`
+
+Failure rate and cause breakdown per strategy, model, and tier, plus an
+`overall` block. This is the reliability counterpart to the accuracy
+tables: `error_taxonomy_by_strategy.json` scores the content of diagrams
+that were produced, while this file covers the runs that produced nothing
+scorable at all. The two populations are disjoint.
+
+Rates are **pooled counts**: the denominator is every run attempted in the
+group, not a mean of per-cell rates. A per-cell mean would weight a cell
+with one run as heavily as a cell with five, which for a rate is the wrong
+grain. This deliberately differs from the F1 path, which aggregates per
+cell because it averages a score rather than counting events.
+
+Every cause appears in every `causes` block, including the zeros: an
+absent key would be ambiguous between "never happened" and "not measured".
+Controls are excluded (they never call a model).
+
+Payload shape (abridged):
+
+```json
+{
+  "status": "ok",
+  "statistic": "pooled_failure_rate",
+  "overall": {
+    "n_runs": 6000,
+    "n_failed": 478,
+    "failure_rate": 0.0797,
+    "causes": { "schema_violation": 261, "truncation": 135, "...": 0 }
+  },
+  "by_strategy": [
+    {
+      "strategy": "sop_based",
+      "n_runs": 1500,
+      "n_failed": 151,
+      "failure_rate": 0.1007,
+      "causes": { "schema_violation": 85, "...": 0 }
+    }
+  ]
+}
+```
+
+### 3.10 `survivor_bias.json`
+
+Per strategy, the primary DV under `valid_only` (survivors only) against
+`intent_to_treat` (every run, failures scored 0.0). The gap between them
+is the survivor bias: a strategy that fails often looks better under
+`valid_only` because its failures were dropped rather than scored, and
+`survivor_bias` measures exactly how much that dropping flatters it.
+
+Reported per strategy rather than as one pooled figure, because the bias
+scales with each strategy's failure rate: a single number would hide the
+comparison that matters. `n_cells_dropped` counts cells where every run
+failed, so `valid_only` had nothing to average.
+
+---
+
+## 3a. Failure taxonomy
+
+`failure_rates.json` classifies each invalid run into exactly one primary
+cause. The classifier lives in `src/maestro/analysis/failures.py`.
+
+### What counts as a failure
+
+Two disjoint shapes, both rejected by `RunResult.success`:
+
+- an **errored** run: `error` is set, and the string is the evidence.
+- a **silent-empty** run: `error` is `None` but the diagram is missing or
+  blank. A provider can return whitespace without raising, so this is a
+  real category, not a data defect.
+
+### Where the evidence lives
+
+On a failed run, `run_results.raw_response` is always `NULL`: the error
+result is built before any text exists. The failing text is retained one
+level down, on the `sub_results` row that failed. Classification therefore
+reads the run's `error` plus the first failed sub-result's `raw_response`.
+
+This means historical runs can be classified retroactively with no model
+re-invocation, which is why re-scoring was never needed for this analysis.
+
+### The causes
+
+| Cause | Meaning |
+|---|---|
+| `rate_limit` | Provider returned a rate-limit error. |
+| `timeout` | Request exceeded the client deadline. |
+| `api_error` | Generic provider API error (the SDKs' catch-all base class). |
+| `safety_block` | Response blocked by a content or safety filter. |
+| `empty_output` | Provider returned no text, or only whitespace. |
+| `truncation` | Response stopped mid-structure, consistent with a token limit. |
+| `parse_error` | Output was not parseable in the requested format. |
+| `schema_violation` | Output parsed but broke the Mermaid output contract. |
+| `orchestration_error` | The framework misbehaved, not the model output. |
+| `unknown` | No rule matched. Visible by design rather than mis-filed. |
+
+### Classification rules
+
+Causes are not mutually exclusive in practice: a truncated response is
+usually *also* a parse error, because the truncation is what broke the
+parse. Rather than multi-label (which makes rates hard to sum and
+compare), each failure gets one primary cause under a fixed precedence,
+most specific first:
+
+1. **Infrastructure** (`rate_limit`, `timeout`, `safety_block`). If the
+   API never returned, nothing downstream is meaningful.
+2. **`empty_output`**. A missing response cannot be a parse error.
+3. **`schema_violation`**, before the generic parse rule: well-formed text
+   that violates the output contract is a different failure from text that
+   is not parseable at all.
+4. **`parse_error`**, promoted to **`truncation`** when the parser message
+   is truncation-shaped (an unterminated string, or a structure that simply
+   stops) *and* the raw response is long enough that running out of tokens
+   is plausible. Without the raw text the two are indistinguishable, so the
+   conservative `parse_error` label stands: under-reporting truncation is
+   safer than inventing it.
+5. **`orchestration_error`**.
+6. **`api_error`** last among the infrastructure family, since a more
+   specific subclass above must win over the catch-all base class.
+7. **`unknown`** as the explicit fallback.
+
+Adding a provider means checking whether its error prefixes are covered.
+An unmatched prefix surfaces as `unknown` rather than being mis-filed,
+which is the failure mode this ordering exists to prevent.
+
 ---
 
 ## 4. `report.md`
@@ -216,8 +347,10 @@ NULL regardless of operating system.
 The corpus can under-populate a factor (only one input tier, only one
 model, only two strategies). Analyses that need at least two levels of
 that factor return `status="skipped"` with a `reason` string that names
-the underpopulated factor. Downstream code should check the status
-before reading terms:
+the underpopulated factor. An analysis with no experimental rows at all
+returns `status="empty"` instead, and carries no `reason`: nothing was
+too sparse to fit, there was simply nothing to fit. Downstream code
+should check the status before reading terms:
 
 ```python
 import json
@@ -226,7 +359,8 @@ payload = json.loads(open("output/analysis/.../anova_strategy_by_tier.json").rea
 if payload["status"] == "ok":
     interaction_p = payload["terms"]["strategy:tier"]["p"]
 else:
-    print(f"skipped: {payload['reason']}")
+    # "empty" carries no reason, so do not index it unconditionally.
+    print(f"{payload['status']}: {payload.get('reason', 'no experimental rows')}")
 ```
 
 Every consumer (the report builder, the dashboard) uses this pattern; a

src/maestro/analysis/__init__.py

@@ -3,6 +3,7 @@
 # Statistical analysis pipeline. Re-exported so callers can do
 # ``from maestro.analysis import describe, anova_strategy`` without reaching
 # into the submodule. The CLI lives in maestro.analysis.__main__.
+from maestro.analysis.failures import FailureCause, classify_failure
 from maestro.analysis.statistics import (
     DEFAULT_CONVENTION,
     INTENT_TO_TREAT,
@@ -16,9 +17,12 @@
     describe,
     effect_sizes,
     error_taxonomy_by_strategy,
+    failure_rates,
     load_dataframe,
+    load_failure_dataframe,
     mixed_effects_robustness,
     posthoc_strategy,
+    survivor_bias,
     tradeoff_correctness_efficiency,
 )
 
@@ -51,8 +55,14 @@
     "describe",
     "effect_sizes",
     "error_taxonomy_by_strategy",
+    "failure_rates",
     "load_dataframe",
+    "load_failure_dataframe",
     "mixed_effects_robustness",
     "posthoc_strategy",
+    "survivor_bias",
     "tradeoff_correctness_efficiency",
+    # failure classification
+    "FailureCause",
+    "classify_failure",
 ]

src/maestro/analysis/__main__.py

@@ -32,9 +32,12 @@
     describe,
     effect_sizes,
     error_taxonomy_by_strategy,
+    failure_rates,
     load_dataframe,
+    load_failure_dataframe,
     mixed_effects_robustness,
     posthoc_strategy,
+    survivor_bias,
     tradeoff_correctness_efficiency,
 )
 from maestro.analysis.timestamps import format_for_display
@@ -59,8 +62,14 @@
     ("descriptive.json", describe),
     ("error_taxonomy_by_strategy.json", error_taxonomy_by_strategy),
     ("tradeoff_correctness_efficiency.json", tradeoff_correctness_efficiency),
+    ("survivor_bias.json", survivor_bias),
 ]
 
+# failure_rates is wired separately rather than added to _ANALYSES: it needs
+# the failure frame as well as the run frame, so it does not share the
+# single-DataFrame signature every entry above has.
+_FAILURE_RATES_FILE = "failure_rates.json"
+
 # Convention-dependent analyses: (stem, callable). Each is emitted once per
 # scoring convention as ``<stem>__<convention>.json`` (content-based naming:
 # the filename states both the test and the convention, so a file is never
@@ -105,6 +114,12 @@
         "tradeoff_correctness_efficiency.json + effect_sizes__intent_to_treat.json",
         "Correctness vs. efficiency trade-off across strategies.",
     ),
+    (
+        "reliability",
+        "failure_rates.json + survivor_bias.json",
+        "Cross-cutting: how often does a strategy produce nothing usable, "
+        "from what cause, and how much does the valid-only view flatter it?",
+    ),
     (
         "robustness",
         "anova_strategy_by_model__intent_to_treat.json + "
@@ -393,6 +408,7 @@ def main(argv: list[str] | None = None) -> int:
     # enforces that at the boundary instead of relying on a no-op commit.
     with get_readonly_connection(args.db) as conn:
         df = load_dataframe(conn)
+        failures = load_failure_dataframe(conn)
 
     if df.empty:
         print(
@@ -407,6 +423,10 @@ def main(argv: list[str] | None = None) -> int:
         results[filename] = payload
         _write_json(run_dir / filename, payload)
 
+    failure_payload = failure_rates(df, failures)
+    results[_FAILURE_RATES_FILE] = failure_payload
+    _write_json(run_dir / _FAILURE_RATES_FILE, failure_payload)
+
     # Convention-dependent analyses: one file per (analysis, convention).
     for stem, fn in _CONVENTION_ANALYSES:
         for convention in _CONVENTIONS: