Minimum detectable effect and power tooling

[Colinho22]

Motivation

Repetition counts are chosen against a cost budget, with no statement of what difference the resulting design can actually resolve. This makes a no-difference result hard to interpret — it could mean no effect exists, or that the design could never have seen one. The variance estimates needed to settle this already exist in collected run data.

Proposal

• Compute the MDE for the realised design from observed within-cell variance.
• Add a standalone script: given target effect size and power, return required repetitions.
• Report MDE next to comparative results, so a no-difference finding reads as "no effect larger than X was detectable".

Done when:

• MDE computed from actual variance, per metric
• Power script runs standalone with documented inputs
• MDE appears in the generated results summary

Scope

• In scope: MDE computation from existing variance, power script, integration into results summary.
• Out of scope: re-running the experiment at a higher repetition count; simulation-based power analysis.

Open questions

• Which variance component drives the MDE — within-cell across repetitions, or across tasks? The mixed-effects structure means these give different answers, and the task-level one is probably the honest number.
• Should the MDE be computed on the aggregated data or under the mixed-effects model? Should match whatever the primary analysis uses.
• Does the analytic approximation hold for a bounded metric like F1 near its ceiling, or is a simulation needed after all?

Related

• Add equivalence testing (TOST) to the analysis pipeline #92 — MDE and SESOI are the same question approached from opposite ends; a SESOI below the MDE means the design cannot answer it.
• Ceiling diagnostics and a high-complexity tier #95 — saturation directly affects variance and therefore the MDE.