Add equivalence testing (TOST) to the analysis pipeline

[Colinho22]

Motivation

The pipeline reports null-hypothesis tests and confidence intervals. Neither supports a positive claim that two strategies perform equivalently — failing to reject is not evidence of no difference. Since comparison is a central output of the harness, it needs a test that can conclude equivalence rather than only report its absence.

Proposal

• Implement two one-sided tests (TOST) over the primary metric, per strategy pair.
• Define a smallest effect size of interest up front, anchored to something defensible — for example, the quality delta at which the added cost and latency of a more expensive strategy becomes worthwhile.
• Report TOST alongside the existing NHST result and CI in generated tables.
• Apply under both the intent-to-treat and valid-only conventions already implemented.

Done when:

• SESOI documented with its justification, not merely chosen
• TOST output present in generated results tables
• Output distinguishes equivalent, inconclusive, and different for every pair

Scope

• In scope: TOST implementation, SESOI definition and documentation, integration into existing table generation.
• Out of scope: Bayesian alternatives; changing the primary metric; re-running any model.

Open questions

• Is one SESOI used across all metrics, or does each metric get its own? Entity F1 and relation F1 plausibly warrant different thresholds.
• Should the SESOI be derived from the cost data — the quality gain that offsets measured cost delta — or set as a flat convention?
• Multiple-comparison correction across pairs: needed, or does it over-penalise given the pairs are not independent?

Related

• Lakens (2017), Equivalence Tests: A Practical Primer.