Model-selection decision rule and cost–quality frontier

[Colinho22]

Motivation

The harness collects enough data to say which model and strategy to choose under a given budget, but stops at reporting measurements. Turning that into an explicit rule makes the output usable without reading the full analysis.

Proposal

• Compute the Pareto frontier over quality, cost per diagram, and latency across all model × strategy × mode combinations.
• Express it as a decision rule: a quality-per-unit-cost threshold, or a decision tree keyed on latency budget and cost ceiling.
• Emit a frontier plot and a decision table.
• Include provider-switching considerations: prompt compatibility, structured-output support, rate limits.

Done when:

• Frontier computed from run data with no hand-entered values
• Decision rule stated so it can be applied standalone
• Plot and table regenerate from one command

Scope

• In scope: frontier computation, decision rule derivation, plot and table generation, a documented note on switching considerations.
• Out of scope: benchmarking providers not already in the harness; live pricing lookups.

Open questions

• Cost per attempt or cost per successful diagram? With non-trivial failure rates these diverge sharply, and the second is the one a practitioner actually pays.
• Should the frontier be computed per complexity tier or pooled? A model that wins on simple inputs may lose on complex ones, which a pooled frontier would hide.
• How should the rule express uncertainty — does a model on the frontier by a margin inside the CI belong there?

Related

• Depends on Retry, fallback, and validation layer #96 — recovery changes effective cost per success.