You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Preserve supported objective axes during Pareto pruning #723
Evaluation removes candidates using fewer axes than the objective function supports. A profile preferred by a quota, latency, rework or review objective can disappear before scoring.
Evidence and reproduction
On main fa0c79b9ae38eca2c59b2b1911becfe1a6b3c9ab, src/eval_harness/issue26.rs:421-464 normalizes and scores cost, tokens, wall time, churn and review, but it only scores profiles retained by the frontier. Its dominance test at lines 752-769 uses monetary cost and raw quality alone.
With equal raw quality, let A cost 1 and consume 100 tokens/time/churn/review units; let B cost 2 and consume 1 of each. The harness prunes B because A costs less, so a 100%-latency or 100%-quota objective cannot choose B.
The newer src/evaluation.rs:3126-3159 dominance path includes cost/quota/latency/quality, but omits retry/rework and human-review metrics that scoring supports at lines 3194-3266.
Acceptance criteria
Dominance must include every typed raw axis used by the applicable objective, or defer pruning when that objective uses an unrepresented axis.
Preserve raw quality safeguards and treat missing metrics as unknown rather than zero.
Add independent trade-off fixtures for quota, latency, retry/rework and human review, showing that a fully weighted objective can retain and select its superior profile.
Preserve deterministic ordering and rescore provenance across both evaluation paths.
Scope and verification
Source-level counterexample; no benchmark or provider run was performed. Closed #26/#150 delivered the harness/objective foundations. No current Issue/PR covers this pruning mismatch.
Activity