Skip to content

Preserve supported objective axes during Pareto pruning #723

Open

Description

@Meta-Develop

Summary

Evaluation removes candidates using fewer axes than the objective function supports. A profile preferred by a quota, latency, rework or review objective can disappear before scoring.

Evidence and reproduction

On main fa0c79b9ae38eca2c59b2b1911becfe1a6b3c9ab, src/eval_harness/issue26.rs:421-464 normalizes and scores cost, tokens, wall time, churn and review, but it only scores profiles retained by the frontier. Its dominance test at lines 752-769 uses monetary cost and raw quality alone.

With equal raw quality, let A cost 1 and consume 100 tokens/time/churn/review units; let B cost 2 and consume 1 of each. The harness prunes B because A costs less, so a 100%-latency or 100%-quota objective cannot choose B.

The newer src/evaluation.rs:3126-3159 dominance path includes cost/quota/latency/quality, but omits retry/rework and human-review metrics that scoring supports at lines 3194-3266.

Acceptance criteria

  • Dominance must include every typed raw axis used by the applicable objective, or defer pruning when that objective uses an unrepresented axis.
  • Preserve raw quality safeguards and treat missing metrics as unknown rather than zero.
  • Add independent trade-off fixtures for quota, latency, retry/rework and human review, showing that a fully weighted objective can retain and select its superior profile.
  • Preserve deterministic ordering and rescore provenance across both evaluation paths.

Scope and verification

Source-level counterexample; no benchmark or provider run was performed. Closed #26/#150 delivered the harness/objective foundations. No current Issue/PR covers this pruning mismatch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:inboxAwaiting agent triagebugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions