Skip to content

Bind accepted-task cost feedback to complete per-candidate cohorts #722

Open

Description

@Meta-Develop

Summary

Same-run accepted-task cost is reduced to one scalar without selection-lane identity. That value is then used as observed cost for unrelated previous candidates, and it remains available when the cohort contains costly incomplete attempts.

Evidence and reproduction

On main fa0c79b9ae38eca2c59b2b1911becfe1a6b3c9ab:

  • src/supervise/accepted_task_cost.rs:31-38 drops rows with unknown terminal results; lines 49-55 omit incomplete cost buckets from the numeric rollup. Lines 70-77 still return a numeric value from the complete subset.
  • The existing test at lines 138-148 combines a complete accepted attempt costing 12 with an incomplete accepted attempt containing 550 of known spend and asserts a value of 12. Incompleteness is counted, but that subset value remains available to selection.
  • src/supervise/scheduler.rs:2655-2661,2680-2706 records assignment/attempt/result/costs without candidate/runtime/model/effort/task-shape identity; lines 2743-2772 aggregate all rows together.
  • Scheduler/selection propagation (scheduler.rs:684-695, selection_bridge.rs:1126-1128,1312-1348) passes the scalar into role reselection. src/selection/selector.rs:1100-1106 assigns it to each role's own previous candidate.

For a two-candidate fixture, distinct actual costs cannot be attributed through the current row type. For the existing 12-plus-550 fixture, the full two-acceptance cohort has a known lower bound of 281 per acceptance, while feedback uses 12 from one complete row.

Acceptance criteria

  • Preserve exact task shape, role, candidate, runtime, model and effort in attempt-cost evidence and group feedback by the corresponding selection lane.
  • Prevent one lane's observation from replacing another candidate's expected cost.
  • Unknown terminal status or missing required buckets must make full-cohort feedback unavailable or explicitly bounded. A complete-subset descriptive average may remain separately labelled.
  • Preserve known partial spend and incompleteness in reporting/provenance; do not discard failed/rejected/rework costs.
  • Test two-lane isolation, complete-plus-incomplete cohorts, unknown outcomes with known spend, and rejected retries followed by acceptance.

Scope and verification

Source/test-definition inspection only; the existing test was not rerun. #631 concerns empirical qualification, and #695 concerns observed-model pricing attribution; neither preserves candidate cohort identity through this aggregation. No matching open Issue/PR covers this feedback defect.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:inboxAwaiting agent triagebugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions