Skip to content

Add per-model cloud rate scheduling and task manifests - #26

Closed
laceyp99 wants to merge 8 commits into
mainfrom
t3code/eval-model-rate-scheduler
Closed

laceyp99 wants to merge 8 commits into
mainfrom
t3code/eval-model-rate-scheduler

Conversation

@laceyp99

@laceyp99 laceyp99 commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add validated per-model RPM pacing with configurable per-model and global cloud concurrency limits.
  • Use Core's provider_call progress event as the dispatch boundary and durably journal dispatched before allowing a provider request to continue.
  • Record task setup and validation errors before that boundary as pre_dispatch_failed; these tasks do not advance the model RPM clock or stop the remaining model queue.
  • Stop queued work after typed provider throttling while allowing already-dispatched work and other model queues to finish.
  • Persist atomic task manifests and an append-only transition journal with expanded summary counters for completed, failed, pre-dispatch-failed, throttled, and unstarted tasks.
  • Document scheduling, dispatch durability, manifest artifacts, and failure behavior, with expanded evaluator and analysis regression coverage.

close #10

Testing

  • uv sync --locked --all-extras - passed
  • uv run --locked --all-extras ruff format --check . - 16 files already formatted
  • uv run --locked --all-extras ruff check . - passed
  • uv run --locked --all-extras pytest -q - 162 passed
  • uv build - passed

Copy link
Copy Markdown
Owner Author

Closing this in favor of #27 after comparing the two approaches holistically.

This PR implements genuine per-model RPM pacing, but doing so also introduces a substantial scheduling and durability subsystem into Eval: dispatch acknowledgements, multiple concurrency layers, task manifests, an append-only transition journal, queue cancellation, and recovery-oriented state transitions. That machinery is defensible for production-grade quota scheduling, but it is disproportionate for this personal evaluator and materially increases the surface area we would need to maintain and trust.

#27 makes the narrower architectural choice explicit: Eval owns a configurable per-provider concurrency cap and records typed provider throttling as a distinct result, but it does not claim to enforce RPM, TPM, or RPD quotas. For the expected supervised workloads, that is the preferred tradeoff. If strict quota scheduling becomes necessary later, it should be implemented once in conductor-core at the provider dispatch boundary rather than as Eval-specific orchestration.

This closure is therefore a scope and maintainability decision, not a judgment that the implementation here lacks rigor.

@laceyp99 laceyp99 closed this Aug 13, 2026
@laceyp99
laceyp99 deleted the t3code/eval-model-rate-scheduler branch August 13, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement real time-based provider rate limiting

1 participant