Skip to content

add invalid retry storm guard#543

Open
wang-tong0 wants to merge 2 commits into
mainfrom
codex/invalid-retry-storm-guard
Open

add invalid retry storm guard#543
wang-tong0 wants to merge 2 commits into
mainfrom
codex/invalid-retry-storm-guard

Conversation

@wang-tong0

Copy link
Copy Markdown
Collaborator

Summary

Adds an automatic guard for challenger models that repeatedly produce invalid evaluation attempts without writing those invalid attempts into sample_results.

What changed

  • Executor workers now publish compact invalid-attempt summaries in worker_status_<env> under invalid_subjects.
  • Invalid attempts are categorized broadly as model, stream, infra retryable, or env invalid.
  • Scheduler checks for long-running invalid retry storms before the overlap gate and terminates the challenger only when the endpoint is still healthy and infra errors are not dominant.
  • Added tests for executor invalid reporting, scheduler auto-termination, and the infra-retryable non-termination path.

Why

A model like UID247 can repeatedly emit malformed tool calls or broken structured output. Those attempts should not become valid zero-score samples, but without a separate signal the scheduler can keep retrying the same invalid work indefinitely.

Validation

  • pytest -q tests/test_executor_worker.py tests/test_scheduler_flow.py tests/test_state_store.py tests/test_db_cli_sample_progress.py -q
  • python -m compileall -q affine/src/executor affine/src/scheduler affine/src/scorer tests/test_executor_worker.py tests/test_scheduler_flow.py
  • git diff --check

@wang-tong0 wang-tong0 marked this pull request as ready for review June 14, 2026 08:06
@wang-tong0 wang-tong0 changed the title [codex] add invalid retry storm guard add invalid retry storm guard Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant