Skip to content

fix(sleep): add optional per-task regression gate - #222

Open
Langning Zhang (Boulea7) wants to merge 1 commit into
microsoft:mainfrom
Boulea7:fix/gate-no-regression-deltas
Open

fix(sleep): add optional per-task regression gate#222
Langning Zhang (Boulea7) wants to merge 1 commit into
microsoft:mainfrom
Boulea7:fix/gate-no-regression-deltas

Conversation

@Boulea7

Copy link
Copy Markdown

Summary

  • add a default-off gate_no_regression option that rejects skill, memory, or final candidates when any validation task gets worse
  • record observed per-task score changes for intermediate trials and the final replay in the human and machine-readable reports
  • treat missing or non-finite task results as regressions when the option is enabled, while keeping persisted JSON standard and redacted

Testing

  • python3.12 -m pytest -q (953 passed, 6 skipped, 130 subtests)
  • python3.12 -m mkdocs build --strict
  • ruff check on the changed Python files
  • python3.12 -m compileall -q skillopt_sleep tests/test_gate_no_regression.py

Closes #174

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate compares slice averages only: an aggregate gain can hide a per-task regression, and improvements are never attributed to the rule that caused them

1 participant