fix(sleep): add optional per-task regression gate

docs/sleep/README.md

@@ -26,6 +26,15 @@ It synthesizes **SkillOpt** (validation-gated bounded text edits), **Claude Drea
 (offline consolidation; review-then-adopt), and the **agent-sleep** idea (short-term
 experience → long-term competence).
 
+The optional `gate_no_regression` config key strengthens the aggregate gate.
+It defaults to `false` for compatibility; when set to `true`, every validation
+task must preserve or improve its score under the configured `gate_metric`.
+The check applies to intermediate skill and memory candidates and to the fresh
+final replay. A missing task result or non-finite task score also blocks the
+candidate; an absent numeric score aborts evaluation. Task-level changes are
+included in `report.md`, `report.json`, `diagnostics.json`, the CLI's `--json`
+output, and the evidence log.
+
 > **Data boundary.** Harvesting is local and read-only. The `mock` backend makes no
 > provider calls. A real backend, however, sends truncated excerpts from harvested
 > sessions and derived tasks to the provider you select for mining, replay, judging,

plugins/README.md

@@ -231,10 +231,11 @@ python -m skillopt_sleep run --project "$(pwd)" \
 ### Advanced config
 
 The JSON/YAML config under `~/.skillopt-sleep/` supports additional engine keys,
-including `gate_mode`, `gate_metric`, `dream_rollouts`, `dream_factor`, `recall_k`,
-`evolve_memory`, and `evolve_skill`. These are config keys, not aliases for the
-unsupported CLI flags listed above. Shipping defaults are conservative:
-`gate_mode="on"`, `dream_rollouts=1`, `dream_factor=0`, and `recall_k=0`.
+including `gate_mode`, `gate_metric`, `gate_no_regression`, `dream_rollouts`,
+`dream_factor`, `recall_k`, `evolve_memory`, and `evolve_skill`. These are config
+keys, not aliases for the unsupported CLI flags listed above. Shipping defaults
+are conservative: `gate_mode="on"`, `gate_no_regression=false`,
+`dream_rollouts=1`, `dream_factor=0`, and `recall_k=0`.
 
 The managed `schedule` command stores only the project, backend, time, and
 optional auto-adopt setting. It does not copy `--source`, `--cursor-home`,

plugins/claude-code/skills/skillopt-sleep/SKILL.md

@@ -109,6 +109,7 @@ Beyond the CLI flags, advanced behavior is controlled via config:
 - **`preferences`** — free-text house rules injected into the optimizer's reflect step (e.g. "Always use async/await", "Answers in `\boxed{}`").
 - **`gate_mode`** — `on` (default, validation-gated) or `off` (greedy, accept all edits).
 - **`gate_metric`** — `hard`, `soft`, or `mixed` (default). Controls how the held-out gate scores.
+- **`gate_no_regression`** — `false` by default. Set to `true` to reject a candidate when any validation task's configured gate score decreases.
 - **`dream_rollouts`** — >1 enables multi-rollout contrastive reflection per task.
 - **`recall_k`** — >0 recalls K similar past tasks into the dream (long-term memory).
 - **`evolve_memory`** / **`evolve_skill`** — independently toggle CLAUDE.md vs SKILL.md consolidation.

plugins/codex/README.md

@@ -79,7 +79,7 @@ mining, replay, and reflection can be slow and otherwise quiet. Use
 prefer tasks that match the target skill's path, headings, and content. The
 implemented main-CLI flags work the same across the shared integrations, and
 `--preferences "..."` is available for house rules. Advanced keys such as
-`gate_mode`, `dream_rollouts`, and `recall_k` belong in the Sleep config; the
+`gate_mode`, `gate_no_regression`, `dream_rollouts`, and `recall_k` belong in the Sleep config; the
 nightly CLI does not expose `--gate`, `--rollouts-k`, token/time-budget, or
 optimizer/target-split flags. See the
 [shared CLI reference](../README.md#supported-cli-surface).

plugins/codex/skills/skillopt-sleep/SKILL.md

@@ -129,6 +129,7 @@ without `crontab`, `schedule` prints a line for manual installation.
 - **`preferences`** — free-text house rules for the optimizer
 - **`gate_mode`** — `on` (validation-gated, default) or `off` (greedy)
 - **`gate_metric`** — `hard` | `soft` | `mixed` (default)
+- **`gate_no_regression`** — `false` by default; set it to `true` to reject a candidate when any validation task's gate score decreases
 - **`dream_rollouts`** — >1 for multi-rollout contrastive reflection
 - **`recall_k`** — >0 recalls similar past tasks from the archive
 

plugins/copilot/copilot-instructions.snippet.md

@@ -38,7 +38,10 @@ my preferences", or "make the agent improve from past usage", use the MCP tools:
 ### Advanced config (`~/.skillopt-sleep/config.json`)
 
 - `preferences` — free-text house rules for the optimizer
-- `gate_mode` — `on` (default) or `off`; `dream_rollouts` — >1 for more signal
+- `gate_mode` — `on` (default) or `off`
+- `gate_no_regression` — `false` by default; set it to `true` to reject a
+  candidate when any validation task's configured gate score decreases
+- `dream_rollouts` — >1 for more signal
 - `evolve_memory` / `evolve_skill` — toggle which docs consolidate
 
 Always show the user the held-out baseline → candidate score and the proposed

plugins/cursor/skills/skillopt-sleep/SKILL.md

@@ -166,6 +166,10 @@ Real backends reject task files that remain unreviewed. Never include raw
 transcripts, credentials, secrets, or sensitive task content in messages,
 commits, or generated summaries.
 
+`gate_no_regression` is a config-only safeguard in
+`~/.skillopt-sleep/config.json`. It defaults to `false`; set it to `true` to
+reject a candidate when any validation task's configured gate score decreases.
+
 ## Scheduling
 
 Scheduling is opt-in. The scheduler persists project, backend, time, and the

skillopt_sleep/__main__.py

@@ -36,7 +36,7 @@
 from skillopt_sleep.harvest_sources import harvest_for_config
 from skillopt_sleep.mine import mine
 from skillopt_sleep.staging import adopt as adopt_staging
-from skillopt_sleep.staging import latest_staging
+from skillopt_sleep.staging import json_safe, latest_staging
 from skillopt_sleep.state import SleepState
 from skillopt_sleep.tasks_file import load_tasks_file, make_tasks_payload, write_tasks_file
 
@@ -50,7 +50,7 @@ def _read_text(path: str) -> str:
 
 
 def _report_payload(rep, outcome) -> Dict[str, Any]:
-    return {
+    return json_safe({
         "night": rep.night,
         "accepted": rep.accepted,
         "gate_action": rep.gate_action,
@@ -63,10 +63,12 @@ def _report_payload(rep, outcome) -> Dict[str, Any]:
         "n_rejected_edits": len(rep.rejected_edits),
         "edits": [e.__dict__ for e in rep.edits],
         "rejected_edits": [e.__dict__ for e in rep.rejected_edits],
+        "gate_no_regression": bool(getattr(rep, "gate_no_regression", False)),
+        "gate_trials": _redact_deep(getattr(rep, "gate_trials", [])),
         "notes": rep.notes,
         "staging_dir": outcome.staging_dir,
         "adopted": outcome.adopted,
-    }
+    })
 
 
 def _add_common(p: argparse.ArgumentParser) -> None:

skillopt_sleep/config.py

@@ -63,6 +63,7 @@
     "preferences": "",            # free-text house rules injected into reflect as a prior
     "gate_metric": "mixed",       # hard | soft | mixed (mixed best for tiny holdouts)
     "gate_mixed_weight": 0.5,
+    "gate_no_regression": False,    # reject any candidate that lowers a val-task score
     "replay_mode": "mock",        # report label; fresh-worktree replay is not implemented
     # ── dream + recall (opt-in; defaults reproduce the prior single-shot loop) ─
     "dream_rollouts": 1,          # >1 => multi-rollout contrastive reflection per task

skillopt_sleep/consolidate.py

@@ -8,22 +8,27 @@
 """
 from __future__ import annotations
 
-import os
+import math
 from dataclasses import dataclass, field
-from typing import List, Optional, Tuple
+from typing import List, Tuple
 
 from skillopt_sleep.backend import Backend
-from skillopt_sleep.memory import apply_edits_detailed
-from skillopt_sleep.replay import aggregate_scores, replay_batch
-from skillopt_sleep.types import EditRecord, ReplayResult, TaskRecord
-
 
 # Self-contained validation gate (vendored from SkillOpt; zero dependency on the
 # research package, so this open-source tool stays decoupled from the paper code).
 from skillopt_sleep.gate import evaluate_gate, select_gate_score
+from skillopt_sleep.memory import apply_edits_detailed
+from skillopt_sleep.replay import aggregate_scores, replay_batch
+from skillopt_sleep.types import EditRecord, ReplayResult, TaskRecord
+
 _HAVE_REPO_GATE = True
 
 
+def _finite_score(value: float) -> float | None:
+    """Return a JSON-safe score, preserving finite values only."""
+    return value if math.isfinite(value) else None
+
+
 @dataclass
 class ConsolidationResult:
     accepted: bool
@@ -49,6 +54,11 @@ class ConsolidationResult:
     # saw, so its comparison cannot detect overfitting. A night in this state
     # stages edits but never certifies them.
     holdout_leaked: bool = False
+    # Each gated candidate carries the task-level comparison that produced its
+    # decision. This includes rejected intermediate skill/memory trials and the
+    # fresh final replay, so aggregate changes never hide an individual task.
+    # Keep new optional fields last to preserve positional construction.
+    gate_trials: List[dict] = field(default_factory=list)
 
 
 def _split(tasks: List[TaskRecord]) -> Tuple[List[TaskRecord], List[TaskRecord], bool]:
@@ -108,6 +118,65 @@ def _holdout_detail(pairs: List[Tuple[TaskRecord, ReplayResult]]) -> List[dict]:
     return out
 
 
+def _task_deltas(
+    tasks: List[TaskRecord],
+    baseline_pairs: List[Tuple[TaskRecord, ReplayResult]],
+    candidate_pairs: List[Tuple[TaskRecord, ReplayResult]],
+    metric: str,
+    mixed_weight: float,
+) -> List[dict]:
+    """Compare candidate scores with the matching baseline validation tasks."""
+    baseline_by_id = {task.id: result for task, result in baseline_pairs}
+    candidate_by_id = {task.id: result for task, result in candidate_pairs}
+    out: List[dict] = []
+    for task in tasks:
+        baseline = baseline_by_id.get(task.id)
+        candidate = candidate_by_id.get(task.id)
+        baseline_score = (
+            select_gate_score(baseline.hard, baseline.soft, metric, mixed_weight)
+            if baseline is not None
+            else None
+        )
+        candidate_score = (
+            select_gate_score(candidate.hard, candidate.soft, metric, mixed_weight)
+            if candidate is not None
+            else None
+        )
+        if baseline_score is None or candidate_score is None:
+            out.append({
+                "task_id": task.id,
+                "tags": list(task.tags or []),
+                "baseline_score": baseline_score,
+                "candidate_score": candidate_score,
+                "status": "regressed",
+                "scores_are_finite": False,
+            })
+            continue
+        scores_are_finite = math.isfinite(baseline_score) and math.isfinite(
+            candidate_score
+        )
+        if not scores_are_finite:
+            # The aggregate gate already rejects NaN, but +inf could otherwise
+            # look like an improvement. Strict mode treats any malformed task
+            # comparison as a regression and therefore fails closed.
+            status = "regressed"
+        elif candidate_score > baseline_score:
+            status = "improved"
+        elif candidate_score < baseline_score:
+            status = "regressed"
+        else:
+            status = "unchanged"
+        out.append({
+            "task_id": task.id,
+            "tags": list(task.tags or []),
+            "baseline_score": _finite_score(baseline_score),
+            "candidate_score": _finite_score(candidate_score),
+            "status": status,
+            "scores_are_finite": scores_are_finite,
+        })
+    return out
+
+
 def consolidate(
     backend: Backend,
     tasks: List[TaskRecord],
@@ -117,6 +186,7 @@ def consolidate(
     edit_budget: int = 4,
     gate_metric: str = "mixed",
     gate_mixed_weight: float = 0.5,
+    gate_no_regression: bool = False,
     gate_mode: str = "on",       # "on" (hard/soft per gate_metric) | "off" (greedy)
     rollouts_k: int = 1,         # >1 => multi-rollout contrastive reflection
     evolve_skill: bool = True,
@@ -145,6 +215,7 @@ def consolidate(
     # both wasted cost and contrary to the "no val set required" design.
     if gate_off:
         base_hard, base_soft = 0.0, 0.0
+        base_pairs: List[Tuple[TaskRecord, ReplayResult]] = []
     else:
         evlog.set_phase(backend, "baseline_val")
         base_pairs = replay_batch(backend, val_tasks, skill, memory)
@@ -167,13 +238,16 @@ def consolidate(
     all_applied: List[EditRecord] = []
     all_rejected: List[EditRecord] = []
     all_unmatched: List[EditRecord] = []
+    gate_trials: List[dict] = []
+    current_pairs = base_pairs
 
     def _edits_payload(edits: List[EditRecord]) -> List[dict]:
         return [{"op": e.op, "content": e.content, "anchor": e.anchor,
                  "rationale": e.rationale} for e in edits]
 
     def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
-        nonlocal cand_skill, cand_memory, base_score, all_applied, all_rejected
+        nonlocal cand_skill, cand_memory, base_score, current_pairs
+        nonlocal all_applied, all_rejected
         if ev is not None:
             ev.log("reflect", "edits_returned", target=which,
                    n_edits=len(edits), edits=_edits_payload(edits))
@@ -201,14 +275,33 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
         pairs = replay_batch(backend, val_tasks, trial_skill, trial_memory)
         h, s = aggregate_scores(pairs)
         cand_score = select_gate_score(h, s, gate_metric, gate_mixed_weight)
-        improved = cand_score > base_score
+        task_deltas = _task_deltas(
+            val_tasks, current_pairs, pairs, gate_metric, gate_mixed_weight
+        )
+        blocked_by_regression = bool(
+            gate_no_regression
+            and any(row["status"] == "regressed" for row in task_deltas)
+        )
+        trial_base_score = base_score
+        improved = cand_score > base_score and not blocked_by_regression
+        gate_trials.append({
+            "target": which,
+            "baseline_score": _finite_score(trial_base_score),
+            "candidate_score": _finite_score(cand_score),
+            "accepted": improved,
+            "blocked_by_regression": blocked_by_regression,
+            "task_deltas": task_deltas,
+        })
         if ev is not None:
             ev.log("gate", "trial", target=which, mode="gated",
-                   baseline_score=base_score, cand_hard=h, cand_soft=s,
+                   baseline_score=trial_base_score, cand_hard=h, cand_soft=s,
                    cand_score=cand_score, accepted=improved,
+                   blocked_by_regression=blocked_by_regression,
+                   task_deltas=task_deltas,
                    n_edits=len(applied))
         if improved:
-            base_score = max(base_score, cand_score)
+            base_score = cand_score
+            current_pairs = pairs
             all_applied.extend(applied)
             return new_doc
         all_rejected.extend(applied)
@@ -218,12 +311,13 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
         if rollouts_k > 1:
             # multi-rollout contrastive reflection: run each train task K times
             # and distill a rule from the good-vs-bad contrast (the imagination signal).
-            from skillopt_sleep.rollout import multi_rollout, contrastive_reflect
             # Parallelize across tasks (each multi_rollout also parallelizes its K
             # attempts). This dream phase is the dominant cost; serial execution
             # times out on real backends. Cap total in-flight at the worker env.
             import os
             from concurrent.futures import ThreadPoolExecutor
+
+            from skillopt_sleep.rollout import contrastive_reflect, multi_rollout
             try:
                 _w = int(os.environ.get("SKILLOPT_SLEEP_WORKERS", "1"))
             except ValueError:
@@ -289,6 +383,13 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
         final_hard, final_soft = aggregate_scores(final_pairs)
         final_score = select_gate_score(final_hard, final_soft, gate_metric, gate_mixed_weight)
         base_gate_score = select_gate_score(base_hard, base_soft, gate_metric, gate_mixed_weight)
+        final_task_deltas = _task_deltas(
+            val_tasks, base_pairs, final_pairs, gate_metric, gate_mixed_weight
+        )
+        final_blocked_by_regression = bool(
+            gate_no_regression
+            and any(row["status"] == "regressed" for row in final_task_deltas)
+        )
         if _HAVE_REPO_GATE:
             gate = evaluate_gate(
                 candidate_skill=cand_skill,
@@ -304,10 +405,18 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
                 mixed_weight=gate_mixed_weight,
             )
             action = gate.action
-            accepted = bool(all_applied) and final_score > base_gate_score
+            accepted = (
+                bool(all_applied)
+                and final_score > base_gate_score
+                and not final_blocked_by_regression
+            )
         else:
             action = "accept" if final_score > base_gate_score else "reject"
-            accepted = bool(all_applied) and final_score > base_gate_score
+            accepted = (
+                bool(all_applied)
+                and final_score > base_gate_score
+                and not final_blocked_by_regression
+            )
         # The gate scores documents, not edit bookkeeping: when every proposed
         # edit was dropped during the per-target trials, `all_applied` is empty
         # and nothing changed, yet the score comparison can still yield an
@@ -332,6 +441,14 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
                 if edit not in all_rejected:
                     all_rejected.append(edit)
             all_applied = []
+        gate_trials.append({
+            "target": "final",
+            "baseline_score": _finite_score(base_gate_score),
+            "candidate_score": _finite_score(final_score),
+            "accepted": accepted,
+            "blocked_by_regression": final_blocked_by_regression,
+            "task_deltas": final_task_deltas,
+        })
 
     if ev is not None:
         w = max(0.0, min(1.0, float(gate_mixed_weight)))
@@ -351,6 +468,10 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
                baseline_hard=base_hard, baseline_soft=base_soft,
                candidate_hard=final_hard, candidate_soft=final_soft,
                metric=gate_metric, mixed_weight=gate_mixed_weight,
+               blocked_by_regression=(
+                   final_blocked_by_regression if not gate_off else False
+               ),
+               task_deltas=(final_task_deltas if not gate_off else []),
                formula=formula, n_applied=len(all_applied),
                n_rejected=len(all_rejected),
                n_unmatched=len(all_unmatched), night=night)
@@ -368,6 +489,7 @@ def _gate_apply(doc: str, edits: List[EditRecord], which: str) -> str:
         holdout_baseline=base_hard,
         holdout_candidate=final_hard,
         holdout_detail=holdout_detail,
+        gate_trials=gate_trials,
         reflect_raw=getattr(backend, "last_reflect_raw", "") or "",
         call_error=getattr(backend, "last_call_error", "") or "",
         holdout_leaked=holdout_leaked,