Catch refused-command loops and unrun tests; run the held-out gate off the turn, deterministically

docs/RFCS/RFC-014-validation-and-procedure.md

@@ -185,10 +185,17 @@ decides.
 ### The gate in production (karamazov-7mo.4 / ylte.4)
 
 `samizdat.heldout` is what runs the battery. Every `cell save`, `manifest
-save`/`patch` and `policy save` calls `heldout/check-edit` before it stores:
-the battery is replayed with the candidate in place and without it, and an
-edit under which a target that passed fails is refused with the targets named
-(`prompts/heldout-refused.md`). Nothing is saved; what runs is unchanged.
+save`/`patch` and `policy save` validates in the turn as before, then hands
+the edit to `heldout/defer!`: the battery is replayed with the candidate in
+place and without it on a background lane, and an edit under which a target
+that passed fails is refused with the targets named
+(`prompts/heldout-refused.md`). **Off the turn** (karamazov-nha1): a refusal is
+confirmed by a second replay, so one case's refusal took 733 s against a
+900 s turn deadline while the gate ran inside the tool call. The tool now
+answers at once that the edit is pending and not live; the verdict reaches
+the branch that made it as a `message` directive on a later turn
+(`prompts/heldout-decided.md`) and is journalled as `:heldout-verdict`. Edits
+are measured and committed one at a time, in the order they were made.
 
 **One child process per case.** A replay must read the candidate as the
 project's userspace from every branch fiber, and a dynamic binding does not
@@ -201,6 +208,13 @@ replays the recording through `beam/run!`, and runs `battery/check`. Side
 calls a cell makes on its own (critic, judges) were never recorded and are
 refused identically on both sides.
 
+**The same computation every time** (karamazov-x0dx). A recording that forked
+replayed its branches concurrently, and which turn landed first was timing:
+one baseline read 11/14 and 13/14 minutes apart. The child runs
+`beam/run!` with `:serial-turns? true` — each round's turns run one after
+another in branch order — and `:oversight? false`, since the supervisor's
+passes fire on a clock and the recording cannot answer them.
+
 **The rule is non-compensatory, per target.** No target that passed at
 baseline may fail on the candidate, and both sides must cover the same
 targets. Ties are accepted. A case that does not run at baseline is set aside

resources/cells/feature.clj

@@ -537,9 +537,16 @@
         (let [r (when-not (str/blank? (str cmd))
                   (proc/run {:timeout-ms (or (get-in config [:run :verify-timeout-ms]) 600000)}
                             "sh" "-c" (str "cd " root " && " cmd)))
-              tests-passed? (if r
-                              (and (not (:timeout r)) (zero? (or (:exit r) 1)))
-                              true)
+              green? (if r
+                       (and (not (:timeout r)) (zero? (or (:exit r) 1)))
+                       true)
+              ;; Green on a suite that never ran the tests this run added or
+              ;; changed is not green about them (karamazov-khzy).
+              unrun (when (and r green?)
+                      (verify/unrun-tests root
+                                          (gitdiff/changed-files root (:git-baseline ctx))
+                                          (str (:out r) "\n" (:err r))))
+              tests-passed? (and green? (empty? unrun))
               ;; The criteria are checked whether or not the suite is green:
               ;; a run whose suite is red AND whose criteria fail should hear
               ;; about both at once rather than one per round. Fail-open on a
@@ -556,6 +563,8 @@
                              (remove str/blank?
                                      [(cond (nil? r) "no :verify-cmd configured"
                                             (:timeout r) "tests TIMED OUT"
+                                            (seq unrun) (prompt/render "tests-not-run"
+                                                                       {:nses (str/join ", " unrun)})
                                             tests-passed? "tests passed"
                                             :else (str "tests FAILED (exit " (:exit r) ")\n"
                                                        (tail (str (:out r) "\n" (:err r)) 25)))

resources/gates.edn

@@ -708,6 +708,22 @@
         the tracked set narrows to the intersection so divergent touches
         break the streak on their own; a no-file call resets it."}
 
+ :refused-repeat-threshold
+ {:value 2 :kind :threshold :capability-tunable? true
+  :provenance ["karamazov-q1tt"]
+  :doc "How many refusals of one command head (by the shell policy: denied,
+        or needing an approval nobody gives) before the refused-again gate
+        says so. Two: the first refusal already explained itself; the second
+        is the branch trying the same thing another way. Run bcd61b39 retried
+        `rm`, `nohup jolt … &` and `true` ~15 times over ~50 turns."}
+
+ :max-refused-nudges
+ {:value 3 :kind :re-fire-guard :capability-tunable? true
+  :provenance ["karamazov-q1tt"]
+  :doc "How often the refused-again gate may fire on one branch. Each firing
+        names the head it is about, so a branch that moves on from `rm` to
+        `nohup` hears about the second too; past this it is the supervisor's."}
+
  :max-file-thrash-nudges
  {:value 1 :kind :re-fire-guard :capability-tunable? true
   :doc "How often the file-thrash gate may fire on one branch. One-shot per
@@ -1978,15 +1994,22 @@
           :trail-turns 12
           :trail-chars 300
           :skip "\\.sqlite3(-wal|-shm|-journal)?$"}
-  :provenance ["karamazov-7mo.4" "karamazov-ylte.4" "2609.24972v2" "2609.09153v1"]
+  :provenance ["karamazov-7mo.4" "karamazov-ylte.4" "karamazov-nha1" "karamazov-x0dx"
+               "2609.24972v2" "2609.09153v1"]
   :kind :policy :capability-tunable? false
   :doc "THE HELD-OUT GATE, STAGE 1 (samizdat.heldout). Every cell, manifest
         and policy edit a tool saves is first replayed against the project's
         frozen battery — .samizdat/<:dir>/<subject>/<case>.edn, stored in
         battery_cases the first time it is read — with the edit in place and
         without it, each case in a child process on a copy of the tree the
         recorded run started from. An edit under which a target that passed
-        fails is refused and nothing is saved. Ties are accepted.
+        fails is refused and nothing is saved. Ties are accepted. The replay
+        runs OFF THE TURN (karamazov-nha1): the tool answers at once that
+        the edit is pending, and the verdict — live, or refused with the
+        targets named — reaches the branch that made it as a message on a
+        later turn. Replays run one turn at a time with no supervisor
+        stream, so two replays of one case are the same computation
+        (karamazov-x0dx).
 
         A project with no cases is not gated: it tunes itself on the compile
         and the soak, as before. :enabled? false turns the gate off where
@@ -3221,7 +3244,12 @@
     :root-strip-regex "^(test|gui|src)/"
     :ext-strip-regex "\\.cljc?$"
     :ns-whitelist-regex "[a-zA-Z0-9.*-]+"
-    :cmd-prefix "jolt -A:test -e '"}
+    :cmd-prefix "jolt -A:test -e '"
+    ;; What the suite prints for each namespace it ran, {{ns}} standing for
+    ;; the namespace (clojure.test's run-tests). A changed test namespace
+    ;; the full suite's output does not show is a test the suite never ran
+    ;; (karamazov-khzy); an output with no marker at all is not judged.
+    :ran-marker "Testing {{ns}}"}
    :kind :threshold :capability-tunable? false
    :doc "The focused-verify conventions as data (see the comment above)."}
 
@@ -3944,6 +3972,30 @@
    :prediction "the branch retracts, decomposes, or changes technique"
    :window 3}
 
+  {:gate :refused-again
+   :provenance ["karamazov-q1tt"]
+   :priority 9.4
+   :budget :max-refused-nudges
+   :doc "A command head the shell policy refused was just refused again. The
+         refusal text explains itself once; the second attempt — usually the
+         same head in a different command — means the branch has not taken
+         it in, and in an unattended run nobody will approve it. Names the
+         head and the count and asks for a different route. Above
+         file-thrash: a branch spending turns on refusals is not working at
+         all."
+   ;; the quantity the :when compares, recorded at every check (na2k.7)
+   :measure (get-in branch [:refused-heads (:last-refused branch)] 0)
+   :when (when-let [h (:last-refused branch)]
+           (>= (get-in branch [:refused-heads h] 0)
+               (threshold :refused-repeat-threshold)))
+   :message-form (sp/render-str (prompt "refused-again")
+                                {:head (:last-refused branch)
+                                 :count (get-in branch [:refused-heads (:last-refused branch)])
+                                 :total (:refusals branch)
+                                 :goal (state/stated-goal branch)})
+   :prediction "the branch stops issuing the refused head"
+   :window 3}
+
   {:gate :file-thrash
    :provenance ["karamazov-g86"]
    :priority 9.5

resources/manual.edn

@@ -85,6 +85,8 @@
     :summary "A checked manifest edit as data: ops (rename-cell, add-cell, remove-cell, set-edge, delete-edge, set-cell-field, set-dispatches) applied to the manifest as written, every reference to a renamed node rewritten (edges, dispatches, :invariants), the whole batch compiled once at the end so a node can be added and wired in one call. `manifest patch` is this over the stored text; `render` writes the result back over the original so comments survive; `cell-refs` and `diff-manifests` are behind `manifest refs` and `manifest diff`."}
    {:name samizdat.workflow/catalog
     :summary "Every workflow available to select or adapt, with its description."}
+   {:name samizdat.heldout/defer!
+    :summary "The held-out gate off the turn: after a save validates, the battery replays on a background lane and the edit is committed or refused there; the branch that made it is told by a message on a later turn. nil when the project has no battery, and the tool saves as before."}
    {:name samizdat.heldout/check-edit
     :summary "The held-out gate, stage 1: replay the project's battery (.samizdat/battery, stored in battery_cases) with a candidate cell, manifest or policy edit in place and without it, each case in a child process on the tree its recorded run started from, and refuse the edit when a target that passed fails. Ties accepted; a case that does not run at baseline is set aside by name. Opt-in by having cases; gates.edn :heldout. Each target is recorded in heldout_checks with its fixture, harness revision, model and scorer."}
    {:name samizdat.heldout/draft!

resources/prompts/heldout-decided.md

@@ -0,0 +1,3 @@
+{% if accepted %}Your edit to {{kind}} `{{name}}` passed the held-out battery{% if total %} ({{passed}} of {{total}} targets, nothing that passed before fails){% endif %} and is now live as v{{version}}.{% endif %}{% if refused %}Your edit to {{kind}} `{{name}}` was NOT saved — the held-out battery refused it.
+
+{{refusal}}{% endif %}{% if save-failed %}Your edit to {{kind}} `{{name}}` passed the held-out battery but did not go live: {{reason}}. What runs is unchanged.{% endif %}

resources/prompts/heldout-pending.md

@@ -0,0 +1 @@
+{{kind}} `{{name}}` compiled{% if soaked %} and dry-ran{% endif %}, and is now being replayed against this project's held-out battery, off your turn. It is NOT live yet: what runs is what ran before. You will get a message on a later turn saying whether it went live or which recorded behaviour it broke. Carry on meanwhile — and do not stack another edit to the same file on top of this one before you hear back.

resources/prompts/refused-again.md

@@ -0,0 +1,5 @@
+`{{head}}` has now been refused {{count}} times in this run ({{total}} refused commands in all). Rewording the command does not change the answer: the policy judges `{{head}}` itself, and in a run nobody is watching, a command that needs approval will not get it.
+
+Stop reaching for `{{head}}`. Take a route that does not need it — a command that has already run in this run, a tool made for the job (`write_file`/`edit_file`/`patch` to change files, `read_file` to read them, `eval` to run code), or leave out the step if it was only tidying up. If the step truly cannot be done without it, say so in your answer and carry on with the rest.{% if goal %}
+
+You are working on: {{goal}}{% endif %}

resources/prompts/tests-not-run.md

@@ -0,0 +1 @@
+tests passed, but the suite never ran {{nses}} — test code this run added or changed that the project's test runner does not include. A green suite says nothing about tests it did not run. Register them where the runner lists its test namespaces, then run the suite again and check they appear in its output.

resources/userspace.edn

@@ -60,7 +60,11 @@
    :acceptance-judge "prompts/acceptance-judge.md"
    :adopt-tool "prompts/adopt-tool.md"
    :battery-tool "prompts/battery-tool.md"
+   :heldout-decided "prompts/heldout-decided.md"
+   :heldout-pending "prompts/heldout-pending.md"
    :heldout-refused "prompts/heldout-refused.md"
+   :refused-again "prompts/refused-again.md"
+   :tests-not-run "prompts/tests-not-run.md"
    :adoption-offer "prompts/adoption-offer.md"
    :architect "prompts/architect.md"
    :ask-tool "prompts/ask-tool.md"

src/samizdat/agent/arbiter.clj

@@ -190,7 +190,7 @@
   :settle-called vocabulary. Kept beside the `case` below and asserted
   against it by the coherence test, so a gate that falls out of both is
   caught at test time rather than by settling :unmet forever."
-  #{:stuck :prologue-cap :progress-stalled :file-thrash :human-directive})
+  #{:stuck :prologue-cap :progress-stalled :file-thrash :human-directive :refused-again})
 
 (defn settle
   "Decide whether an open prediction came true, given what the branch did.
@@ -271,6 +271,11 @@
         ;; counter measures (karamazov-g86).
         :file-thrash (< (get-in branch-after [:file-touch :streak] 0)
                         (gates/threshold :file-thrash-threshold))
+        ;; Met when a turn after it was NOT refused: the branch took another
+        ;; route. Read off the refusal record rather than a tool name, since
+        ;; the refused call and the one that replaces it are both `shell`
+        ;; (karamazov-q1tt).
+        :refused-again (nil? (:last-refused branch-after))
         :human-directive true
         (tools-met? gate))
       (if late? :met-late :met)