Share the turn chain, gate self-edits on a held-out battery, fix resumed supervisor passes

dev/samizdat/dev/arena.clj

@@ -53,6 +53,7 @@
             [samizdat.agent.acceptance :as acceptance]
             [samizdat.agent.beam :as beam]
             [samizdat.agent.gates :as gates]
+            [samizdat.heldout :as heldout]
             [samizdat.agent.verify :as verify]
             [samizdat.config :as config]
             [samizdat.leakage :as leakage]
@@ -1438,6 +1439,19 @@
                             :verify-timeout-ms 600000})]
     (println "\n=== SUMMARY ===")
     (clojure.pprint/pprint (summarize rows))
+    ;; THE HELD-OUT GATE, STAGE 2 (karamazov-7mo.4): the candidate arm read
+    ;; against the baseline arm, per task, floor -> cost -> guards.
+    ;; ARENA_ACCEPT=<baseline>,<candidate>; ARENA_STRUCTURAL=1 when the
+    ;; candidate adds a component, which is admissible inside the band.
+    (when-let [spec (System/getenv "ARENA_ACCEPT")]
+      (let [[b c] (map keyword (str/split spec #","))
+            fitness (gates/threshold :fitness)]
+        (println "\n=== HELD-OUT STAGE 2 ===")
+        (clojure.pprint/pprint
+         (heldout/live-verdict rows {:baseline b :candidate c
+                                     :structural? (= "1" (System/getenv "ARENA_STRUCTURAL"))
+                                     :cost-rule (:cost-rule fitness)
+                                     :noise (:noise fitness)}))))
     (println "\n=== RECURRING EDITS ===")
     (clojure.pprint/pprint (recurring-edits rows))
     ;; Opt-in, because it spends a model call per task that has both a

docs/RFCS/RFC-002-manifests-and-cells.md

@@ -57,9 +57,34 @@ glob-scoped interceptors match on.
  :constraints  [{:type :must-follow :if node :then node}]
  :subworkflows {cell-id manifest-name}     ; optional: a nested manifest as one node
  :prompt       "name"                      ; optional: prompt appended to the base
- :turn-sliceable? false}                   ; optional, default true — see below
+ :turn-sliceable? false                    ; optional, default true — see below
+ :extends      "manifest-name"             ; optional: carry only what differs from it
+ :replaces     [:invariants]               ; optional: keys that are this file's alone
+ :fragment?    true}                       ; optional: a base, not a workflow to run
 ```
 
+`:extends` names another manifest this one is built on, and the file carries
+only what differs (karamazov-xtd3). `manifests/turn.edn` holds the turn chain
+— assemble, the context-budget ladder, infer, parse, dispatch, journal,
+settle, arbiter, route — and `loop`, `worker`, `reviewer` and `supervisor`
+extend it and add their tail: the supervisor nothing, worker and reviewer a
+`:distil` on every ending, loop a `:distil` then `:finish`. They were four
+copies of one graph until then, and copies drift. `read-definition` resolves
+the link, so every reader sees the whole graph: `:cells`, `:edges` and
+`:dispatches` merge key by key with the child winning and a `nil` removing the
+base's entry; the base's `:invariants` hold and the child's are added; any
+other key is the child's where it has one; a key listed in `:replaces` is the
+child's alone, which is how a child that renames a node drops the base's
+invariants naming the old one. A base that resolves to nothing, and a chain
+that returns to itself, are refused by name. The base is a manifest role like
+any other — editable, versioned, validated — and saving it compiles every
+manifest that extends it against the candidate first, refusing, by name, a
+change that breaks one (`manifests/validate-edit!`). `manifest patch` applies
+its ops to the resolved graph and writes back only the delta
+(`manifests/extension-delta`), so the file keeps its `:extends`. `:fragment?`
+marks a base that is not itself a workflow: it stays in `catalog`, is off the
+switch menu and selection, and is not inherited.
+
 `:turn-sliceable?` declares that a manifest may **not** be a run's loop. The
 slice cuts every edge returning to `:start` into `:end`, which is the
 definition of a turn for an iterating loop and silent data loss for anything
@@ -95,7 +120,8 @@ checked (karamazov-viht.2; the same disclosure rides `policy show gates` and
 manifests/beam.edn         the ROUND    advance · score · cull · settle ·
                                         repopulate · spawn · tick · back edge
   └─ manifests/loop.edn    the TURN     assemble · infer · parse · dispatch ·
-     (per-turn slice)                   journal · settle · arbiter · route
+     (per-turn slice,                   journal · settle · arbiter · route
+      extends turn.edn)
 ```
 
 `turn-manifest` **derives** the per-turn slice from a whole-run manifest by

docs/RFCS/RFC-003-security-model.md

@@ -108,6 +108,12 @@ flowchart LR
     webfetch --> egress
     egress --> redact
 
+    heldout[held-out gate: battery tool, and cell/manifest/policy saves —
+            git archive a recorded fixture, replay it in a jolt child]
+    toolcall --> heldout
+    scrub --> heldout
+    heldout --> redact
+
     plan[plan tool: records declared paths on the branch]
     toolcall --> plan
     plan --> redact

docs/RFCS/RFC-014-validation-and-procedure.md

@@ -1,10 +1,10 @@
 # RFC-014 — Validation and procedure: measuring a change to the loop
 
 **Status:** partially implemented. The replay substrate, the battery's
-expectations, the gate inside the mutation protocol, and the procedural graph's
-mechanism are built and tested (karamazov-ylte). The battery's CASES and the
-graph itself are not; both are data, and where they come from is specified
-below.
+expectations, the held-out gate in production (stage 1 on every cell, manifest
+and policy save; stage 2 over the arena's arms — karamazov-7mo.4 / ylte.4) and
+the procedural graph's mechanism are built and tested. A project's CASES are
+its own data, added with the `battery` tool; the graph is not built.
 
 ## Purpose
 
@@ -182,6 +182,54 @@ at all* on ALFWorld — 54.48 against a 72.58 baseline, at 5.3× the tokens —
 silently serving the whole graph is the measured-worse option and the caller
 decides.
 
+### The gate in production (karamazov-7mo.4 / ylte.4)
+
+`samizdat.heldout` is what runs the battery. Every `cell save`, `manifest
+save`/`patch` and `policy save` calls `heldout/check-edit` before it stores:
+the battery is replayed with the candidate in place and without it, and an
+edit under which a target that passed fails is refused with the targets named
+(`prompts/heldout-refused.md`). Nothing is saved; what runs is unchanged.
+
+**One child process per case.** A replay must read the candidate as the
+project's userspace from every branch fiber, and a dynamic binding does not
+cross a fiber while a global override would leak the candidate into the live
+run proposing it. So the candidate is materialized: the case's fixture — the
+recorded run's `:git-baseline`, unpacked with `git archive` — gets a copy of
+the project's `.samizdat/` with the candidate file written in, and
+`samizdat.heldout.child` starts the harness there on an in-memory database,
+replays the recording through `beam/run!`, and runs `battery/check`. Side
+calls a cell makes on its own (critic, judges) were never recorded and are
+refused identically on both sides.
+
+**The rule is non-compensatory, per target.** No target that passed at
+baseline may fail on the candidate, and both sides must cover the same
+targets. Ties are accepted. A case that does not run at baseline is set aside
+by name; one that ran and no longer runs is a regression.
+
+**One replay per edit, not two.** Measurements are cached by the project and
+the CONTENT of its userspace, so the candidate's measurement is the baseline
+of the state a commit of it produces.
+
+**The battery may grow and may not be weakened — now enforced.**
+`battery_cases` (v37) is the authority: a case is written once under its id.
+A case file edited under `.samizdat/battery/` runs as stored and the edit is
+named; a deleted one still runs. The `battery` tool adds a case from a
+finished run (`heldout/draft!`, which pins its baseline under
+`refs/samizdat/battery/`) and has no remove.
+
+**Recorded beside every number.** `heldout_checks` holds one row per target
+per edit: before, after, the verdict, and the fixture sha, harness revision,
+model and scorer it was measured on.
+
+**What stage 1 does not judge.** Prompt edits: the replies are fixed. That is
+stage 2 — `heldout/live-verdict` over the arena's baseline and candidate
+arms, per task, in RRSI's order: floor (median fitness no lower than the best
+kept score less the baseline's own noise band), cost (a gain pays for its
+tokens by `:fitness :cost-rule`; inside the band only a cost cut or a declared
+structural change is admissible), guards (no acceptance criterion the baseline
+always met is lost, no suite goes red). The arena prints it with
+`ARENA_ACCEPT=<baseline>,<candidate>`.
+
 ## API
 
 | fn | contract |
@@ -243,15 +291,19 @@ is a worse failure than the one the gate prevents.
 
 ## What this does not do
 
-**The battery has no cases yet**, and until it does the gate is inert wherever
-nobody wires `battery-fn`. Cases come from two subjects, and both are required:
-recorded `endless-flight` runs and recorded runs of samizdat working on its own
-repo. A battery of one subject would pass an edit that breaks the other, and
-self-modification is the project's reason for existing.
+**A project with no cases is not gated**: the tools consult the battery on
+every save, and with no cases it passes nothing and refuses nothing.
+
+**A battery is per project.** Userspace is per project, so the cases that
+gate an edit are that project's own runs: an edit tested on endless-flight's
+cases is tested on endless-flight, and samizdat working on its own repo keeps
+its own battery in its own `.samizdat/`. Promoting a project's edit into the
+shipped templates is where both subjects have to be replayed, and that is not
+automated.
 
-**"May add, may not weaken or delete" is stated, not enforced.** The rule that
-a running agent may add a case from an observed failure and may never weaken
-one is policy with nothing behind it yet.
+**"May add, may not weaken or delete" is enforced by the table, not the
+files** (`battery_cases`); a person with the database can still delete a row,
+and that is deliberate.
 
 **No graph ships.** Building one by hand is the 58.93 row. It should be grown
 by a refiner from `Start → End` against the gate — scratch-with-evolution beat

resources/cells/oversight.clj

@@ -347,9 +347,24 @@
   instantly, so it spoke once and went quiet for the rest of the run.
 
   Concluding is not the same as having nothing left to say. A pass ends; the
-  stream does not."
-  [b]
-  (-> b (dissoc :final-answer :verdict :done? :status) (assoc :advisory? true)))
+  stream does not.
+
+  And THIS PASS'S BRIEF is appended (karamazov-3keg). Without it the resumed
+  branch woke to a conversation ending in its own accepted `done` and nothing
+  new: run bcd61b39's second pass, woken by three unmet gates, answered \"the
+  pass is complete\", and no later pass could see a round sent back, a
+  failing criterion or anything else the brief exists to carry."
+  [b brief]
+  ;; ACTIVE, not merely unfinished: state/active? is (= :active status), and
+  ;; dissoc'ing :status left a branch route read as inactive, so every
+  ;; resumed pass ran one turn and ended :abandoned (run bcd61b39, passes 2
+  ;; and 3). The last pass's ending reason and failure streaks go with it —
+  ;; a pass is not charged for how the one before it ended.
+  (cond-> (-> b
+              (dissoc :final-answer :verdict :done? :inactive-reason
+                      :consecutive-mechanics-failures :consecutive-provider-errors)
+              (assoc :status :active :advisory? true))
+    brief (update :messages (fnil conj []) {:role "user" :content brief})))
 
 (cell/defcell :oversight/reason
   {:doc "One turn of the supervisor ROLE, in the stream's OWN branch.
@@ -554,7 +569,7 @@
              ;; run-scoped resources every driver provides, and the carry is
              ;; this pass's value. Putting it in ctx would have meant claiming
              ;; the beam driver provides it, which it does not.
-             b (or (some-> (:oversight/carry data) resume-branch)
+             b (or (some-> (:oversight/carry data) (resume-branch prob))
                    (assoc (state/new-branch
                            {:id bid :problem prob
                             :messages (turn/initial-messages prob suffix :supervisor)})

resources/gates.edn

@@ -1966,6 +1966,46 @@
         judgement — it hands over the episodes that have earned the
         question."}
 
+ :heldout
+ {:value {:enabled? true
+          :dir "battery"
+          :max-cases 5
+          :case-timeout-ms 600000
+          :stage-timeout-ms 120000
+          :turns-slack 5
+          :harness-dir nil
+          :command "jolt"
+          :trail-turns 12
+          :trail-chars 300
+          :skip "\\.sqlite3(-wal|-shm|-journal)?$"}
+  :provenance ["karamazov-7mo.4" "karamazov-ylte.4" "2609.24972v2" "2609.09153v1"]
+  :kind :policy :capability-tunable? false
+  :doc "THE HELD-OUT GATE, STAGE 1 (samizdat.heldout). Every cell, manifest
+        and policy edit a tool saves is first replayed against the project's
+        frozen battery — .samizdat/<:dir>/<subject>/<case>.edn, stored in
+        battery_cases the first time it is read — with the edit in place and
+        without it, each case in a child process on a copy of the tree the
+        recorded run started from. An edit under which a target that passed
+        fails is refused and nothing is saved. Ties are accepted.
+
+        A project with no cases is not gated: it tunes itself on the compile
+        and the soak, as before. :enabled? false turns the gate off where
+        cases exist. At most :max-cases run per edit; each has
+        :case-timeout-ms, and unpacking its fixture :stage-timeout-ms. A
+        replay runs the recording's longest branch plus :turns-slack turns,
+        so a candidate that takes more turns exhausts the recording (a named
+        result) rather than being cut off. :harness-dir is the samizdat
+        checkout a child is started from; nil means the server's working
+        directory, and :command the jolt it is started with. A .samizdat file
+        whose path matches :skip (the databases) is not copied into a replay.
+        A replay hands back the last :trail-turns turns of its run, each
+        result cut at :trail-chars, since its database dies with it.
+
+        Prompt edits are not replayed: the replies are fixed, so a prompt
+        cannot change what the model says under replay. That is stage 2's —
+        the arena's live arms (heldout/live-verdict), read with :fitness
+        :noise and :cost-rule."}
+
  :pruning
  {:value {:min-runs 3 :limit 5}
   :provenance ["karamazov-na2k.10" "2609.24972v2"]