Keep subprocess timeouts, space supervisor passes, bound silent jobs

AGENTS.md

@@ -197,10 +197,13 @@ at 2048 tokens a turn, the vendor's sampling, and the vision projector.
 The machine-wide config (`~/.config/samizdat/config.edn`) declares it as the
 `:bonsai` provider (`:type :local`) with the knobs sized for it: `:thinking?
 true`, `:gen-floor-tps 15`, `:max-tokens 8192`, `:read-timeout-overhead-ms
-120000`, `:max-response-ms 870000`, and selects it with `:roles {:default
-:bonsai}`. A `:roles :default` in any file beats `HARNESS_PROVIDER`, so a
-project that wants GLM says `{:roles {:default :glm}}` in its own
-`.samizdat/config.edn`, as endless-flight does.
+120000`, `:max-response-ms 870000`. Since 2026-09-25 it no longer selects
+it: `:roles {:default :deepseek-flash}` makes DeepSeek flash the default, and
+Bonsai is a switch away — `/model bonsai` for a run, or `{:roles {:default
+:bonsai}}` in a project's `.samizdat/config.edn`. A `:roles :default` in any
+file beats `HARNESS_PROVIDER`, so that project file is how one project runs
+on a different model than the machine's default. endless-flight has none and
+runs on the machine's.
 
 What was measured on this M1 Max, and what it costs a run: decode is 20
 tok/s on short replies and 14-17 on long ones; prefill is 135 tok/s, so the

resources/gates.edn

@@ -297,6 +297,20 @@
         supervisor before :provider-error-limit abandoned them. False starts
         the run anyway and leaves the outage to that limit."}
 
+ :job-stall-ms
+ {:value 1800000 :kind :threshold :capability-tunable? false
+  :provenance ["karamazov-abzd" "sweep5 run 2"]
+  :doc "How long a NON-ITERATING manifest's job (feature, team, decompose —
+        the whole job is one turn, so there is no turn deadline) may go
+        without anything journalled anywhere in the run before it is
+        cancelled and its branch abandoned. A bound on silence rather than
+        length: a job may run for hours while it keeps journalling. Run
+        sweep5/2 went quiet for twenty minutes and only a rig outside the
+        harness stopped it. Twice the turn deadline, the same line the run
+        detail's stalled flag draws, so a single slow model call inside the
+        job never trips it. Absent (a project's older gates.edn), a job has
+        no bound but an abort."}
+
  :run-error-frames
  {:value 12 :kind :threshold :capability-tunable? false
   :provenance ["run ab935047" "karamazov-ma00"]
@@ -1256,7 +1270,8 @@
           :poll-ms 5000
           ;; A pass may be a model call, so the spacing is minutes and the
           ;; budget is small. The stream is meant to be the thing that
-          ;; notices, not the thing that runs.
+          ;; notices, not the thing that runs. The spacing runs from the END
+          ;; of one pass to the start of the next.
           :every-ms 120000
           :budget 12
           ;; The read timeout for the supervisor's OWN model calls, in ms;
@@ -3139,8 +3154,15 @@
   ;; of its own eval and shell calls (ship.clj observed-output) — beside any
   ;; artifacts; before that was added the rung checked a branch's figures
   ;; against its siblings' accepted answers and nothing else (karamazov-3s54).
+  ;; An advisory branch (reviewer, supervisor) delivers a verdict that quotes
+  ;; the run's own figures, so only the rungs marked :advisory? true bind it;
+  ;; `advisory?` is in scope for a rung's forms.
   {:value
    [{:name :answer-exists
+     ;; The one rung an advisory branch (reviewer, supervisor) is held to:
+     ;; its verdict is its product, and run 9ead0638's supervisor called done
+     ;; with {} and was told "Answer accepted." (karamazov-qtj5).
+     :advisory? true
      :when (str/blank? (str answer))
      ;; Shows the CALL, not just the missing name. base/missing already works
      ;; this way, on the reasoning that a bare list of argument names teaches
@@ -3150,13 +3172,21 @@
      ;; cost two live runs: both did the work, passed their tests, closed
      ;; their task, then spent their remaining turns calling `done` empty and
      ;; ended :exhausted with nothing shipped.
-     :message-form (str "`done` needs an `answer` — the text a person reads as"
-                        " the result of this run. Not a status line: say what"
-                        " you built and what shows it works.\n\n"
-                        "```tool-call\n"
-                        "{\"name\": \"done\", \"args\": {\"answer\":"
-                        " \"<what you built, and the evidence for it>\"}}\n"
-                        "```")}
+     :message-form (if advisory?
+                     (str "`done` needs an `answer` — your verdict on the run, which"
+                          " is what this branch is for: what you found, and what"
+                          " you did or want done about it.\n\n"
+                          "```tool-call\n"
+                          "{\"name\": \"done\", \"args\": {\"answer\":"
+                          " \"<your verdict, and what it rests on>\"}}\n"
+                          "```")
+                     (str "`done` needs an `answer` — the text a person reads as"
+                          " the result of this run. Not a status line: say what"
+                          " you built and what shows it works.\n\n"
+                          "```tool-call\n"
+                          "{\"name\": \"done\", \"args\": {\"answer\":"
+                          " \"<what you built, and the evidence for it>\"}}\n"
+                          "```"))}
     {:name :figure-coverage
      :when (and (seq evidence) (seq uncovered-numbers))
      ;; Says what COVERS a figure, not only that these are uncovered. Run

src/samizdat/agent/beam.clj

@@ -593,11 +593,20 @@
   The deadline is skipped for a NON-ITERATING manifest (team, feature,
   decompose): there a `turn` is the branch's entire job rather than one model
   call, so the turn deadline would abandon the run partway through the work it
-  was asked to do. Those runs stop by the abort flag, which is the stop path
-  that never depended on cooperation anyway."
+  was asked to do. What bounds such a job instead is SILENCE: gates.edn
+  :job-stall-ms of nothing journalled anywhere in the run and the job is
+  cancelled and its branch abandoned, saying so. A job may run for hours as
+  long as it keeps journalling; run sweep5/2 went quiet for twenty minutes
+  and nothing in the harness could stop it (karamazov-abzd). An abort
+  reaches a job either way: it cancels the run's task, which this wait sees."
   [ctx branches turn]
-  (let [deadline (when (get ctx :iterating-loop? true)
+  (let [iterating? (get ctx :iterating-loop? true)
+        deadline (when iterating?
                    (or (:turn-deadline-ms ctx) (turn-deadline-ms)))
+        ;; nil — no key in a project's older gates.edn — is the behaviour
+        ;; from before the bound existed.
+        stall-ms (when (and (not iterating?) (:conn ctx) (:run-id ctx))
+                   (or (:job-stall-ms ctx) (gates/threshold :job-stall-ms)))
         ;; {branch-id promise} of turns cancelled at their deadline and not yet
         ;; terminated. A cancel is observed at the turn's next check
         ;; (RFC-013); until then it is still a running turn that shares the
@@ -643,9 +652,20 @@
                         ;; Parks on the turn's SIGNAL, never on the turn task itself:
                         ;; ebb's timeout would wait for the cancelled child, and the
                         ;; barrier must not stall on a read the cancel cannot reach.
-                        (let [[tag r] (cancel/await-or-cancel t deadline)]
+                        (let [[tag r] (if stall-ms
+                                        (cancel/await-unless
+                                         t (max 1 (quot stall-ms 10))
+                                         #(runs/stalled? (:conn ctx) (:run-id ctx) stall-ms))
+                                        (cancel/await-or-cancel t deadline))]
                           (case tag
                             :ok r
+                            :stalled
+                            (do (log/warn "branch" (:id b) "has journalled nothing for"
+                                          stall-ms "ms on turn" turn "— cancelled")
+                                (when cancelling (swap! cancelling assoc (:id b) (:done t)))
+                                (assoc b :status :abandoned
+                                       :inactive-reason (str "stalled: silent "
+                                                             (quot stall-ms 1000) " s")))
                             :timeout
                             (do (log/warn "branch" (:id b) "exceeded the turn deadline on turn" turn
                                           "— cancelled")

src/samizdat/agent/oversight.clj

@@ -289,8 +289,14 @@
                                          :boundary? (if ch
                                                       (pos? (or (:turns-ended @state) 0))
                                                       true)})
-                       (swap! state assoc :last-at (now) :turns-ended 0)
-                       (pass! ctx state pass-fn)))
+                       (swap! state assoc :turns-ended 0)
+                       (pass! ctx state pass-fn)
+                       ;; Stamped at the END: the spacing is the gap between
+                       ;; passes. From the start, a pass longer than the
+                       ;; spacing — a GLM deliberation of 3-4 minutes against
+                       ;; 2 — was due again the moment it ended, and the passes
+                       ;; ran back to back through the budget (karamazov-qtj5).
+                       (swap! state assoc :last-at (now))))
                    (catch Throwable e
                      (when (cancel/control-signal? e) (throw e))
                      (log/warn "oversight loop:" (ex-message e)))))

src/samizdat/agent/tools/ship.clj

@@ -362,7 +362,9 @@
                    ~'uncovered-numbers (get ~'ctx :uncovered-numbers)
                    ;; Whether the branch's role may write: what covers a
                    ;; figure, and so what a refusal should ask for, differs.
-                   ~'can-write?        (get ~'ctx :can-write? true)]
+                   ~'can-write?        (get ~'ctx :can-write? true)
+                   ;; A reviewer or supervisor, whose answer is a verdict.
+                   ~'advisory?         (get ~'ctx :advisory? false)]
                ~form)))))
 
 (def ship-gates
@@ -384,13 +386,18 @@
 
 (defn ship-gate-block
   "The first lexical ship rung that fires on this evidence, or nil. Pure —
-  `done` computes the evidence and calls this."
+  `done` computes the evidence and calls this.
+
+  On an ADVISORY branch (`:advisory?` in the evidence) only the rungs marked
+  `:advisory? true` apply: its answer is a verdict that quotes the run's own
+  figures, so the evidence rungs do not fit it, but some — that there is an
+  answer at all — do, and which is a gates.edn decision."
   [evidence]
   (some (fn [rung]
           (when ((:when rung) evidence)
             (let [m (:message rung)]
               (if (fn? m) (m evidence) m))))
-        (ship-gates)))
+        (cond->> (ship-gates) (:advisory? evidence) (filter :advisory?))))
 
 ;; --- the give_up rungs ------------------------------------------------------
 
@@ -489,12 +496,12 @@
         borrowed (when (seq elsewhere)
                    (seq (remove (set uncovered)
                                 (uncovered-tokens answer own [problem]))))
-         block (when-not advisory?
-                 (ship-gate-block
-                  {:answer answer :problem problem
-                   :evidence evidence
-                   :uncovered-numbers uncovered-numbers
-                   :can-write? (roles/may-use? (:role branch) "write_file")}))
+         block (ship-gate-block
+                {:answer answer :problem problem
+                 :evidence evidence
+                 :uncovered-numbers uncovered-numbers
+                 :can-write? (roles/may-use? (:role branch) "write_file")
+                 :advisory? advisory?})
         ;; The test rung — what makes the loop test-driven rather than one-shot.
         ;; `done` is not terminal until the unit's tests actually pass: run the
         ;; unit's tests, and a red / hollow / untested result is fed back so the

src/samizdat/cancel.clj

@@ -128,6 +128,23 @@
       (do (cancel) [:timeout])
       r)))
 
+(defn await-unless
+  "Park on a started task's signal, asking `stalled?` every `slice-ms`.
+  Returns its [:ok v] or [:err e], or [:stalled] after asking it to stop the
+  first time `stalled?` answers true — detached, as `await-or-cancel` leaves a
+  timed-out task. The bound for work that may rightly take hours but must
+  not go quiet: a slice that ends is not a timeout, only a chance to look."
+  [{:keys [signal cancel]} slice-ms stalled?]
+  (loop []
+    (let [r (try (ebb/? (ebb/timeout signal slice-ms ::slice))
+                 (catch Throwable e
+                   (when (control-signal? e) (cancel))
+                   (throw e)))]
+      (cond
+        (not= ::slice r) r
+        (stalled?) (do (cancel) [:stalled])
+        :else (recur)))))
+
 (defn with-deadline
   "Run `task` and wait up to `ms` for it: [:ok v], [:err e], or [:timeout]
   with the task cancelled and detached. The one deadline idiom for every wait

src/samizdat/engine/proc.clj

@@ -49,6 +49,29 @@
   [pid sig]
   (try (c-kill pid sig) (catch Throwable _ -1)))
 
+(def ^:private wait-slice-ms
+  "The longest single .waitFor inside `wait-for`: short enough that jolt's
+  overshoot on one slice is a few milliseconds, long enough not to spin."
+  50)
+
+(defn- wait-for
+  "Whether `p` exited within `ms`, against the wall clock.
+
+  Not one timed .waitFor: jolt's counts its 10 ms polls rather than reading a
+  clock, so each poll's overrun accumulates — 1000 ms came back at ~1170 and
+  5000 at ~5840 on 0.8.13 (karamazov-bkjm). Short slices against a deadline
+  keep jolt's own wait, which parks a fiber and sees an interrupt, and bound
+  the overshoot to one slice's."
+  [^java.lang.Process p ms]
+  (let [deadline (+ (System/currentTimeMillis) ms)]
+    (loop []
+      (let [left (- deadline (System/currentTimeMillis))]
+        (cond
+          (not (.isAlive p)) true
+          (<= left 0) false
+          (.waitFor p (min left wait-slice-ms) java.util.concurrent.TimeUnit/MILLISECONDS) true
+          :else (recur))))))
+
 (defn- reap!
   "Kill `proc`'s whole tree and do not return until the root is gone.
 
@@ -64,17 +87,15 @@
   (let [^java.lang.Process p (:proc proc)
         root (try (.pid p) (catch Throwable _ -1))]
     (doseq [pid (cons root (descendant-pids p))] (kill! pid 15))
-    (when-not (try (.waitFor p sigterm-grace-ms java.util.concurrent.TimeUnit/MILLISECONDS)
-                   (catch Throwable _ false))
+    (when-not (try (wait-for p sigterm-grace-ms) (catch Throwable _ false))
       ;; Still alive. SIGKILL the children BEFORE the root — enumerated now,
       ;; because once the root dies they reparent to init and vanish from
       ;; descendants(). Re-enumerated here, not reused: the tree may have
       ;; spawned between the TERM and the KILL.
       (doseq [pid (descendant-pids p)] (kill! pid 9))
       (kill! root 9)
       (try (.destroyForcibly p) (catch Throwable _ nil))
-      (try (.waitFor p sigterm-grace-ms java.util.concurrent.TimeUnit/MILLISECONDS)
-           (catch Throwable _ nil)))))
+      (try (wait-for p sigterm-grace-ms) (catch Throwable _ nil)))))
 
 (defn run
   "Run `args` with `input` on stdin, capturing stdout and stderr.
@@ -84,8 +105,8 @@
   after `(get-model)` on an unsat formula even though the verdict itself was
   emitted cleanly, so the caller reads the output and decides.
 
-  The wait is `.waitFor` with an explicit timeout rather than a timed `deref`,
-  which does not work. jolt's `clojure.core/deref` forwards no opts to a record
+  The wait is `wait-for` — timed `.waitFor` slices against a deadline —
+  rather than a timed `deref`, which does not work. jolt's `clojure.core/deref` forwards no opts to a record
   implementing IBlockingDeref, so (deref proc ms ::timeout) silently calls the
   blocking one-arity and waits for however long the process takes. It fails
   quietly, in the direction of doing nothing: the timeout branch below was
@@ -109,7 +130,7 @@
         ^java.lang.Process p (:proc proc)
         ms (or timeout-ms 30000)
         finished? (try
-                    (.waitFor p ms java.util.concurrent.TimeUnit/MILLISECONDS)
+                    (wait-for p ms)
                     (catch Throwable _ false))]
     (try
       (if-not finished?

test/samizdat/base_test.clj

@@ -327,7 +327,10 @@
    {:threshold {30000 "Subprocess wall-clock bound. A capability's own safety
                        bound, not a decision about the work."
                 2000 "SIGTERM grace before SIGKILL. How long a dying process
-                      gets to die, which is a property of processes."}}
+                      gets to die, which is a property of processes."
+                50 "The slice wait-for polls in, which bounds how far past
+                    its deadline a wait can run. How precisely a timeout is
+                    kept, not how long it is."}}
 
    "src/samizdat/events.clj"
    {:threshold {256 "The event tap's sliding-buffer size. RFC-009 states it as

test/samizdat/beam_cancel_test.clj

@@ -15,6 +15,7 @@
             [samizdat.api.control :as api-control]
             [samizdat.cancel :as cancel]
             [samizdat.store.db :as db]
+            [samizdat.store.journal :as journal]
             [samizdat.store.runs :as runs]))
 
 (defn- now [] (System/currentTimeMillis))
@@ -36,6 +37,46 @@
 (defn- ctx [deadline-ms]
   {:iterating-loop? true :turn-deadline-ms deadline-ms :cancelling (atom {})})
 
+;; --- a job that goes quiet (karamazov-abzd) ---------------------------------
+;;
+;; A non-iterating manifest (feature, team, decompose) runs the whole job as
+;; one turn, so the turn deadline is skipped — and with it every bound: run
+;; sweep5/2 journalled its last event and then sat silent for twenty minutes
+;; until a rig outside the harness killed it. The bound is on SILENCE, not on
+;; length: a job may run for hours as long as the run keeps journalling.
+
+(defn- job-ctx [conn rid stall-ms]
+  {:iterating-loop? false :job-stall-ms stall-ms :cancelling (atom {})
+   :conn conn :run-id rid})
+
+(deftest a-job-that-goes-silent-is-cut-off-and-says-why
+  (let [c (db/open! ":memory:")
+        rid (runs/start-run! c {:problem "p"})
+        b (state/new-branch {:id "B1" :problem "p"})]
+    (try
+      (with-redefs [beam/advance-branch (fn [_ b _] (ebb/? (ebb/sleep 60000)) b)]
+        (let [t0 (now)
+              [r] (beam/advance-all (job-ctx c rid 300) [b] 1)]
+          (is (< (- (now) t0) 3000) "cut off after the silence, not after the job")
+          (is (= :abandoned (:status r)))
+          (is (re-find #"(?i)silent|stall" (str (:inactive-reason r))) (str (:inactive-reason r)))))
+      (finally (db/close c)))))
+
+(deftest a-job-that-keeps-journalling-is-left-to-run
+  (let [c (db/open! ":memory:")
+        rid (runs/start-run! c {:problem "p"})
+        b (state/new-branch {:id "B1" :problem "p"})]
+    (try
+      (with-redefs [beam/advance-branch (fn [_ b _]
+                                          (dotimes [i 8]
+                                            (ebb/? (ebb/sleep 100))
+                                            (journal/note! c rid :step {:data {:i i}}))
+                                          (assoc b :finished true))]
+        (let [[r] (beam/advance-all (job-ctx c rid 300) [b] 1)]
+          (is (:finished r) "eight hundred ms of work under a 300 ms silence bound")
+          (is (not= :abandoned (:status r)))))
+      (finally (db/close c)))))
+
 (deftest a-turn-past-the-deadline-is-cancelled-and-the-branch-forfeits-until-it-stops
   ;; The turn parks (a provider call would); the deadline cancels it; the park
   ;; sees the cancel at once, so by the next round the branch is free again.