[schickling-assistant]

Stacked on #6264 (base schickling/2026-11-10-program-engine-core), which depends on #6260.

Problem

Declared program steps need an engine: a supervised host that runs program bodies, journals each inner step durably, resumes after crashes, stops promptly when the step ends, and cannot be wedged by one program's busy loop or a dead host.

Goal

M1.1b of docs/st3/program-engine.md: ProgramEngine + program_host.mjs per host protocol v1.

Decisions

• One Bun host per member; one Worker per running attempt; the main thread owns the pipe, effect process groups and liveness. Context method for linked runs is child(…) (field run is the run subject), per the contract fix.
• Module from the store blob by hash into a verified, content-addressed cache file (a damaged copy is rewritten from the blob); the host re-verifies before import. A missing blob fails the step (R06).
• Results fenced by program:STEP:ATTEMPT:KEY:result; $result journaled on done before the step completes; replay = current attempt's completed/failed entries, paged by 500.
• Eager cancellation: ProgramEngine::release_left(run) runs at the top of every evaluate_one_mission_run (before the inactive-run return) and cancels attempts whose step ended, failed, timed out, retried, was cancelled with its run, or left the generation. The journal-write check stays as backstop.
• DQ2: the host records each group (pid, boot id, leader start time) and kills it when its leader exits, on cancel, worker termination and stdin close; before any host starts the engine kills recorded groups whose leader still has the recorded start time (or is gone while the group lives) and drops earlier-boot records without a kill.
• DQ8: host pings workers (deadline default 30 s): an unresponsive worker is terminated with its groups, the host sends unresponsive, the step fails naming the deadline. The engine pings the host every 5 s and replaces one silent for 30 s; programs resume.
• Visibility API (agreed with that slice): wake(step) callback, Dispatched::HostDown(reason), host_fault(step, attempt).
• Runtime: FRACTAL_PROGRAM_RUNTIME, else bun on PATH (documented). Bun added to the dev shell and the Rust check environment so the engine tests run, never skip.
• Doc advisories fixed: §8 lists worker liveness as a program/host source; terminating a worker kills that attempt's groups.

Verification

At head 3ba3725 (merge of #6264 @9aa1096557) on the Linux build host, with the real Bun host (bun 1.3.13 from the dev shell):
CARGO_TARGET_DIR=<slice target> nix develop -c nice -n 19 cargo test -p fractal-reconcile --lib -- reconcile::program reconcile::tests::program_engine: 9 passed, 0 failed (1.62 s):

• unit: recorded_groups_are_killed_only_when_their_leader_is_verified, messages_for_another_attempt_or_host_change_nothing, a_revision_runs_the_stored_blob_it_pins, a_key_has_one_result_per_attempt, replay_returns_this_attempts_results_in_order, replay_pages_through_long_journals_in_order
• integration with a real Bun host and a hand-written bundle: a_program_resumes_from_its_journal_after_its_host_is_killed, a_step_that_leaves_its_attempt_stops_its_program_and_effects_at_once, an_unresponsive_worker_fails_only_its_own_step

Hosted CI at the exact head is the proof (pushed with ST_EVIDENCE_LOCAL=off). The #6264 tests listed in that PR passed at its head 9aa1096. python3 scripts/check-sqlite-large-values passed at this head (4285 sites).

Review findings answered

• 659853f: merges Program engine M1.1a: declare program steps and carry their bundles through apply #6264's P1 hold fix; dispatch replaces the hold here and the hold test is removed.
• 20a7701 (+ 07f6ab3, a compile fix for it):
  • P2 "Only remove the running entry the stale message belongs to": forget removes a running entry only when step, attempt and host generation still match (in stop_if_left, done, unresponsive).
  • P2 "Send ack/nack only to the host generation that sent the entry": send_to targets that generation, and only while its attempt still runs.
  • P3 "Nack a refused journal entry": every refused completed/failed entry that carries a seq (reserved $result, missing key, missing status) is nacked.
  • P2 "Don't treat a failed ps as a dead leader on macOS": a leader counts as gone only when kill(pid, 0) fails with ESRCH; a live leader whose start time cannot be read keeps its record and is not killed; macOS uses /bin/ps and /usr/sbin/sysctl by absolute path.
  • P3 "A leftover .tmp group record names a group that already started": .tmp records are verified and killed like .json records; comment corrected; test asserts a .tmp record's verified group is killed.
• 1ad7cf6, 9d7762f, 3ba3725: merge Program engine M1.1a: declare program steps and carry their bundles through apply #6264 forward (KDL-version test fix, Mission run authority: parse, render and store the authority block #6260 barrier table, mission_program barrier row, guard buffer test, main after docs: accepted program engine design and build plan #6253, corpus v19).
• 9fb2ba4: registers include_str!("program_host.mjs") in the large-value inventory (marker + record); the inventory guard failed without it (not a review finding).

Complexity

~730 lines engine + host, ~400 lines tests.

Concerns

• macOS start-time/boot-id paths (ps -o lstart=, sysctl kern.boottime) are untested here (Linux build host only).
• Relies on Bun's Worker.terminate() interrupting a busy loop (covered by the unresponsive-worker test).

Friction & bottlenecks

Build host load ~60–95 on 32 cores, ~6 GiB RAM free; cold builds serialized across slices.

Follow-ups

M1.2 children and asks; M1.3 faults/visibility; M1.4 authority (journal actor, run credential).

References

docs/st3/program-engine.md §1, §2, §5, §6, DQ2, DQ8; spike PR #6254.

[schickling-assistant]

Build-host verification at efe0519 (nix develop -c nice -n 19 cargo test -p fractal-reconcile --lib -- reconcile::program reconcile::tests::program_engine, Bun 1.3.13 from the dev shell): 9 passed, 0 failed (6 engine unit tests incl. DQ2 record verification; 3 real-Bun-host integration tests: resume after host SIGKILL, eager cancel on run cancel and step failure, unresponsive worker fails only its step). Hosted CI at the exact head is the proof.

[schickling-assistant]

Review record for head 3ba372596521e3bb20acd8143c8b25d3db51385d (stacked on #6264 @9aa1096557).

• Independent reviewer, round 1 at efe0519c77: CHANGES. Running entries could be removed by stale messages (double dispatch); acks could reach another host generation (R01); refused entries were never nacked; macOS treated a failed ps as a dead leader (fail-open kill); .tmp records were not verified.
• Round 2 at 3ba3725965 (delta read excluding Program engine M1.1a: declare program steps and carry their bundles through apply #6264 merges): GO, no findings. All six were answered: forget, send_to, nack on every refused entry, ESRCH-only gone checks with absolute /bin/ps//usr/sbin/sysctl, verified .tmp records, and the large-value marker.
• Risk medium: only program steps are affected, but this adds a supervised Bun process with SIGKILL authority over its recorded groups and changes flake inputs. Queue alone, after Program engine M1.1a: declare program steps and carry their bundles through apply #6264.
• The owner pre-approved queueing this effort's pull requests on an independent GO plus green CI.
• Verdict: GO; queue once Program engine M1.1a: declare program steps and carry their bundles through apply #6264 has merged and hosted CI is green at the head (a merge of main after Program engine M1.1a: declare program steps and carry their bundles through apply #6264 lands needs a remerge-diff delta read).