[schickling-assistant]

Problem

A program step that stops journaling never raises anything, and a host that cannot start only logs a line (design §8, R11, DQ8).

Goal

• program/stalled: a running program step whose journal has no new entry within its stalled-after raises a fault on the step naming its open key; closed on progress or when the step stops running.
• program/host: raised from the engine's host_fault(step, attempt): a host that cannot start (Dispatched::HostDown), cannot resume after an exit, or a worker that missed its liveness deadline.
• Both closed by the inactive-fault sweep when the run ends.

Decisions

• Faults are on the step subject (one per program step, as "on every program step it held" requires), not the run subject, which would collide when a run has several program steps.
• reconcile_program_faults runs once per program step in every run evaluation, right after the step's own evaluation, so a dispatch that just hit HostDown is seen in the same pass.
• Stall deadline = max(last journal entry of the attempt, execution start) + stalled-after (default 30 minutes), armed with smallclaims::touched::note_due; journal entries are claims on the step, which its run's evaluation reads, so an entry moves the deadline without polling. The reason names no age, so a lasting stall is recorded once.
• A HostDown step keeps its state and is dispatched again at a 30 s deadline (HOST_RETRY_MS); while program/host is open the step does not also raise program/stalled.
• The engine's wake(step) (in Program engine M1.1b: run program steps on a supervised Bun host #6265) re-evaluates the run when the engine fails a step for liveness or cannot resume it.

Verification

On the build host, on an integration with #6261 and #6262, with real Bun:

• cargo test -p fractal-reconcile --lib program_faults: 2 passed.
  • a_quiet_journal_raises_program_stalled_at_its_deadline_and_an_entry_closes_it: the due time equals the last entry plus stalled-after. At the deadline the fault names the open key and is recorded once; a new entry closes it and moves the due time; leaving working closes it.
  • a_host_that_cannot_start_raises_program_host_until_the_run_ends: a host that cannot start raises program/host with the engine's reason. The step keeps working with no stall, and the sweep closes the fault after the run ends.
• cargo test -p fractal-reconcile --lib program_engine: 3 passed. The engine's real-host tests still pass with the fault hook.
• python3 scripts/check-sqlite-large-values: passed.
  Hosted CI at the exact head is the proof.

Complexity

About 280 changed lines of its own (tests and docs included), plus #6261 merged in.

Concerns

• A host fault lives in engine memory: after a daemon restart it is gone and the fault closes on the next evaluation (the restart retries the host anyway).
• A step-subject fault does not set mission_run in attention items (attention_snapshot); run-filtered fault views do not list it yet.

Friction & bottlenecks

Build host memory: cold builds serialized across the program engine slices.

Follow-ups

Link step-subject faults to their run in attention items.

References

docs/st3/program-engine.md §8, DQ8; stacked on #6265 and merges #6261.