The silent-shard sweep imposes an undocumented liveness obligation on the engine

[velo]

Found while verifying the barrier fixes. Verdict-safe and self-healing — not a bug to fix now, but it must not be forgotten when the engine lands.

The obligation

Session.departSilentShards presumes dead any unreleased shard that holds no lease and has been silent for 3×5s. The 15s tolerance is calibrated to the barrier-poll cadence — but it is applied to shards in every phase, including phases where no cadence is mandated.

So the coordinator has silently imposed a contract on the engine: never be silent for more than 15s without holding a lease or being released.

Nothing documents that. The engine does not exist yet, so nothing honours it either.

The scenario, reproduced against a real container

Shard 0 waits at a barrier. Shard 1 is alive, mid-pass, in a ~20s gap between test classes — holding no lease — with a PENDING unit only it will claim.

At t+15s shard 1 is presumed dead. Shard 0 is answered DONE and released, irreversibly, while claimable work remains — the one property the release condition promises not to violate. Shard 1 then revives on its next claim, and runs and retries that unit entirely alone: zero rebalance. If shard 1 later dies for real, the work strands and the session reads INCOMPLETE.

Real triggers: a slow @AfterAll/@BeforeAll between classes, a long GC pause, a 15s network blip.

Every consequence is red-or-correct — never a false green — and a merely-partitioned shard self-heals on its next call. That is why it did not block the barrier PR.

Options when the engine is built

• Raise the tolerance for shards not at a barrier. watermark == current pass is the observable "actually waiting" predicate. Care needed: the broad sweep also covers the post-restart mid-pass ghost, so narrowing it must not reopen that.
• Document and honour the obligation engine-side — a keepalive, or claiming eagerly enough that a lease is always held across class boundaries.

Whichever is chosen, the engine and the sweep have to agree, and today only one of them knows the rule exists.

Smaller, related

LogRecord.departed does not record which kind of departure it was, so replayDeparted marks every replayed departure explicit. After a restart, an expiry- or presumption-departed shard that is actually alive comes back explicitly departed, so its barrier polls no longer revive it — only a claim will. Bounded, degraded-rebalance-only, never a wedge and never a wrong verdict. Recording the departure kind in the DEPARTED record would close it.

[velo]

Partially addressed by the engine (PR #6), and the remainder is now documented rather than silent.

The engine honours the obligation where it can: LivenessKeepalive runs a daemon thread for the whole of execute(), posting an empty claim every 5s. That grants nothing and mutates nothing beyond lastSeenAt, so it covers the lease-free gaps inside a pass — a slow @AfterAll between classes, long setup before the first result, a long-running test body. stop() interrupts and joins, and the barrier deadline path stops it before depart(), so a ping can no longer land after departure and resurrect a shard through Session.join().

What remains uncovered, and cannot be fixed engine-side: the inter-execution fork gap — JVM teardown, fork spin-up, classpath scan, discovery of the next pass. The shard genuinely is not running then, so there is nothing to send from. On a large consumer classpath that gap can exceed the 15s tolerance, producing a premature presumed death: degraded rebalance, self-healing on the shard's next call, red-or-correct and never a false green. This is now stated in the README's liveness section rather than left for someone to discover.

So the open question is unchanged in substance but narrower in scope: should the coordinator's tolerance distinguish a shard waiting at a barrier from one between forks? watermark == current pass is the observable "actually waiting" predicate. Any narrowing must not reopen the post-restart mid-pass ghost case, which the broad sweep also covers.

Also still open from the original writeup: LogRecord.departed does not record which kind of departure it was, so replay marks every departure explicit. An expiry- or presumption-departed shard that is actually alive returns from a restart explicitly departed, and only a claim revives it — not a barrier poll. Bounded, degraded-rebalance-only, never a wedge or a wrong verdict.