Concurrent restore finalizers fail with a non-retryable InvalidPhase when a peer advances more than one phase #515

Symptom

executor::tests::completed_fork_concurrent_finalize_replays_exact_manifest_publication (four exact FinalizeRestore requests released through a barrier) fails intermittently on the nokv-workspace CI job with

every exact concurrent finalizer must converge: RpcFailure { code: PreconditionFailed,
  message: "restore is DestinationSealing, expected SourceSealed", retryable: false,
  conflict: Some(OperationState), ... }

(#514, run 35423726254). On clean main 7bb3ffd the same test fails locally under CPU oversubscription (8 parallel copies × 150 iterations) with the sibling message "restore is Complete, expected DestinationSealing"; 300 iterations without contention never reproduce it, which is why it surfaces on shared runners rather than on a developer machine.

Root cause (architecture)

Restore finalization is a multi-step, idempotent driver over a monotonic phase lineage SourceSealed → DestinationBuilding → DestinationSealing → Ready → Complete. Concurrent exact finalizers are a supported pattern: they are expected to converge on the same durable result. Convergence is implemented inside each nokv-meta step function as "the operation is already in the phase this step produces, so return the durable record as a replay":

• apply_restore_initialization recognises DestinationBuilding only;
• build_restore_commit_members recognises DestinationSealing only;
• seal_restore_commit_revisions recognises Ready only;
• complete_restore requires exactly Ready, and the executor patches the one-phase lead with completed_restore_replay.

The recognition is therefore one phase deep. A finalizer that loads phase P and then loses the race by two or more phases hits require_phase → RestoreError::InvalidPhase, which restore_failure maps to a non-retryable PreconditionFailed. The finalizer fails for work that has already finished, and the executor's own multi-phase acceptance (restore_step_advances already accepts any monotonic lead) is never reached. Two sanity checks in the executor (initialized.phase == DestinationBuilding, phase != Ready → terminal failure) also reject a legitimately further-advanced operation.

Fix

One rule in nokv-meta: a finalize step converges on any forward phase at or beyond the phase it produces (restore_phase_reached), never on preparation, abort, cleanup or quarantine phases. The executor accepts any monotonic forward jump after each step and returns the durable terminal row when a step converged on Complete. A feature-gated (restore-crash-test-support) load observer lets a test hold one finalizer after its load while a peer completes the restore, so the race is covered by a deterministic test rather than by scheduling luck; the concurrent stress test stays as it is.

The fix is independent of #514 (which only adds a publish fence); #514 will re-run against main once this lands.