Skip to content

fix: recover native first-fsync lock crashes before TTL expiry - #105

Open
unbraind wants to merge 15 commits into
mainfrom
fix/crash-restart-recovery
Open

unbraind wants to merge 15 commits into
mainfrom
fix/crash-restart-recovery

Conversation

@unbraind

@unbraind unbraind commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

The native first-fsync update/comment crashes found by #104 preserve acknowledged data but previously refused restart with exit 4. This PR builds on #104 and must merge after #104; its base remains main.

The integration round pulled the remote head, confirmed no unresolved review threads and no substantive CodeRabbit review body, and merged current main a600bd8 plus the completed #104 branch at 328ff009483d754b3e25412580495378025d5d87. Both merges were clean. Main contributes its audited tracker reconciliation. The harness contributes three typed-refusal fixes and four regressions; the recovery implementation and its strict acceptance assertions remain unchanged.

The recovery holds the permanent kernel lease, confirms original-process death on the same Linux host/namespace, and admits identity-checked empty-gate reclamation before TTL expiry. Its durable recovery intent still precedes lock removal. Live or unverified owners, occupied gates and replacement identities keep their refusal policy. Item/history serialization and transaction recovery are unchanged.

Before: native update/comment killed at fsync ordinal 1 restarted with exit 4. After: both recover to the exact pre-mutation restart reference, including item/history bytes and raw receipt. The full Linux corpus requires every observed boundary to pass, and permanent independent replay manifests cover both first-fsync reproductions. The non-git concurrency contract retains 50 simultaneous writers and all 150 comments/history links.

Local merged-head verification:

  • Current native binary build passes.
  • Strict erasable TypeScript, ESLint, Rust formatting, changelog generation/check and isolated pinned-PM health pass.
  • Pinned-PM linked node --test scripts/crash-harness.test.mts passes 20/20 with zero failures or skips, including both retained replays and all four merged refusal regressions. It completed within its unchanged stored 300-second budget.
  • Original history streams, fixtures, dependency pins and coverage thresholds are unchanged. No new review finding or recovery-code change requires a new negative control in this integration round.
  • The full Linux release gate at integrated code head 46178e2 passed: 580 ordinary and 580 instrumented Rust tests, zero failed or ignored; lines 14030/14030, functions 1467/1467, regions 23449/23449 and branches 3298/3298. Strict TypeScript, ESLint, the Linux crash harness (20/20, zero skips), audit, changelog and strict tracker health also passed. macOS full release CI passed as well. Final receipt-only head 53e4988177a2298f28f14e49406768484ec4cd0e has all nine CI jobs passing, including all three full release gates. Its downloaded Linux log confirms 580 ordinary and 580 instrumented Rust tests, zero failed/ignored; all four required Rust metrics are 100 percent, and the crash harness passes 20/20 with zero skips. Source files are unchanged from the validated integrated head. Earlier incomplete local cold builds are not used as full-gate evidence. The existing allowed RUSTSEC-2024-0449 advisory remains.

Retained original evidence:

Process-kill evidence does not establish power-loss durability, unmeasured syscall coverage or cross-runtime journal recovery. Each runtime uses its own raw restart reference; cross-runtime acknowledgement uses the shared projection declared by #104. Separate existing comment lock-conflict diagnostic differences remain on pm-rust-comment-lock-receipt.

pm-rust-crash-lock-restart remains in progress with its claim released for orchestrator verification. Receipt-only changelog and isolated pinned-PM health checks pass; final-head CI receipts are linked above. No review replies or resolutions, PR merge or release.

Summary by Sourcery

Recover native mutations from verified first-fsync lock crashes while adding comprehensive Linux crash-conformance validation.

New Features:

  • Add a Linux crash-consistency harness that injects process kills at observed rename and fsync boundaries and compares native behavior with the published CLI.
  • Expose standalone crash-corpus execution and replay commands with bounded reports, typed refusals, and privacy-safe retained evidence.

Bug Fixes:

  • Recover native update and comment operations after first-fsync lock-initialization crashes before TTL expiry when the original owner is verified dead, while preserving refusal behavior for unsafe or unverified ownership states.

Enhancements:

  • Strengthen crash recovery validation to require exact item and history preservation, restart receipt matching, and independent replay coverage for update and comment scenarios.
  • Increase the concurrency contract to exercise 50 simultaneous writers and verify all accepted comments and history links.

CI:

  • Install strace on Linux CI and include the crash harness in the release validation gate.

Documentation:

  • Document the crash-consistency corpus, its measured boundaries, replay workflow, evidence handling, and durability limitations.

Tests:

  • Add crash harness unit, refusal, syscall-injection, full-corpus, and retained first-fsync recovery tests.
  • Add regression coverage for live, unknown, occupied, nonempty, and replacement lock gates.

Chores:

  • Add crash harness fixtures, immutable receipts, replay proofs, and project tracking records.

Updated harness stack integration

After PR 104 head f555a4238fc4d0fa5768feaee0e81dce66031202 passed all nine exact-head CI jobs, this branch merged that completed update under the original stacked-PR instruction. Current main a600bd8 was already an ancestor. The merge was clean and brings the correct baseline-parity replay guide/docstring distinction, two documentation regressions and LF/CRLF checks. PR 105 still has zero review threads and no substantive CodeRabbit review.

All Rust source is byte-identical to previously certified recovery head 53e4988; both permanent first-fsync reproductions and the strict every-boundary recovery assertions are intact. Current native build, strict erasable TypeScript, ESLint and required check-only tracker health pass. The existing pinned-PM linked node --test scripts/crash-harness.test.mts passes 22/22, zero failed/canceled/skipped, in about 16 seconds inside its unchanged 300-second budget. All 78 pre-existing history streams retain their original bytes as prefixes. There is no new runtime fix requiring another recovery revert; the original two-fail/two-restored-pass recovery proof and PR 104 documentation controls remain linked and unchanged.

Final integration head ca52b3608505542519781d6709799c163d9e01de includes the merge and append-only pm-rust-crash-lock-restart records. The item remains in progress with its claim released. The final exact-head CI run passes all nine jobs, including unchanged just release-check on Linux, macOS and Windows. Downloaded logs confirm ordinary/instrumented Rust tests: Linux 580/580, macOS 568/568 and Windows 562/562, zero failed or ignored. All four required Rust coverage metrics are exactly 100 percent on every platform with unchanged scope and thresholds. Linux counts: lines 14030/14030, functions 1467/1467, regions 23449/23449 and branches 3298/3298. The final Linux crash suite passes 22/22 with zero skips; macOS passes 13 with nine declared Linux-only skips and Windows passes 12 with ten declared platform skips. Strict TypeScript, ESLint, audit, changelog and required check-only tracker health pass. The existing allowed RUSTSEC-2024-0449 advisory and separately tracked comment lock-diagnostic mismatch remain. No TypeScript coverage percentage or power-loss durability is claimed. Earlier receipts above remain historical evidence. No review replies/resolutions, PR merge or release.

@sourcery-ai

sourcery-ai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Sorry @unbraind, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 5 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 71fb670a-1fc7-4ff5-b2cc-19feb6fa8492
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sourcery-ai

sourcery-ai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR adds narrowly scoped Linux recovery for first-fsync lock crashes, backed by an extensive isolated syscall-injection harness that compares exact durable state and restart receipts, plus CI, documentation, fixtures, and 50-writer concurrency validation.

Sequence diagram for first-fsync lock crash recovery

sequenceDiagram
    participant Mutation
    participant Lock as Lock manager
    participant Gate as Publication gate
    participant Kernel as Linux kernel lease
    participant Owner as Original process
    participant Recovery as Durable recovery

    Mutation->>Lock: acquire_lock_attempt()
    Lock->>Gate: write complete lock payload
    Lock->>Gate: fsync()
    Kernel-->>Lock: SIGKILL during first fsync
    Mutation--xOwner: process dies
    Recovery->>Kernel: hold permanent lease
    Recovery->>Owner: confirmed_dead_owner()
    Recovery->>Gate: read_lock_owner()
    alt identity matches and gate is empty
        Recovery->>Recovery: write recovery intent receipt
        Recovery->>Gate: reclaim_stale_gate()
        Recovery-->>Mutation: restart succeeds before TTL expiry
    else live, unknown, replaced, or occupied gate
        Recovery-->>Mutation: LockConflict
    end
Loading

Flow diagram for syscall-injection crash corpus validation

flowchart TD
    A[Create acknowledged synthetic seed] --> B[Run update or comment reference]
    B --> C[Observe task-local rename and fsync ordinals]
    C --> D[Inject SIGKILL at each declared boundary]
    D --> E[Restart the same operation family]
    E --> F{State and receipt match reference?}
    F -->|yes| G[Verify exact item/history bytes]
    F -->|no| H[Retain bounded synthetic failure evidence]
    G --> I{Native and published parity?}
    I -->|yes| J[Pass corpus boundary]
    I -->|no| H
Loading

File-Level Changes

Change Details Files
Permit safe pre-TTL recovery of crashes occurring during first fsync of the lock payload.
  • Under the existing kernel lease, recognize a fresh empty publication gate as reclaimable only when the original owner is identity-verified dead on the same Linux host/namespace.
  • Preserve durable recovery-intent recording before lock removal and retain refusals for live, unknown, replaced, occupied, or nonempty gates.
  • Add focused unit and parity coverage for owner identity, gate occupancy, lease contention, recovery receipts, and refusal-byte preservation.
src/mutation.rs
tests/support/mutation_unit.rs
tests/parity_1021_contract.rs
Add a syscall-injection crash-conformance harness with permanent first-fsync replays.
  • Run native and pinned published CLI implementations through isolated, no-Git workspaces while injecting SIGKILL at observed per-task rename/fsync boundaries.
  • Compare exact item/history snapshots, acknowledged projections, restart state, and runtime-specific raw receipt hashes; retain privacy-safe synthetic failure artifacts and controlled replay manifests.
  • Add typed CLI refusal handling, source immutability and symlink/path safety checks, bounded execution, deterministic clock setup, storage snapshot utilities, and harness unit/integration tests.
scripts/crash-harness.mts
scripts/crash-cli.mts
scripts/crash-clock.mts
scripts/crash-process.mts
scripts/crash-storage.mts
scripts/crash-harness.test.mts
tests/fixtures/crash-restart/update.json
tests/fixtures/crash-restart/comment.json
tests/fixtures/crash-restart/fixture/.agents/pm/settings.json
tests/harness/package.json
tests/harness/package-lock.json
tests/harness/tsconfig.json
tests/harness/eslint.config.mjs
docs/CRASH_HARNESS.md
docs/CRASH_PUBLIC_COPY_2026_10_22.json
docs/CRASH_RESTART_RECOVERY_2026_10_22.json
docs/CRASH_RESTART_REVERT_PROOF_2026_10_22.json
docs/CRASH_REVERT_PROOF_2026_10_22.json
Integrate crash recovery validation and stronger concurrency coverage into project workflows.
  • Install strace on Linux CI and make crash-harness checks part of release-check.
  • Document crash corpus scope, commands, evidence, limitations, and fixture replay procedures.
  • Synchronize 50 concurrent comment writers with a barrier and verify all 150 comments/history entries survive.
  • Update ignore/task/history metadata for the crash-restart and comment lock-receipt work.
.github/workflows/ci.yml
justfile
README.md
tests/concurrency_contract.rs
.gitignore
.agents/pm/features/pm-rust-crash-harness.toon
.agents/pm/history/pm-rust-comment-lock-receipt.jsonl
.agents/pm/history/pm-rust-crash-harness.jsonl
.agents/pm/history/pm-rust-crash-lock-restart.jsonl
.agents/pm/tasks/pm-rust-comment-lock-receipt.toon
.agents/pm/tasks/pm-rust-crash-lock-restart.toon

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant