Skip to content

workspace: add the spate-faults crate with its journal format - #998

Merged
MarcusKainth merged 1 commit into
mainfrom
workspace/836-fault-journal
Oct 7, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
workspace/836-fault-journal

Conversation

@MarcusKainth

@MarcusKainth MarcusKainth commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What this changes, and why

Part of #836.

No test checks the delivery contract across separately running workers under faults. This is the first of eleven changes that add such a run. It fixes the journal format every worker and the harness will write, and the outcome kinds a run reports, as tested code before anything writes them.

spate-faults is an unpublished workspace member beside bench/ and test-support/, outside crates/, so CI's container selection never picks it up and it has no semver surface. Its manifest joins is_manifest, so a change to it runs the gates that read declared floors.

Each worker will write one append-only journal: the rows its sink made durable, every durable split.* write it sent with its reply (including a write cancelled by a timeout), and every split.* entry it read. A call number pairs each send with its done, so a reply never has to be matched by order. The harness writes the faults it injected, with their times, in the same format. The reader drops a last line cut short by a kill, and fails on a progress record at a schema other than 3, so a record schema change breaks the harness loudly.

A failing run ends in one of four kinds: a delivery violation, a worker failure, a scenario whose own expectation failed, or an infrastructure failure. outcome::classify is a pure function over the run's evidence, and its tests pin the rule order, including that a property-3 or property-5 violation stays a violation through a container outage outside a broken-fence scenario. Only a violation will ever be reported as a delivery problem.

Classifier names what a split.* write does (claim, commit, complete, release, fail report, quarantine, renew) from the value at its expected revision. Later changes will draw fault schedules from a seed through the inline SplitMix64 added here, pinned to the reference outputs by a unit test. A seed will replay the fault schedule. OS scheduling and real time will decide the interleaving, including which split a worker holds when a fault lands.

Implemented by Claude Opus 5.5, working as an agent.

Invariants

None. No published crate changes.

Semver

  • Additive — nothing existing changes

Checks

  • cargo xtask ci
  • Title is area: description for one area, no AI attribution trailers
  • A fragment under changelog.d/ if this touches what a crate ships, or a line reading Changelog: none in this body. changelog.d/README.md says when

Changelog: none. spate-faults is unpublished and nothing published changes.

Anything else

The diff is about 1,520 lines, most of it rustdoc on the journal's line fields and the outcome tests. Each test was checked against a mutant of the rule it pins (a reader that parses every line, a skipped schema check, a container rule that covers every violation, a single failed poll counted as an outage, and so on); every mutant failed its test.

@github-actions github-actions Bot added area: workspace The root manifest, the lockfile, and cross-crate plumbing area: ci Workflows, actions, and the scripts they run labels Oct 7, 2026
@codecov

codecov Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.10984% with 17 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
faults/src/outcome.rs 93.4% 9 Missing ⚠️
faults/src/journal.rs 94.4% 8 Missing ⚠️

📢 Thoughts on this report? Let us know!

@spate-review spate-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Six things inline: five tests that leave a rule unpinned, and one doc line. One more is in the description.

Description: "A run ends in one of four kinds" lists the four failing kinds, but Kind (faults/src/outcome.rs:8-20) has five, including Pass. "A failing run ends in one of four kinds" says what the code does.

Comment thread faults/src/outcome.rs
Comment thread faults/src/outcome.rs
Comment thread faults/src/journal.rs
Comment thread faults/src/classify.rs
Comment thread faults/src/seed.rs
Comment thread faults/src/journal.rs Outdated
@MarcusKainth
MarcusKainth force-pushed the workspace/836-fault-journal branch from e004662 to 2cffdb6 Compare October 7, 2026 12:52

@spate-review spate-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four things inline, all in faults/src/, plus one in the description.

Description. Missed in an earlier round: "A seed replays the fault schedule. OS scheduling and real time decide the interleaving, including which split a worker holds when a fault lands." reads as shipped here. This change has no fault schedule: outside seed.rs, nothing under faults/ uses SplitMix64. The rest of the description uses "will" for later behaviour, e.g. "Later changes will draw fault schedules from a seed through the inline SplitMix64 added here".

Minor
  • faults/src/seed.rs:32: Missed in an earlier round: "slightly biased" does not hold for spans above 2^63. There 2^64 % span leaves the lowest 2^64 - span offsets twice as likely as the rest. I ran in_range(0, 0xAAAA_AAAA_AAAA_AAAA) 1,000,000 times from seed 42: 0.6669 of the draws fell in the lower half, where a uniform draw gives 0.5000. Nothing in this change calls in_range with a span that large.
        /// A value in `[low, high]`, biased when the span is not a power of two,
        /// negligibly only for spans far below 2^64.
    
  • faults/src/outcome.rs:23: Missed in an earlier round: the doc says this is a failing scenario's prefix, but line 27 also returns one for Kind::Pass, which line 5 says does not fail the scenario. Either the doc covers every kind ("The prefix a scenario's result message starts with."), or Pass gets no prefix.

Comment thread faults/src/outcome.rs
Comment thread faults/src/classify.rs
Comment thread faults/src/classify.rs Outdated
Comment thread faults/src/classify.rs Outdated
@MarcusKainth
MarcusKainth force-pushed the workspace/836-fault-journal branch from 2cffdb6 to 1b5be62 Compare October 7, 2026 13:16

@spate-review spate-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things inline, both in the classify.rs tests.

Comment thread faults/src/classify.rs Outdated
Comment thread faults/src/classify.rs Outdated
@MarcusKainth
MarcusKainth force-pushed the workspace/836-fault-journal branch from 1b5be62 to 0cd2310 Compare October 7, 2026 13:28

@spate-review spate-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing inline, in the classify.rs tests.

Comment thread faults/src/classify.rs
@MarcusKainth
MarcusKainth force-pushed the workspace/836-fault-journal branch from 0cd2310 to 67bd757 Compare October 7, 2026 13:42
@MarcusKainth
MarcusKainth enabled auto-merge October 7, 2026 13:43
@MarcusKainth
MarcusKainth added this pull request to the merge queue Oct 7, 2026
Merged via the queue into main with commit 1610487 Oct 7, 2026
42 checks passed
@MarcusKainth
MarcusKainth deleted the workspace/836-fault-journal branch October 7, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci Workflows, actions, and the scripts they run area: workspace The root manifest, the lockfile, and cross-crate plumbing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant