Repository navigation
workspace: add the spate-faults crate with its journal format - #998
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Six things inline: five tests that leave a rule unpinned, and one doc line. One more is in the description.
Description: "A run ends in one of four kinds" lists the four failing kinds, but Kind (faults/src/outcome.rs:8-20) has five, including Pass. "A failing run ends in one of four kinds" says what the code does.
e004662 to
2cffdb6
Compare
There was a problem hiding this comment.
Four things inline, all in faults/src/, plus one in the description.
Description. Missed in an earlier round: "A seed replays the fault schedule. OS scheduling and real time decide the interleaving, including which split a worker holds when a fault lands." reads as shipped here. This change has no fault schedule: outside seed.rs, nothing under faults/ uses SplitMix64. The rest of the description uses "will" for later behaviour, e.g. "Later changes will draw fault schedules from a seed through the inline SplitMix64 added here".
Minor
faults/src/seed.rs:32: Missed in an earlier round: "slightly biased" does not hold for spans above 2^63. There2^64 % spanleaves the lowest2^64 - spanoffsets twice as likely as the rest. I ranin_range(0, 0xAAAA_AAAA_AAAA_AAAA)1,000,000 times from seed 42: 0.6669 of the draws fell in the lower half, where a uniform draw gives 0.5000. Nothing in this change callsin_rangewith a span that large./// A value in `[low, high]`, biased when the span is not a power of two, /// negligibly only for spans far below 2^64.faults/src/outcome.rs:23: Missed in an earlier round: the doc says this is a failing scenario's prefix, but line 27 also returns one forKind::Pass, which line 5 says does not fail the scenario. Either the doc covers every kind ("The prefix a scenario's result message starts with."), orPassgets no prefix.
2cffdb6 to
1b5be62
Compare
1b5be62 to
0cd2310
Compare
0cd2310 to
67bd757
Compare
What this changes, and why
Part of #836.
No test checks the delivery contract across separately running workers under faults. This is the first of eleven changes that add such a run. It fixes the journal format every worker and the harness will write, and the outcome kinds a run reports, as tested code before anything writes them.
spate-faultsis an unpublished workspace member besidebench/andtest-support/, outsidecrates/, so CI's container selection never picks it up and it has no semver surface. Its manifest joinsis_manifest, so a change to it runs the gates that read declared floors.Each worker will write one append-only journal: the rows its sink made durable, every durable
split.*write it sent with its reply (including a write cancelled by a timeout), and everysplit.*entry it read. Acallnumber pairs eachsendwith itsdone, so a reply never has to be matched by order. The harness writes the faults it injected, with their times, in the same format. The reader drops a last line cut short by a kill, and fails on a progress record at a schema other than 3, so a record schema change breaks the harness loudly.A failing run ends in one of four kinds: a delivery violation, a worker failure, a scenario whose own expectation failed, or an infrastructure failure.
outcome::classifyis a pure function over the run's evidence, and its tests pin the rule order, including that a property-3 or property-5 violation stays a violation through a container outage outside a broken-fence scenario. Only a violation will ever be reported as a delivery problem.Classifiernames what asplit.*write does (claim, commit, complete, release, fail report, quarantine, renew) from the value at its expected revision. Later changes will draw fault schedules from a seed through the inline SplitMix64 added here, pinned to the reference outputs by a unit test. A seed will replay the fault schedule. OS scheduling and real time will decide the interleaving, including which split a worker holds when a fault lands.Implemented by Claude Opus 5.5, working as an agent.
Invariants
None. No published crate changes.
Semver
Checks
cargo xtask ciarea: descriptionfor one area, no AI attribution trailerschangelog.d/if this touches what a crate ships, or a line readingChangelog: nonein this body.changelog.d/README.mdsays whenChangelog: none.
spate-faultsis unpublished and nothing published changes.Anything else
The diff is about 1,520 lines, most of it rustdoc on the journal's line fields and the outcome tests. Each test was checked against a mutant of the rule it pins (a reader that parses every line, a skipped schema check, a container rule that covers every violation, a single failed poll counted as an outage, and so on); every mutant failed its test.