docs(specs): add background work complexity gate

[yokoszn]

PR 5450 Staging Description

Summary

Review-only staging PR for the background-work reliability specifications from upstream PR open-mercato#5450. This branch adds a narrowly scoped complexity and verification gate to the existing spec decisions.

The intended contract is not absolute O(1). For a fixed repair batch B, unrelated live jobs must not increase repair work, rows read, or memory beyond index-depth effects. Actual due work costs O(B) per tick and drains across ticks.

Comparison lanes

These are four separate states. Results must never be described as if they compare the wrong pair.

Lane	Meaning	Code or documents	Status
A. Current baseline	Open Mercato before PR open-mercato#5450. develop is the normal contribution baseline. main is the release baseline.	open-mercato/open-mercato:develop and main	Existing behavior
B. PR open-mercato#5450	Current upstream reliability documentation and spec series. It describes the proposed leased-job design but does not implement repair cells or the upgraded runtime.	open-mercato/open-mercato#5450, head 60e0bac3b7	Open, changes requested
C. This staging branch	Fork-only documentation follow-up adding the complexity gate and measurement contract. No runtime implementation.	TWN-Systems/open-mercato, docs/pr5450-review-candidate	Review staging
D. Upgraded implementation	Fork implementation branch with repair cells, leases, data-sync worker wiring, and local queue recovery work.	TWN-Systems/open-mercato#9, feat/pr5450-repair-cells	Partially validated; runtime gates remain explicit

The benchmark below is not a measurement of lane D or production Open Mercato. It is a disposable PostgreSQL model comparing a faithful sweep-shaped query with an indexed due-cell query. Lane D must reproduce this comparison inside Open Mercato using real progress/data-sync tables and workers.

Table of contents

• Scope
• Comparison lanes
• Model before and after
• Database model evidence
• Redis and Valkey evidence
• Correctness and safety
• FMEA and failure conditions
• Validation commands
• Open gates

Scope

• Three spec files changed.
• No runtime code, migration, generated file, enterprise file, credential, or fork-only infrastructure file changed.
• The candidate branch is based on review/pr5450-base, pinned to upstream PR docs(specs): background-work reliability series v5 — problems, requirements, staged decision, phase-1 specs open-mercato/open-mercato#5450 head 60e0bac3b7.
• This is a staging branch for review. It does not claim that the implementation has landed.

Model before and after

This is a model comparison, not a claim that lane C changed runtime behavior. The model baseline represents lane A's sweep-shaped behavior. The model candidate represents the intended lane D repair-cell behavior. Lanes B and C are documentation/spec states.

The model baseline is the faithful wide-row live-job predicate. The model candidate is an indexed due-cell lookup with a fixed batch of 300.

Live rows	Due rows	Baseline p50/p95/p99 ms	Candidate p50/p95/p99 ms	Baseline plan	Candidate plan	Rows selected	Result
10,000	300	0.164 / 0.182 / 0.182	0.725 / 0.769 / 0.769	sequential scan	index scan, zero sequential scans	300	exact due-count match
100,000	300	0.164 / 0.327 / 0.327	0.614 / 0.664 / 0.664	sequential scan	index scan, zero sequential scans	300	exact due-count match
1,000,000	300	102.767 / 120.112 / 120.112	0.590 / 0.762 / 0.762	sequential scan	index scan, zero sequential scans	300	exact due-count match

At 1M live rows, the model candidate is approximately 157.6x faster at p99. The small-fixture model candidate is slower because index and sort overhead dominate. That is reported rather than hidden. No speedup is attributed to lane B or lane C.

The JS oracle reports identical selected IDs and changes inspected rows from live count to 300 due cells. It is an algorithmic shape check, not a production claim.

Database model evidence

PostgreSQL 17.11 was run in the Open Mercato compose service with disposable benchmark tables. The benchmark records EXPLAIN (ANALYZE, BUFFERS) execution time, p50/p95/p99, rows, shared hits/reads, sequential scans, and index scans. This is evidence for the shape of lane D, not evidence that lane D exists.

ACID checks pass:

• rollback preserves both domain and repair state;
• commit moves both together;
• replay is idempotent;
• wrong-tenant mutation changes zero rows;
• orphan repair cells fail the foreign key;
• duplicate repair insertion is rejected.

Redis and Valkey evidence

The compose Redis service was started and smoke-tested:

image: redis:7-alpine
reported version: 7.4.11
PING: PONG
SET with NX: OK
GET: 1
DEL: 1

This confirms service reachability and basic key semantics only. It does not claim worker recovery after broker loss, eviction safety, or at-least-once delivery under a real Open Mercato worker. Those remain explicit integration gates.

Correctness and safety

The regression suite passes due-cell selection shape, atomic state movement, durable retry attempt state, tenant scope, duplicate prevention, monotone backlog drain, foreign-key ownership, and basic Redis reachability.

Every implementation PR must additionally test worker kill/restart, concurrent redrive, domain-row reopen, operator audit, and both queue strategies (local and async) inside Open Mercato module integration tests.

FMEA and failure conditions

The SRE/FMEA material is reader context only. It gives the failure vocabulary and mental model behind the performance and deliverability questions. It is not a request to add an Open Mercato-wide FMEA framework, new repository process files, or external operational tooling.

The concrete review risks remain local to this spec: wide reconciliation scans, unbounded due work, lost repair coordinates, duplicate transitions, retry-budget loss, redrive/domain divergence, and tenant-scope errors. The implementation PR must test those risks in the affected modules. Worker death/restart, Redis/Valkey loss, operator audit, and rollback are follow-up operational scenarios, not claims that this documentation-only PR has implemented them.

The failure rule is fail-closed: a faster result with missed repair, duplicate repair, scope leakage, lost attempts, or missing operator evidence is a failure.

Validation commands

Private modeling and evidence commands:

npm run check
npm run test:acid
npm run test:fmea
npm run check:fmea
npm run bench:repair-cells -- 10000 100000 1000000
npm run bench:postgres
lake build

Open Mercato validation required before an implementation PR:

yarn build:packages
yarn generate
yarn build:packages
yarn i18n:check-sync
yarn i18n:check-usage
yarn typecheck
yarn test
yarn build:app
yarn workspace @open-mercato/queue test
yarn workspace @open-mercato/queue build
yarn test:integration

Current implementation evidence

PR #9 currently reports the following fork validation:

• Queue local strategy: 35/35 tests passed, including directory recreation recovery.
• Full queue package: 104/104 tests passed in approximately 8.6s with --forceExit.
• Queue package typecheck and build passed.
• git diff --check passed.
• The normal Jest process still reports an open-handle condition after the suite; this is recorded as an open validation item rather than hidden by the force-exit result.
• The host had stale Jest/Yarn processes and approximately 18 GB of Cargo incremental artifacts; those were cleaned before rerunning.

These results belong to lane D/PR #9, not this documentation-only PR.

Open gates

This staging PR is documentation-only. Its purpose is to make the complexity and deliverability gate reviewable. PR #9 is the current lane D implementation candidate. It must add only the repo-native tests and code required by the accepted specs, run the configured validation sequence, pass GitHub required checks, and receive maintainer review. The SRE/FMEA context does not require repository-level additions in this PR.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[yokoszn]

Evidence added to the PR description

The description now includes a table of contents, before/after measurements, database plan evidence, Redis/Valkey smoke evidence, ACID results, FMEA coverage, validation commands, and explicit open gates.

Key measured result with a fixed batch of 300:

Live rows	Baseline p99	Candidate p99	Candidate plan
10,000	0.182 ms	0.769 ms	index scan, zero sequential scans
100,000	0.327 ms	0.664 ms	index scan, zero sequential scans
1,000,000	120.112 ms	0.762 ms	index scan, zero sequential scans

The 1M-row candidate result is approximately 157.6x faster at p99. The smaller fixtures expose index overhead and are reported as such.

PostgreSQL ACID checks pass for rollback, atomic commit, idempotent replay, tenant scope, foreign-key ownership, and duplicate prevention. Redis 7.4.11 smoke checks pass for PING, SET NX, GET, and DEL.

These are evidence gates, not claims that the runtime implementation is complete. Worker kill/restart, broker-loss recovery, domain-aware redrive, both queue strategies, connection-pool soak, operator audit, and rollback drills remain open and are listed in the description.

[yokoszn]

Clarification: the SRE/FMEA material in this description is non-normative reader context. It is included to make failure modes and deliverability concerns easier to reason about. It does not propose adding a repository-wide FMEA framework, new process files, or external operational tooling. The only contribution obligations are the affected Open Mercato specs and, in a later implementation PR, the minimal repo-native code and tests required by the accepted design.

[yokoszn]

Clarification for reviewers: the evidence has four lanes. A = current Open Mercato baseline (develop, with main as release baseline). B = upstream PR open-mercato#5450 at 60e0bac3b7, documentation/spec only. C = this fork staging branch, documentation-only. D = the future upgraded runtime implementation, not created or measured yet. The before/after numbers compare only the baseline-shaped sweep query with the intended indexed repair-cell query in disposable PostgreSQL tables. They are not being attributed to PR open-mercato#5450 or this staging branch.

[yokoszn]

The docs/spec lane has now been integrated with PR 9 onto the PR open-mercato#5450 base in integration PR 10.