From f93c94c589e87383994cb69c4d4f32840d7ba43c Mon Sep 17 00:00:00 2001 From: Clay Townsend Date: Sun, 23 Aug 2026 16:14:31 +1000 Subject: [PATCH 1/2] docs(specs): add background work complexity gate --- .../2026-08-21-background-work-04-solution.md | 24 +++++++++++++++++++ ...kground-work-06-leased-jobs-in-progress.md | 8 ++++--- ...1-background-work-07-data-sync-adoption.md | 1 + 3 files changed, 30 insertions(+), 3 deletions(-) diff --git a/.ai/specs/2026-08-21-background-work-04-solution.md b/.ai/specs/2026-08-21-background-work-04-solution.md index 2291427e8f9..57196218cfb 100644 --- a/.ai/specs/2026-08-21-background-work-04-solution.md +++ b/.ai/specs/2026-08-21-background-work-04-solution.md @@ -196,6 +196,30 @@ What the pattern implies elsewhere in the repo, as **named follow-up specs**: None of these is needed for the leased tier or for `data_sync`; all of them reuse the same two ideas, which is why they are listed here rather than designed. +## 📝 Complexity model and benchmark gate + +The leased tier must keep the unit of work addressable by key; it must not move the failure from "one long job" to "one repairer that scans every job". The model is: + +| Path | Required shape | Bound | +|---|---|---| +| Enqueue / re-enqueue | deterministic `queueJobId = pj---`; broker dedup is a keyed lookup | O(log queue) or broker-native O(1) | +| Claim | primary-key lookup plus `(seq, redrives)` and lease predicate | O(log jobs), no backlog scan | +| Single-runner check | partial unique index on `(lock_key, tenant, org)` for live leased rows | O(log live_keys) | +| Heartbeat | one HOT update on the row; no indexed heartbeat/lease columns | O(1) row update, no index churn | +| Q2/Q5 repair | indexed pending/mirror cells (`pending_since`, `finished_at`) | O(due + log jobs) | +| Q1 orphan detection | bounded scan of live leased rows because `lease_expires_at` is intentionally unindexed | O(live_leased) until the measured ceiling below | + +Q1 is the only intentional scan. It is allowed for phase 1 only with a measured ceiling: the implementation PR must show the reconciler tick stays below **100 ms p95 at 10k live leased rows with 100 expired rows** on the integration Postgres, and below **1 s p95 at 100k live leased rows** or else add a narrow repair-cell side table before merge. Do **not** index `lease_expires_at` on `progress_jobs` to satisfy this; that would turn every heartbeat into index churn and violate invariant 6. The side-table fallback is a separate cell index owned by the repairer, not another authoritative clock. + +Benchmark command expected from the implementation PR (exact script name may change, but the dimensions may not): + +```bash +spbench run "yarn mercato progress bench-leases --live 10000 --expired 100 --pending 100 --mirror-pending 100" --binary open-mercato-progress-reconciler +spbench run "yarn mercato progress bench-leases --live 100000 --expired 100 --pending 100 --mirror-pending 100" --binary open-mercato-progress-reconciler +``` + +The acceptance report must include tick p50/p95, database queries per tick, rows scanned, rows repaired, and `pg_stat_user_tables.n_tup_hot_upd` for `progress_jobs` during the heartbeat phase. A speed number without the repaired-row count is not a result. + ## 📋 Implementation specs (replaces the single phasing table of v3) | Part | Spec | Owner surface | Depends on | Ships alone? | Closes | Approval state | diff --git a/.ai/specs/2026-08-21-background-work-06-leased-jobs-in-progress.md b/.ai/specs/2026-08-21-background-work-06-leased-jobs-in-progress.md index fc15c599d5b..7c7f00b8dad 100644 --- a/.ai/specs/2026-08-21-background-work-06-leased-jobs-in-progress.md +++ b/.ai/specs/2026-08-21-background-work-06-leased-jobs-in-progress.md @@ -244,7 +244,7 @@ flowchart TB Every reconciler transition that ends a row (Q1 park, Q3 cancel) is a §7 terminal transaction: the CAS and the domain mirror commit together, and a throwing mirror rolls the CAS back so the row is selected again next tick. Q5 is the only path in which the mirror runs *after* a terminal CAS committed, and it exists for two cases only: kinds that declared `mirror: 'deferred'`, and rows whose terminal CAS committed before the column existed (migration backfill sets `domain_mirrored_at = finished_at` for rows without `subject_type`, and leaves it null where a subject exists so Q5 re-mirrors them once). -Constraint: `pendingTtlMs` must exceed the worst-case queue backlog for that kind, otherwise Q2 re-drives healthy queued deliveries (they are refused on `redrives` and cost nothing but a wasted delivery, yet each one counts toward `never_started`); the default 15 min is per kind and the reconciler logs a warning when a Q2 re-drive finds the previous delivery still queued (via `Queue.getJobState` where the capability exists). Cost note: Q1 scans the `running ∧ leased` partial index and filters `lease_expires_at` in the heap — a bounded scan of live leased rows every 30 s, which stays cheap to ~10k live rows; beyond that the follow-up is a narrow `progress_job_leases` side table, not an index on `lease_expires_at` (invariant 6). +Constraint: `pendingTtlMs` must exceed the worst-case queue backlog for that kind, otherwise Q2 re-drives healthy queued deliveries (they are refused on `redrives` and cost nothing but a wasted delivery, yet each one counts toward `never_started`); the default 15 min is per kind and the reconciler logs a warning when a Q2 re-drive finds the previous delivery still queued (via `Queue.getJobState` where the capability exists). Cost note: Q1 scans the `running ∧ leased` partial index and filters `lease_expires_at` in the heap — a bounded scan of live leased rows every 30 s. This is the only intentional O(live) path in the design; all other repair cells are indexed by their due class. It is acceptable for phase 1 only if the part 4 benchmark gate is met (100 ms p95 at 10k live rows / 100 expired, and 1 s p95 at 100k live rows). If it misses, the implementation must add the narrow `progress_job_leases` repair-cell side table before shipping, rather than indexing `lease_expires_at` on `progress_jobs` and breaking HOT heartbeats (invariant 6). **Connection ceiling** (`packages/queue/AGENTS.md` → Connection Budget): a leased slice uses its request-scoped EM plus one *transient* pooled connection during each heartbeat/lease statement (≤ 1 query every 20 s per slice), and the reconciler worker uses one connection per batch query. The worst case per worker process is therefore `Σconcurrency + 1` concurrent connections instead of `Σconcurrency`; the existing DB-budget clamp in `mercato worker --all` is updated to reserve that one extra connection and the integration test asserts `pg_stat_activity` stays under the clamp during the soak. @@ -335,6 +335,7 @@ Real Postgres + Redis (`__integration__`, docker-compose runner), with a test ki 9. Reconciler vs live driver → take fences the driver at its next heartbeat; at most one in-flight page applied twice. 10. Connection ceiling: `pg_stat_activity` stays under the `mercato worker --all` clamp during a 3-replica soak. 11. Retention never deletes a terminal row with `domain_mirrored_at is null` and a subject. +12. Reconciler complexity benchmark: a synthetic leased-job fixture with 10k live rows / 100 expired rows and 100k live rows / 100 expired rows runs through the repo-native benchmark target and, when `spbench` is available, through the part 4 `spbench run` commands. The report includes p50/p95 tick time, rows scanned, rows repaired, query count, and `n_tup_hot_upd` growth during heartbeats. ## 📋 Implementation Plan @@ -344,8 +345,9 @@ Real Postgres + Redis (`__integration__`, docker-compose runner), with a test ki 4. Kind registry (`mirror` option, `maxMirrorAttempts`) + `runSlice` factory + generic worker binding. Tests: refused claim does no work; budget → yield → rewritten delivery claims; throw → `failSlice` + rethrow → retry claims; unrecoverable → terminal via §7; shutdown → interrupted yield with counters untouched; mirror throw on `drained` → slice fails, retry completes. 5. Reconciler worker (`progress-reconcile` repeatable every 30 s via `upsertRepeatable`, registered at worker boot) + `progress-retention`; Q1–Q5; `SKIP LOCKED` batches; per-row isolation; stale sweep moved here; `OM_PROGRESS_SWEEP_ON_READ` (default on). Tests: orphan take bumps epoch and re-drives with backoff; Q1 waits for `next_run_at`; poison park on `redrives_since_commit`; lost hand-back re-drive; Q3 cancels a dead slice with the mirror in the same commit; Q5 mirrors a deferred row and a backfilled legacy row; a throwing park mirror leaves the row for the next tick and counts; two reconcilers on one set; tracked-tier sweep parity; repeatable survives a failed run and a Redis flush + boot; retention skips unmirrored rows. 6. Leased cancel semantics + minimal cascade (§8); `POST /api/progress/jobs/[id]/redrive` (ACL `progress.update` + kind features); top bar "cancelling"/"parked"; OpenAPI. -7. Integration coverage above. -8. AGENTS.md (`progress` incl. the repairer carve-out and the `onTransition` idempotency rule, root Task Router row), docs page "leased jobs", `BACKWARD_COMPATIBILITY.md` and UPGRADE_NOTES entries, `.ai/lessons.md`. +7. Repo-native `progress bench-leases` command (or equivalent test-only CLI) that seeds synthetic leased rows and prints the part 4 benchmark columns; run it directly and through `spbench` where available. +8. Integration coverage above. +9. AGENTS.md (`progress` incl. the repairer carve-out and the `onTransition` idempotency rule, root Task Router row), docs page "leased jobs", `BACKWARD_COMPATIBILITY.md` and UPGRADE_NOTES entries, `.ai/lessons.md`. ## Open items for review diff --git a/.ai/specs/2026-08-21-background-work-07-data-sync-adoption.md b/.ai/specs/2026-08-21-background-work-07-data-sync-adoption.md index 939a0c85731..3522c79e65b 100644 --- a/.ai/specs/2026-08-21-background-work-07-data-sync-adoption.md +++ b/.ai/specs/2026-08-21-background-work-07-data-sync-adoption.md @@ -91,6 +91,7 @@ Adapter contract: **signature unchanged**. Three `data_sync/AGENTS.md` Ask-First 4. Cancel during adapter I/O → adapter receives `signal` → run `cancelled` and mirrored in one commit. 5. Existing `data_sync` integration specs green. 6. **Soak**: docker-compose with 3 worker replicas, a fake adapter producing 500 batches, a supervisor SIGKILLing one replica per minute and flushing Redis once; assert every `(run_id, batch_no)` appears exactly once in a test ledger the adapter writes inside the fenced commit, `sync_runs.batches_completed = 500`, `sync_runs.status = 'completed'` with `finished_at` set, and no row is left `running`/`pending`/unmirrored after the reconciler's next tick. +7. **Performance/complexity gate**: the same fake adapter runs under a backlog fixture with 10k and 100k unrelated live leased jobs. The data_sync run must still complete with exactly-once ledger rows, and the progress reconciler must meet the part 4 p95 targets. Report quality (`500/500` unique committed batches), speed (slice throughput and reconciler p95), and baseline (today's one-job data_sync path or the previous implementation PR run on the same fixture). ## 📋 Implementation Plan From 4953dc9a5d1b9f2c1f1c6e3714e7516e029c452d Mon Sep 17 00:00:00 2001 From: Clay Date: Mon, 24 Aug 2026 18:07:58 +1000 Subject: [PATCH 2/2] Remove benchmark command examples from specs Removed internal benchmark command examples from the implementation PR requirements. --- .ai/specs/2026-08-21-background-work-04-solution.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.ai/specs/2026-08-21-background-work-04-solution.md b/.ai/specs/2026-08-21-background-work-04-solution.md index 57196218cfb..9fb26a95128 100644 --- a/.ai/specs/2026-08-21-background-work-04-solution.md +++ b/.ai/specs/2026-08-21-background-work-04-solution.md @@ -211,13 +211,6 @@ The leased tier must keep the unit of work addressable by key; it must not move Q1 is the only intentional scan. It is allowed for phase 1 only with a measured ceiling: the implementation PR must show the reconciler tick stays below **100 ms p95 at 10k live leased rows with 100 expired rows** on the integration Postgres, and below **1 s p95 at 100k live leased rows** or else add a narrow repair-cell side table before merge. Do **not** index `lease_expires_at` on `progress_jobs` to satisfy this; that would turn every heartbeat into index churn and violate invariant 6. The side-table fallback is a separate cell index owned by the repairer, not another authoritative clock. -Benchmark command expected from the implementation PR (exact script name may change, but the dimensions may not): - -```bash -spbench run "yarn mercato progress bench-leases --live 10000 --expired 100 --pending 100 --mirror-pending 100" --binary open-mercato-progress-reconciler -spbench run "yarn mercato progress bench-leases --live 100000 --expired 100 --pending 100 --mirror-pending 100" --binary open-mercato-progress-reconciler -``` - The acceptance report must include tick p50/p95, database queries per tick, rows scanned, rows repaired, and `pg_stat_user_tables.n_tup_hot_upd` for `progress_jobs` during the heartbeat phase. A speed number without the repaired-row count is not a result. ## 📋 Implementation specs (replaces the single phasing table of v3)