test: expand concurrency validation methodology

CHANGELOG.md

@@ -7,6 +7,10 @@ All notable changes to this project. The format is loosely based on
 
 ### Added
 
+- M33: deterministic pipeline scheduling perturbation (`PipelinePerturbation`) so concurrency tests
+  exercise different input/engine/output pacing patterns without timing-sensitive sleeps.
+- M33: `make concurrency-stress` / `scripts/concurrency_stress.sh`, an opt-in repeated
+  concurrency-test loop for longer local or Linux/TSan runs outside normal CI.
 - M32: `OrderBook::Storage::{Baseline,Pooled}` and `MatchingEngine(OrderBook::Storage)` for a
   scoped PMR-backed order-book node-allocation experiment. `Storage::Pooled` routes per-book
   `std::list`, `std::map`, and `std::unordered_map` node allocation through
@@ -26,6 +30,9 @@ All notable changes to this project. The format is loosely based on
 
 ### Documentation
 
+- M33: concurrency docs now distinguish static happens-before reasoning, TSan, deterministic
+  schedule perturbation, and repeated stress as evidence over executed schedules rather than proof
+  over all interleavings.
 - M32: added `docs/pool_backed_storage.md` and ADR 0009 to distinguish M28 raw-object pool
   mechanics, M32 PMR container-node allocation, and the future intrusive/custom-node order-book
   redesign. Added `results/pool_backed_storage.txt` as the measured engine-level artifact.

CLAUDE.md

@@ -247,7 +247,13 @@ Keep this synchronized with the Makefile.
 - `make fmt` — apply clang-format
 - `make tidy` — clang-tidy target if available
 - `make bench` — run benchmark suite
+- `make bench-diff` — run differential harness benchmarks
+- `make bench-allocator` — run M28 allocator experiment
+- `make bench-storage` — run M32 storage experiment
 - `make asan` — build/run sanitizer preset
+- `make tsan` — build/run ThreadSanitizer concurrency tests
+- `make concurrency-stress` — opt-in repeated concurrency validation loop
+- `make socket-stress` — UDP socket-buffer / burst-loss experiment
 - `make clean` — remove build artifacts
 
 Run `make check` before every PR.

HANDOFF.md

@@ -14,8 +14,10 @@ Keep all four in the repo root.
 ---
 ## Current handoff
 
-The repo is released at `v0.1.0`. M0–M31 are merged. Most recently M31 = PR #93 (squash commit
-b7926ac): external-review request package plus issue #94. **M32 is the active milestone.**
+The repo is released at `v0.1.0`. M0–M32 are merged. Most recently M32 = PR #96 (squash commit
+f122ee8): PMR-backed order-book node allocation experiment, engine-level storage benchmark, and
+issue #95 for future intrusive/custom-node `OrderPool<Capacity>` storage. **M33 is the active
+milestone; draft PR #97 is open, CI is green, and Codex review found no major issues.**
 
 Background — M29 delivered (merged, constrained-environment only):
 
@@ -28,7 +30,7 @@ Background — M29 delivered (merged, constrained-environment only):
 - The repository does **not** currently claim real hardware PMU evidence.
 - Issue #90 tracks full PMU-backed evidence generation on a bare-metal or PMU-capable Linux target.
 
-M32 is in progress on `feat/m32-pool-backed-order-book-storage`. To resume it:
+M33 is in progress on `feat/m33-advanced-concurrency-validation`. To resume it:
 
 ```text
 /resume
@@ -55,23 +57,18 @@ gh release view v0.1.0
 
 Current state:
 
-- main tip: `b7926ac` (PR #93, M31)
-- active branch: `feat/m32-pool-backed-order-book-storage` (M32 in progress)
+- main tip: `f122ee8` (PR #96, M32)
+- active branch: `feat/m33-advanced-concurrency-validation` (M33 draft PR #97 open; CI/Codex clean)
 - release tag: `v0.1.0`
 - open follow-up issue: #90 for full Linux hardware PMU perf evidence
 - open follow-up issue: #95 for future intrusive/custom-node `OrderPool<Capacity>` order-book
   storage
 
 ### Next milestone
 
-M32 — Pool-backed order-book storage experiment. Use the PMR design:
-`OrderBook::Storage::{Baseline,Pooled}`, `std::pmr::unsynchronized_pool_resource`, PMR-backed
-`std::list`/`std::map`/`std::unordered_map` node allocation, and
-`MatchingEngine(OrderBook::Storage)`.
-
-Do not force direct M28 `OrderPool<Capacity>` into the current order book. `std::list<Order>`
-allocates list nodes, not bare `engine::Order` objects; direct `OrderPool` integration requires an
-intrusive/custom-node redesign and is tracked in issue #95.
+M33 — Advanced concurrency validation. Add non-flaky scheduling perturbation or longer stress
+modes, document that TSan/stress are empirical evidence rather than proof, and keep long-running or
+Linux-only validation as explicit opt-in commands rather than normal CI requirements.
 
 ### Phase III / IV purpose
 
@@ -84,9 +81,9 @@ networking research.
 Current priority order:
 
 1. Issue #90 — real Linux hardware PMU perf evidence.
-2. M32 — PMR-backed order-book node allocation (in progress).
-3. M33 — advanced concurrency validation.
-4. M34 — epoll gateway architecture.
+2. M33 — advanced concurrency validation (in progress).
+3. M34 — epoll gateway architecture.
+4. M35 — multi-client socket pressure.
 
 ### Forbidden shortcuts
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: configure build test check fmt fmt-check tidy bench bench-diff bench-allocator bench-storage perf-stat perf-record profile-io socket-stress asan tsan demo check-fixtures check-manifest determinism divergence-demo clean
+.PHONY: configure build test check fmt fmt-check tidy bench bench-diff bench-allocator bench-storage perf-stat perf-record profile-io socket-stress concurrency-stress asan tsan demo check-fixtures check-manifest determinism divergence-demo clean
 
 BUILD_DIR := build/dev
 
@@ -68,6 +68,9 @@ profile-io:
 socket-stress: build
 	bash scripts/socket_stress.sh
 
+concurrency-stress:
+	bash scripts/concurrency_stress.sh
+
 asan:
 	cmake --preset asan
 	cmake --build --preset asan

PROGRESS.md

@@ -19,15 +19,15 @@ Do not rely on prior chat memory.
 
 ## Current state
 
-- **Active milestone:** M32 — Pool-backed order-book storage experiment (in progress)
-- **Status:** draft PR open (#96); local verification passed
-- **Active branch:** `feat/m32-pool-backed-order-book-storage`
-- **Last completed milestone:** M31 — External review / maintainer signal (squash-merged, PR #93, commit b7926ac; Codex auto-review clean — reacted 👍, no findings); external review request opened as issue #94 (labels backlog/documentation/help wanted)
+- **Active milestone:** M33 — Advanced concurrency validation (in progress)
+- **Status:** draft PR open (#97); local verification passed; CI green; Codex review clean
+- **Active branch:** `feat/m33-advanced-concurrency-validation`
+- **Last completed milestone:** M32 — Pool-backed order-book storage experiment (squash-merged, PR #96, commit f122ee8; Codex auto-review clean — no major issues; CI green)
 - **Release:** `v0.1.0` published as a GitHub release (tag on commit 9857e1a); no packages published
-- **`make check` passing:** last verified on M32 (188/188) on 2026-06-02; `make asan` also passed (188/188).
-- **Last action:** opened draft PR #96 for M32 after implementing PMR-backed order-book node allocation, opening issue #95 for future intrusive/custom-node `OrderPool<Capacity>` storage, adding an engine-level `make bench-storage` benchmark and `results/pool_backed_storage.txt`, and verifying `make check` / `make asan` / `make bench-storage`.
-- **Next action:** monitor PR #96 CI and Codex review; do not merge.
-- **Blockers:** none for M32. Issue #90 (full hardware PMU evidence) remains blocked on PMU-capable Linux access; issue #95 tracks future direct intrusive/custom-node storage and is not part of M32.
+- **`make check` passing:** last verified on M33 (189/189) on 2026-06-02; `make asan` also passed 189/189, `make tsan` passed 19/19 concurrency tests, and a two-loop `make concurrency-stress` smoke passed.
+- **Last action:** PR #97 CI passed all 6 jobs and Codex review found no major issues.
+- **Next action:** human review/squash-merge PR #97; do not merge from the automation side.
+- **Blockers:** none for M33. Issue #90 (full hardware PMU evidence) remains blocked on PMU-capable Linux access; issue #95 tracks future direct intrusive/custom-node storage and is not part of M33.
 
 ---
 
@@ -202,7 +202,7 @@ compiler-, and build-dependent — these are from one machine, not a production-
 
 > If stopping mid-milestone, write exactly what is half-done and the precise next step. Clear this when the milestone merges.
 
-- _M32 draft PR #96 is open on `feat/m32-pool-backed-order-book-storage`. Implemented `OrderBook::Storage::Baseline` and `Storage::Pooled`, PMR-backed list/map/unordered_map node allocation, and `MatchingEngine(OrderBook::Storage)` propagation. Added the generated-flow equivalence test (per-command event streams, final snapshot, last_seq, non-vacuity), `make bench-storage`, `results/pool_backed_storage.txt`, `docs/pool_backed_storage.md`, ADR 0009, and issue #95 for future direct intrusive/custom-node `OrderPool<Capacity>` storage. Verification passed: `make check` 188/188, `make asan` 188/188, `make bench-storage`. Next: monitor CI/Codex review; do not merge. Clear this block when M32 merges._
+- _M33 draft PR #97 is open on `feat/m33-advanced-concurrency-validation`. Implemented deterministic `PipelinePerturbation` hooks/tests, `make concurrency-stress`, and concurrency-methodology docs. Local verification: `make check` 189/189, `make asan` 189/189, `make tsan` 19/19 concurrency tests, `QSL_CONCURRENCY_STRESS_LOOPS=2 make concurrency-stress` 2/2 loops, `bash -n scripts/concurrency_stress.sh`, and `git diff --check`. PR #97 CI passed all 6 jobs and Codex review found no major issues. Next: human review/squash-merge; do not merge. Clear this block when M33 merges._
 
 
 ---
@@ -256,8 +256,8 @@ Lower priority:
 | M29 | Linux perf profiling workflow and artifacts | `feat/m29-linux-perf-profiling` | ☑ merged | #89 | perf workflow + constrained validation; full PMU evidence backlogged in #90 |
 | M30 | Kernel/socket path profiling and Linux socket hardening | `feat/m30-socket-profiling-hardening` | ☑ merged | #92 | syscall/socket-buffer/UDP pressure evidence; epoll deferred to M34/M35 |
 | M31 | External review / maintainer signal | `docs/m31-external-review` | ☑ merged | #93 | Review-request checklist + feedback template; review request opened as issue #94 |
-| M32 | Pool-backed order-book storage experiment | `feat/m32-pool-backed-order-book-storage` | ◐ draft PR | #96 | PMR-backed node allocation in order-book paths; direct intrusive `OrderPool` storage deferred to #95 |
-| M33 | Advanced concurrency validation | `feat/m33-advanced-concurrency-validation` | ☐ not started | — | Scheduling perturbation, longer stress, and stronger concurrency methodology |
+| M32 | Pool-backed order-book storage experiment | `feat/m32-pool-backed-order-book-storage` | ☑ merged | #96 | PMR-backed node allocation in order-book paths; direct intrusive `OrderPool` storage deferred to #95 |
+| M33 | Advanced concurrency validation | `feat/m33-advanced-concurrency-validation` | ◐ draft PR | #97 | Scheduling perturbation, longer stress, and stronger concurrency methodology |
 | M34 | epoll gateway architecture | `feat/m34-epoll-gateway-architecture` | ☐ not started | — | Event-driven multi-client gateway design |
 | M35 | Multi-client load and socket-pressure testing | `feat/m35-multi-client-socket-pressure` | ☐ not started | — | TCP/UDP stress, buffer pressure, backpressure investigation |
 | M36 | NUMA awareness study | `feat/m36-numa-awareness-study` | ☐ not started | — | CPU affinity and NUMA locality measurements where hardware exists |
@@ -269,6 +269,13 @@ Lower priority:
 
 ## Decision log additions
 
+- [2026-06-02] M33: started after M32 (#96) squash-merged (commit f122ee8). Scope: advanced concurrency validation only — deterministic scheduling perturbation and/or longer stress modes, stronger concurrency methodology docs, opt-in long-running/Linux checks where appropriate. Do not claim proof; TSan and stress tests remain dynamic evidence over executed schedules.
+- [2026-06-02] M33: added deterministic `PipelinePerturbation` yield hooks to the threaded pipeline and a regression test that compares perturbed pipeline output against the single-threaded reference across seeded property flows, queue capacities, and per-stage yield patterns.
+- [2026-06-02] M33: added `make concurrency-stress` / `scripts/concurrency_stress.sh` as an opt-in repeated concurrency-label test loop. Normal CI remains non-flaky; longer local/Linux runs are documented through explicit knobs rather than hidden in the default gate.
+- [2026-06-02] M33: local verification passed: `make check` 189/189, `make asan` 189/189, `make tsan` 19/19 concurrency tests, `QSL_CONCURRENCY_STRESS_LOOPS=2 make concurrency-stress` 2/2 loops, `bash -n scripts/concurrency_stress.sh`, and `git diff --check`.
+- [2026-06-02] M33: opened draft PR #97 and triggered `@codex review`; do not merge from the automation side.
+- [2026-06-02] M33: PR #97 CI passed all 6 jobs (`build-test`, `sanitizers`, `thread-sanitizer`, `determinism`, `differential-sweep`, `ocaml-verifier`) and Codex review found no major issues.
+- [2026-06-02] M32: PR #96 squash-merged (commit f122ee8); Codex review found no major issues and CI passed all jobs. M32 delivered PMR-backed order-book node allocation, an engine-level storage benchmark, docs/ADR, and issue #95 for future intrusive/custom-node `OrderPool<Capacity>` storage.
 - [2026-06-02] M32: started after M31 (#93) squash-merged (commit b7926ac). Corrected scope: integrate pool-backed order-book node allocation using PMR, informed by the M28 allocator experiment, and measure baseline-vs-pool-backed engine-level workloads — not another allocator microbenchmark. Direct `OrderPool<Capacity>` order-book integration is deferred because the current `std::list<Order>` design allocates implementation-defined list nodes, not bare `engine::Order` objects; intrusive/custom-node storage is tracked separately in issue #95. Matching must remain deterministic (replay/differential tests stay green); no storage-architecture claim beyond what the committed measured artifact supports; document even a negative result.
 - [2026-06-02] M32: implemented the scoped PMR path: `OrderBook::Storage::{Baseline,Pooled}`, per-book `std::pmr::unsynchronized_pool_resource`, PMR list/map/unordered_map node allocation, and `MatchingEngine(OrderBook::Storage)` propagation via `books_.try_emplace(id, book_storage_)`. Baseline remains the default.
 - [2026-06-02] M32: added a generated-flow equivalence invariant test proving baseline and pooled storage produce identical per-command event streams, final `EngineSnapshot`, and `last_seq`; the test includes non-vacuity guards for real trades and resting liquidity.
@@ -338,17 +345,17 @@ Quant Systems Lab — Linux Systems + Exchange Infrastructure Simulator
 
 ## Next action remains
 
-Continue Phase III/IV after M29 review:
+Current action is M33 on `feat/m33-advanced-concurrency-validation`; see the top-level current state
+block for the exact next step.
+
+After M33 squash-merges, resume with:
 
 ```text
-/start-milestone 30
+/start-milestone 34
 ```
 
-Expected branch after PR #89 squash-merges: `feat/m30-socket-profiling-hardening`.
-
-Priority before/around M30: issue #90 remains the evidence debt for full Linux hardware PMU
-artifacts. Work it only on a PMU-capable Linux host; do not relabel constrained Docker artifacts as
-full evidence.
+Issue #90 remains the evidence debt for full Linux hardware PMU artifacts. Work it only on a
+PMU-capable Linux host; do not relabel constrained Docker artifacts as full evidence.
 
 After each squash merge, return to this file and update state factually. If benchmark numbers are not measured, write `not measured`. Do not guess. Nobody is impressed by imaginary throughput.
 

README.md

@@ -58,11 +58,12 @@ make demo      # end-to-end local demo (see below)
 ```
 
 Other targets: `make check` (format-check + build + test), `make fmt`, `make asan`
-(AddressSanitizer + UBSan), `make bench` / `make bench-diff` (build the bench preset and write
-`results/latest.txt` / `results/differential.txt`), `make check-fixtures` (regenerate the
-differential fixtures and verify they match current C++ output), `make check-manifest` (verify
-the fixture provenance manifest), and `make determinism` (assert fixtures are byte-identical
-across compilers).
+(AddressSanitizer + UBSan), `make tsan` (ThreadSanitizer over concurrency-labelled tests),
+`make concurrency-stress` (opt-in repeated concurrency validation), `make bench` /
+`make bench-diff` / `make bench-storage` (committed benchmark harnesses), `make check-fixtures`
+(regenerate the differential fixtures and verify they match current C++ output),
+`make check-manifest` (verify the fixture provenance manifest), and `make determinism` (assert
+fixtures are byte-identical across compilers).
 
 ## Demo
 
@@ -106,7 +107,9 @@ the core numbers above.
 
 - **Synthetic and local.** No real market data, no real venue connectivity, no order types
   beyond limit/market + GTC/IOC.
-- **Single-threaded** gateway and feed (a single accept/event loop); no concurrency model.
+- **Networking remains simple.** The TCP gateway is intentionally one-connection-at-a-time; the
+  threaded gateway-engine-feed pipeline is an opt-in correctness prototype, not a production
+  event loop.
 - **Benchmarks are microbenchmarks**, not end-to-end or production latency (see above).
 - **Networking is minimal**: loopback TCP order entry and a UDP market-data feed,
   unauthenticated, no TLS, no framing recovery beyond disconnect-on-malformed. The socket path is

docs/adr/0005-dynamic-concurrency-validation-is-evidence.md

@@ -26,6 +26,6 @@ correctness proof. Concurrency claims must distinguish:
 
 ## Consequences
 
-Future concurrency work belongs in M33 and may add randomized scheduling perturbation, longer
-Linux stress runs, expanded capacity coverage, and additional happens-before documentation. The
-repo must not state that TSan proves all interleavings correct.
+M33 adds deterministic scheduling perturbation and an opt-in repeated stress target. These broaden
+the executed schedule set, but the repo must not state that TSan, perturbation, or stress proves all
+interleavings correct.