test: 1M records/sec for 5 minutes and 5M records / sec for 1 minute soak — sustained backpressure behaviour

[allamiro]

Gap: every measurement so far is a 60-second window

All the throughput work (#94, #97) was measured over 60s windows under a 120s load. That is enough to compare builds, and it is NOT enough to find the failures that only appear under sustained backpressure — which is exactly the state the engine is in, since it archives ~9k rec/s against a ~1M rec/s inflow.

Nothing currently tests what happens when the engine runs hopelessly behind, for a long time. That is the normal condition for this engine today, so it is worth testing deliberately rather than discovering in production.

The test

Sustained 1M records/sec for 5 minutes into the lab, engine running, then observe.

Feasible on the current rig — already demonstrated at 999,720 rec/s for 120s (5 producers x 200k/s, 150-byte records, 143 MB/s):

• 5 min at 1M/s = 300M records ~= 45 GB on disk (126 GB free) - fits
• Contrast: 5M/s would need 225 GB and 10M/s 450 GB, neither of which fits, and the single-node broker saturates near 1M/s regardless (one producer peaked at 586k/s; three unthrottled produced only 910k/s total)

What to watch, and why each matters

observable	failure it would catch
vtop_inflight_batches over time	unbounded buffer growth -> OOM. Buffers accumulate across cycles; nothing has run long enough to see whether this plateaus
engine RSS	slow leak, only visible over minutes
vtop_verification_failures_total, vtop_failed_total	must stay 0
verified_total vs commits_total	the invariant, under sustained pressure rather than a burst
lag growth linearity	whether the engine degrades further as lag grows, or holds a steady deficit
vtop_source_read_errors_total	broker-side timeouts/errors appearing only under sustained load
state store size + write latency	SQLite ledger growth (relates to #77: startup loads the entire ledger into memory)
stage p95s over time	a stage degrading as the object store fills
work_dir disk usage	temp objects not being cleaned up

Specific hypotheses worth falsifying

1. inflight_batches grows without bound. Buffers accumulate per (source, partition) and only sealed ones flush. With 24 partitions x many topics and reads that cannot keep up, the count may climb steadily. A 60s window would not reveal this.
2. The ledger grows unboundedly. Every batch writes ~6 state transitions. At sustained load that is a lot of rows; perf: startup loads the entire ledger into memory #77 already notes startup loads the whole ledger into memory, so ledger size is a real operational limit.
3. Recovery time degrades. Restart the engine after the soak and time recover(). With a large ledger and many incomplete batches this could be slow enough to matter.
4. Kafka retention silently drops data. The load topic is set to retention.ms=1800000 (30 min). At 1M/s the engine falls so far behind that records may age out before being read — which is data loss the engine cannot detect, since a committed offset that no longer exists just resets. Worth confirming whether it is observable, and whether the engine should alarm on it.

Hypothesis 4 is the one I would most want an answer to, because it is a correctness question, not a performance one.

Acceptance

Not a pass/fail gate — an observation run. The outputs are: a plot or table of the metrics above over 5 minutes, and an issue filed for each hypothesis that turns out to be true.

Sequencing

Best run after the read-path fix (#96 steps A and B), so the soak reflects an engine that is actually trying to keep up rather than one that is 99.7% idle. Running it now would mostly measure "the queue grows", which is already known.

The exception is hypothesis 4 (retention-driven data loss), which is worth checking now precisely because the engine is far behind — that is the condition that triggers it.

[allamiro]

Status audit — 2026-07-25

WAITING FOR THE REQUIRED SOAK — keep open.

The read-path prerequisite #96 is complete, but the acceptance run has not been performed.

The repository has benchmarks/scenarios/06-longrun-5min.yaml, but that is a small-file, mock-backend longevity scenario. It is not the requested Kafka 1M records/s for 5 minutes (and it does not cover the title’s 5M/s for 1 minute).

Still required:

• 1M records/s × 5 minutes with the listed time-series metrics;
• restart afterward and recovery-time measurement;
• explicit test/observation for Kafka retention-driven silent loss;
• 5M records/s × 1 minute on a rig with sufficient broker/disk capacity;
• follow-up issues for any falsified hypothesis.

Waiting on: performance-lab capacity and an observation window. No compliant soak result exists yet, so no throughput or correctness claim should be made from this issue.

[allamiro]

Native-broker soak additions

In addition to archive/pipeline soak goals, please include sustained native-broker runs that stress:

• Quorum produce under RF=3 with one slow follower
• Producer flood against configured memory ceilings (see broker: enforce end-to-end memory budgets and replication backpressure #187)
• Consumers keeping up vs intentionally slow consumers
• mTLS + proof-carrying segments enabled for at least one long run

Link results to #185/#186 once those land. Avoid duplicating a second soak issue.