diff --git a/CHANGELOG.md b/CHANGELOG.md index 69833b32..0d137e5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,9 @@ Entries reference the originating pull request and the closed issue where applic ## [Unreleased] ### Fixed +- **Fixed**: `Util::SleepUntil` never slept. Its comparison was inverted, so it slept only for deadlines already in the past, and since 2014 every interval passed through it has been a no-op. It now sleeps until the deadline, and that matters for three kinds of caller. **The merge-mem and merge-disk loops** still honoured each repo's next-merge deadline (#227), so the broken sleep made them busy-wait for up to 40ms after every merge step. Measured on native arm64 release builds with writes under way, the fix cuts `orlyi`'s CPU by 65–85%, with throughput within run-to-run noise. With one writer, the old build used 10.5–15.5 CPU-seconds per 20s and the fix 1.8–4.6; with eight, 11.6–12.3 against 4.9–5.0 per 15s. **The replication loop and the durable writer and merger** never actually waited out `--replication_interval`, `--durable_write_interval` or `--durable_merge_interval`, and still don't. Their dead sleeps are removed, so turning a real delay on is a measured decision of its own. A working 100ms replication interval, for example, would hold every write back from merging for up to that long, because the loop gates `ReleaseUpdate` even in SOLO. The flags are still accepted. `orly/perf/kv_exercise` now paces its inserts at `FlushIntervalMs`, as it always intended (#576). +- **Fixed**: concurrent writers no longer exhaust the Update pool and abort `orlyi`. Eight writers on one shared POV used to kill the server within seconds whenever the pool was small, and the pool is sized from the host's free RAM at startup. The root cause was the repo layer cleaner wedging. To free a dead disk layer it visits every runner to drop that file's caches, and the Tetris player's runner never let it back on: the player looped `Play(); usleep(0);` without yielding, and a runner only takes in frames handed over from other runners once its own queue goes idle. From then on nothing was ever freed. Every child merge copied the unpromoted backlog into a new layer while the old copy stayed allocated, so occupancy grew with the square of the write count and kept climbing after the writes stopped. Once the pool filled, `StepMergeMem` called `abort()`. Five changes. The player yields with `YieldSlow()` each round. The cleaner frees cheap memory layers first and visits the runners once per batch of disk layers rather than once per layer (~24ms each, which had fallen behind the merges). Write backpressure caps each POV's backlog at 1/32 of the Update pool and also waits, bounded at 5s, while the Update or Update Entry pool is more than half full. A merge that runs out of pool space before it publishes anything rolls back and retries instead of aborting. A Tetris round that hits `bad_alloc` is retried instead of silently killing the player. A new CI smoke (`clients/smoke/run-pool-pressure.sh`) runs 8 writers, on one shared POV and then on a POV each, against a 20000-update pool on the release build. Before the fix it aborts within about 2,000 writes; after it, both phases run 30s with no errors, and the pool drains once the writes stop. The single-POV, 8-writer throughput on a 100k pool is about 15–20% lower than with an uncapped backlog, and single-writer throughput is unchanged (#584). +- **Added**: the docker workflow now checks that the server *survives* writes, not just that writes answer. After the MCP smoke, it writes through all four POV flavours (five fresh POVs each) against the running container, waits, and asserts the container is still running with exit 0. The `v0.1.0` arm64 image passed every existing smoke while segfaulting just after the first write (#578). Run locally against the published images, the new step fails on `v0.1.0` arm64 (exit 139) and passes on `v0.1.1`. ## [v0.1.1] — 2026-10-02 diff --git a/changelog.d/576-sleep-until.md b/changelog.d/576-sleep-until.md deleted file mode 100644 index ba8535f5..00000000 --- a/changelog.d/576-sleep-until.md +++ /dev/null @@ -1 +0,0 @@ -- **Fixed**: `Util::SleepUntil` never slept. Its comparison was inverted, so it slept only for deadlines already in the past, and since 2014 every interval passed through it has been a no-op. It now sleeps until the deadline, and that matters for three kinds of caller. **The merge-mem and merge-disk loops** still honoured each repo's next-merge deadline (#227), so the broken sleep made them busy-wait for up to 40ms after every merge step. Measured on native arm64 release builds with writes under way, the fix cuts `orlyi`'s CPU by 65–85%, with throughput within run-to-run noise. With one writer, the old build used 10.5–15.5 CPU-seconds per 20s and the fix 1.8–4.6; with eight, 11.6–12.3 against 4.9–5.0 per 15s. **The replication loop and the durable writer and merger** never actually waited out `--replication_interval`, `--durable_write_interval` or `--durable_merge_interval`, and still don't. Their dead sleeps are removed, so turning a real delay on is a measured decision of its own. A working 100ms replication interval, for example, would hold every write back from merging for up to that long, because the loop gates `ReleaseUpdate` even in SOLO. The flags are still accepted. `orly/perf/kv_exercise` now paces its inserts at `FlushIntervalMs`, as it always intended (#576). diff --git a/changelog.d/578-image-survives-writes.md b/changelog.d/578-image-survives-writes.md deleted file mode 100644 index d290e627..00000000 --- a/changelog.d/578-image-survives-writes.md +++ /dev/null @@ -1 +0,0 @@ -- **Added**: the docker workflow now checks that the server *survives* writes, not just that writes answer. After the MCP smoke, it writes through all four POV flavours (five fresh POVs each) against the running container, waits, and asserts the container is still running with exit 0. The `v0.1.0` arm64 image passed every existing smoke while segfaulting just after the first write (#578). Run locally against the published images, the new step fails on `v0.1.0` arm64 (exit 139) and passes on `v0.1.1`. diff --git a/changelog.d/584-pool-exhaustion.md b/changelog.d/584-pool-exhaustion.md deleted file mode 100644 index 1bacc42b..00000000 --- a/changelog.d/584-pool-exhaustion.md +++ /dev/null @@ -1 +0,0 @@ -- **Fixed**: concurrent writers no longer exhaust the Update pool and abort `orlyi`. Eight writers on one shared POV used to kill the server within seconds whenever the pool was small, and the pool is sized from the host's free RAM at startup. The root cause was the repo layer cleaner wedging. To free a dead disk layer it visits every runner to drop that file's caches, and the Tetris player's runner never let it back on: the player looped `Play(); usleep(0);` without yielding, and a runner only takes in frames handed over from other runners once its own queue goes idle. From then on nothing was ever freed. Every child merge copied the unpromoted backlog into a new layer while the old copy stayed allocated, so occupancy grew with the square of the write count and kept climbing after the writes stopped. Once the pool filled, `StepMergeMem` called `abort()`. Five changes. The player yields with `YieldSlow()` each round. The cleaner frees cheap memory layers first and visits the runners once per batch of disk layers rather than once per layer (~24ms each, which had fallen behind the merges). Write backpressure caps each POV's backlog at 1/32 of the Update pool and also waits, bounded at 5s, while the Update or Update Entry pool is more than half full. A merge that runs out of pool space before it publishes anything rolls back and retries instead of aborting. A Tetris round that hits `bad_alloc` is retried instead of silently killing the player. A new CI smoke (`clients/smoke/run-pool-pressure.sh`) runs 8 writers, on one shared POV and then on a POV each, against a 20000-update pool on the release build. Before the fix it aborts within about 2,000 writes; after it, both phases run 30s with no errors, and the pool drains once the writes stop. The single-POV, 8-writer throughput on a 100k pool is about 15–20% lower than with an uncapped backlog, and single-writer throughput is unchanged (#584).