From b087b74020d88335aba9dd9fa3a7cb1917d589c3 Mon Sep 17 00:00:00 2001 From: Rob Konsdorf Date: Tue, 25 Aug 2026 10:20:19 -0400 Subject: [PATCH] docs(readme): let operators tell a starved SHIP feed from a slow filler An operator whose filler crawls far behind head has no way to tell whether the SHIP node is starving it or the filler is the bottleneck, so a slow catch-up reads as a filler defect. The progress line already carries the answer in its queue counters, but nothing said what they measure, that both stages run one block at a time, or that they only discriminate while the in-flight window is larger than one. The throughput keys were undocumented as well. The shipped example sits far below what a mainnet deployment needs and says nothing about it, and removing one of those keys wedges the reader at zero blocks with no error that names the cause. The Configuration pointer also promised comments in the example files, which JSON cannot carry. Signed-off-by: Rob Konsdorf --- README.md | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 97 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e08d6e09..8f0baca6 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,35 @@ Three JSON files in `config/` drive runtime behaviour: - `server.config.json`: HTTP server port, rate limits, CORS, cache policies, provider name and URL displayed in `/docs`. -See `config/*.example.json` for the full schema with comments. +See `config/*.example.json` for working starting values. They are examples +rather than a complete reference: `IReaderConfig` and its siblings in +`src/types/config.ts` carry keys the example files leave out, among them +`ship_max_blocks_queue`, `delete_data` and `list_polls`. + +### Filler throughput + +`readers.config.json` ships conservative values. A filler catching up from a +published dump or from a long outage needs larger ones. These keys decide the +ingestion rate; the table compares each shipped value with the one the WAX +mainnet deployment runs. + +| Key | Example | WAX mainnet | What it controls | +| --- | --- | --- | --- | +| `db_group_blocks` | `10` | `500` | Blocks per database transaction. Grouping applies only to an irreversible block while the reader is at least twice this many blocks behind head. A reversible block commits on its own whatever the distance, and so does every block once the reader is inside that distance. | +| `ship_prefetch_blocks` | `50` | `1000` | Becomes SHIP's `max_messages_in_flight`. The node stops sending after this many unacknowledged blocks, so it caps how deep the pipeline runs. | +| `ship_min_block_confirmation` | `30` | `30` | Blocks the client accumulates before acknowledging them. Keep it at or below `ship_prefetch_blocks`. | +| `ship_ds_queue_size` | `20` | `20` | Blocks allowed between deserialization and the database stage. | +| `ds_ship_threads` | `4` | core count | Worker threads for SHIP-level deserialization. Zero or absent runs deserialization on the filler's own event loop. | + +Never remove `ship_ds_queue_size`. An absent key leaves the semaphore without a +limit, every acquire parks forever, and the reader wedges before its first block. +The log shows `No blocks processed` every five seconds until the stall timeout +exits the process and the supervisor restarts it into the same wedge. + +A larger `db_group_blocks` writes more WAL per transaction, so raise +`max_wal_size` with it or Postgres checkpoints often enough to cancel the gain. +Set `PGSSLMODE=disable` when the database is local, because the client defaults +to `prefer` and negotiates TLS on a loopback connection otherwise. ## Restore from a published dump @@ -398,6 +426,74 @@ published dump is the durable fix. See `pnpm start:reconcile` seeds v2 contract state and template deletions and does not rebuild base rows. +**The filler ingests at the chain's own block rate while it is far behind +head.** CPU sits near idle and restarting changes nothing. Nothing in the filler +throttles the rate, so the filler is waiting on something. The progress line says +what. + +``` +Reader atomic-1 - Progress: 451800000 / 452500000 (12.34%) Speed: 2.0 B/s 118 W/s [DS:0|SH:0|JQ:0] (Syncs in 1000 hours) +``` + +`W/s` counts database write operations, not blocks. `DS` counts blocks waiting on +the database stage and `SH` counts SHIP messages received but not started. Both +exclude the item running, and both stages run one at a time. + +That last detail bounds the reading. With `ship_prefetch_blocks` set to 1 the +node never has a second block outstanding, so both counters sit at zero whatever +the cause. Take the live window from the `Requesting ship blocks` line at startup +before you trust a pair of zeros. + +So a filler that is itself the bottleneck backs up, showing `DS` close to +`ship_ds_queue_size` and `SH` holding the remainder of the +`ship_prefetch_blocks` window. Both at zero means the node is not filling that +window and the filler is idle. That reading, not the block rate, decides where to +look. + +The estimate is not itself a fault signal. It models the chain producing two +blocks per second while you sync, so an average below that prints +`(Syncs never)` and an average barely above it prints an enormous hour count. + +With `DS` and `SH` both high and `W/s` low, check whether the reader is stuck in +head mode. `contract_readers.live` is set when the reader first reaches head and +nothing sets it back, and the filler reads it at startup to choose head mode or +catch-up mode, without consulting the distance to head. A reader that once +reached head therefore starts in head mode however far behind it is. Before +2.2.1, head mode published every trace and delta to Redis inside the commit path, +a direct throughput cost on a backlog. Stop the filler, then: + +```sql +SELECT name, block_num, live FROM contract_readers; +UPDATE contract_readers SET live = false WHERE name = ''; +``` + +Start it again. It promotes itself back to head mode once it is genuinely near +head. Do this with the filler stopped, or the next commit rewrites the flag. Do +not change `block_num`: the checkpoint is correct and only the flag is wrong. + +With `DS` and `SH` both high and `W/s` high, the database write path is the +limit. Raise `db_group_blocks` and `max_wal_size` together. See +[Filler throughput](#filler-throughput). + +With both at zero, the SHIP node is the limit. One node fault is already ruled +out by steady progress: a range below the node's state-history retention floor +produces a reconnect loop, not slow progress. Confirm it with a log search for +`does not contain` and `Empty block #`. + +The other is not ruled out by anything the filler prints. The blocks-behind +figure comes from the head the node reports over the same socket, which is the +node's own view of itself. A node that is replaying or lagging reports a stale +local head and still feeds the filler steadily. Compare that head against a +trusted source for the chain before concluding the node's chain state is +current. + +Isolate the node by pointing the filler at a different one. `CHAIN_SHIP` +overrides the endpoint in `connections.config.json`, and the reader resumes from +its own checkpoint, so the swap costs nothing and reverses cleanly. A rate that +jumps against another node settles it. When the rate holds, measure disk service +time on the node's state-history volume and check whether the same process also +carries p2p sync and API traffic. + ## Development ```sh