docs(readme): let operators tell a starved SHIP feed from a slow filler - #197
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Documents how operators can distinguish SHIP starvation from filler bottlenecks.
Changes:
- Documents filler throughput settings and WAX mainnet values.
- Explains progress counters, estimates, and recovery steps.
- Adds SHIP and database troubleshooting guidance.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| policies, provider name and URL displayed in `/docs`. | ||
|
|
||
| See `config/*.example.json` for the full schema with comments. | ||
| See `config/*.example.json` for the full key set. |
Comment on lines
+440
to
+442
| `ship_prefetch_blocks` window. Both at zero means the node is not filling that | ||
| window and the filler is idle. That reading, not the block rate, decides where to | ||
| look. |
Comment on lines
+469
to
+473
| With both at zero, the SHIP node is the limit. Two node faults are already ruled | ||
| out by steady progress. The blocks-behind figure comes from the head the node | ||
| itself reports, so the node is at chain head. And a range below the node's | ||
| state-history retention floor produces a reconnect loop, not slow progress. | ||
| Confirm the second with a log search for `does not contain` and `Empty block #`. |
robrigo
force-pushed
the
docs/filler-throughput-tuning
branch
from
August 25, 2026 14:35
8da28c5 to
3a3174d
Compare
|
|
||
| | Key | Example | WAX mainnet | What it controls | | ||
| | --- | --- | --- | --- | | ||
| | `db_group_blocks` | `10` | `500` | Blocks per database transaction while the reader is more than twice this many blocks behind head. Within that distance it commits one block at a time regardless. | |
Comment on lines
+236
to
+238
| Never remove `ship_ds_queue_size`. An absent key leaves the semaphore without a | ||
| limit, every acquire parks forever, and the reader wedges at zero blocks with | ||
| only `Not receiving any blocks` in the log. |
An operator whose filler crawls far behind head has no way to tell whether the SHIP node is starving it or the filler is the bottleneck, so a slow catch-up reads as a filler defect. The progress line already carries the answer in its queue counters, but nothing said what they measure, that both stages run one block at a time, or that they only discriminate while the in-flight window is larger than one. The throughput keys were undocumented as well. The shipped example sits far below what a mainnet deployment needs and says nothing about it, and removing one of those keys wedges the reader at zero blocks with no error that names the cause. The Configuration pointer also promised comments in the example files, which JSON cannot carry. Signed-off-by: Rob Konsdorf <rob@facings.io>
robrigo
force-pushed
the
docs/filler-throughput-tuning
branch
from
August 25, 2026 14:43
3a3174d to
b087b74
Compare
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
An operator whose mainnet filler crawls far behind head has no way to tell whether the SHIP node is starving the filler or the filler is the bottleneck, so a slow catch-up reads as a filler defect and the conversation starts from scratch every time.
The progress line already carries the answer.
DSandSHare queue depths on two stages that each run one item at a time, so a filler that cannot keep up backs up and both counters climb, while a starved feed leaves both at zero. Nothing said that, nothing saidW/scounts database writes rather than blocks, and nothing said the counters only discriminate while the in-flight window is larger than one. The estimate compounds a hardcoded two blocks per second, which is why a filler at the chain's own rate prints an absurd hour count ornever.The throughput keys were undocumented too. The shipped example sits far below what a mainnet deployment needs and says nothing about it, and one of those keys wedges the reader at zero blocks with no error naming the cause if it is removed.
The new Troubleshooting entry also covers
contract_readers.live, which is set when a reader first reaches head and never set back, so a reader that once reached head starts in head mode however far behind it is. Before 2.2.1 that published a whole backlog to Redis inside the commit path.Finally, the Configuration pointer promised comments in the example files, which JSON cannot carry.
Validation
Documentation only, no source or config change.
pnpm teston this branch runs 419 passing and 37 pending, matchingmain. The behaviours described were read back againstsrc/filler/filler.ts,src/filler/receiver.ts,src/filler/notifier.ts,src/filler/database.tsand the SHIP client, and the queue-depth reading was checked against a filler log whose request line shows a window of 50 while both counters hold at zero.