Problem when crawling page with a lot of file downloads - ends up with http GOAWAY even when using delays

[clydzik]

Hey.

Recently i tried to archive one page / domain.

The page is in general zimed but the problem is with files. This page has a lot of files, many grouped on single page.
The server itslef start rejecting me (what i understand) but i tried to mitigate it with (important parameters)

--pageLoadTimeout 90 --behaviorTimeout 90 --pageExtraDelay 2 --waitUntil networkidle0 --workers 1

here are the logs (cut for whats i think is important).

{"timestamp":"2026-03-20T12:41:49.256Z","logLevel":"info","context":"general","message":"Waiting 2 seconds before moving on to next page","details":{"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/SGM3122_s.png","workerid":0}}
{"timestamp":"2026-03-20T12:41:52.388Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt"}}
{"timestamp":"2026-03-20T12:41:52.392Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":967,"total":10000,"pending":41,"failed":0,"limit":{"max":10000,"hit":true},"pendingPages":["{\"seedId\":1,\"sta
rted\":\"2026-03-20T12:39:52.870Z\",\"extraHops\":0,\"url\":\"http:\\/\\/download.rockbox.org\\/bootloader\\/agptek\\/ROCKER-20171101-patched.upt\",\"added\":\"2026-03-20T10:16:45.958Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12
:38:16.255Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v18-patched.upt\",\"added\":\"2026-03-20T10:16:37.110Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:37:56.710Z\",\"extraHops\
":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v12-patched.upt\",\"added\":\"2026-03-20T10:16:37.077Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:48.624Z\",\"extraHops\":0,\"url\":\"https:\\/\\/
www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/step2.jpg\",\"added\":\"2026-03-20T10:16:45.996Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:35.928Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/boot
loader\\/surfans\\/F20-v27-patched.upt\",\"added\":\"2026-03-20T10:16:37.131Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:39.917Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/surfans\\/F20-v30-p
atched.upt\",\"added\":\"2026-03-20T10:16:37.136Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:43.804Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/surfans\\/F20-v33-patched.upt\",\"added\":\"202
6-03-20T10:16:37.142Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:08.390Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v15-patched.upt\",\"added\":\"2026-03-20T10:16:37.097Z\",\"
depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:39:13.466Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/dmesg_output_surfans_v3.4_hw4.txt\",\"added\":\"2026-03-20T10:16:37.169Z\",\"depth
\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:37:36.612Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/aigo\\/EROSQ-v18-patched.upt\",\"added\":\"2026-03-20T10:16:37.047Z\",\"depth\":2}","{\"seedId\":1,\"start
ed\":\"2026-03-20T12:37:48.700Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/agptek\\/H3-20180905-patched.upt\",\"added\":\"2026-03-20T10:16:37.063Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:2
0.313Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v22-patched.upt\",\"added\":\"2026-03-20T10:16:37.112Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:32.010Z\",\"extraHops\":0,\
"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/surfans\\/F20-v25-patched.upt\",\"added\":\"2026-03-20T10:16:37.127Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:37:52.716Z\",\"extraHops\":0,\"url\":\"https:\\/\\/downloa
d.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v11-patched.upt\",\"added\":\"2026-03-20T10:16:37.069Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:37:40.643Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/aigo\\/EROSQ-v20-patched.upt\",\"added\":\"2026-03-20T10:16:37.053Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:24.302Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v23-patched.upt\",\"added\":\"2026-03-20T10:16:37.120Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:04.550Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v14-patched.upt\",\"added\":\"2026-03-20T10:16:37.096Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:37:32.749Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/aigo\\/EROSK-v13-patched.upt\",\"added\":\"2026-03-20T10:16:37.043Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:47.748Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/surfans\\/F20-v34-patched.upt\",\"added\":\"2026-03-20T10:16:37.151Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:59.087Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/dmesg_output_hifiwalker_v1.3_hw1.5.txt\",\"added\":\"2026-03-20T10:16:37.161Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:37:44.516Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/aigo\\/EROSQ-v21-patched.upt\",\"added\":\"2026-03-20T10:16:37.054Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:41:43.380Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/FiioM3K\\/PCB_front_s.png\",\"added\":\"2026-03-20T10:17:36.998Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:00.649Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v13-patched.upt\",\"added\":\"2026-03-20T10:16:37.094Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:28.113Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/surfans\\/F20-v22-patched.upt\",\"added\":\"2026-03-20T10:16:37.122Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:39:15.871Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/amixer_contents_output_surfans_v3.4_hw4.txt\",\"added\":\"2026-03-20T10:16:37.170Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:39:11.263Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/amixer_contents_output_erosq_hw2.txt\",\"added\":\"2026-03-20T10:16:37.167Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:30.482Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/PCB26495_(2).jpg\",\"added\":\"2026-03-20T10:16:45.983Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:34.377Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/n0x8s7.jpg\",\"added\":\"2026-03-20T10:16:45.986Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:51.855Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/dmesg_output_hifiwalker_v1.2_hw1.txt\",\"added\":\"2026-03-20T10:16:37.157Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:41:51.262Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/FiioM3K\\/m3k_alsa.txt\",\"added\":\"2026-03-20T10:17:37.004Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:39:49.356Z\",\"extraHops\":0,\"url\":\"http:\\/\\/download.rockbox.org\\/bootloader\\/agptek\\/rocker.zip\",\"added\":\"2026-03-20T10:16:45.955Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:41:31.441Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/FiioM3K\\/AK4376A_s.png\",\"added\":\"2026-03-20T10:17:36.979Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:39:08.872Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/dmesg_output_erosq_hw2.txt\",\"added\":\"2026-03-20T10:16:37.165Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:12.387Z\",\"extraHops\":0,\"url\":\"https:\\/\\/download.rockbox.org\\/bootloader\\/hifiwalker\\/H2-v16-patched.upt\",\"added\":\"2026-03-20T10:16:37.097Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:56.283Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/step5.jpg\",\"added\":\"2026-03-20T10:16:46.003Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:41:39.403Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/FiioM3K\\/PCB_back_s.png\",\"added\":\"2026-03-20T10:17:36.997Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:41:47.274Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/FiioM3K\\/SGM3122_s.png\",\"added\":\"2026-03-20T10:17:37.002Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:26.627Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/PCB26495_(1).jpg\",\"added\":\"2026-03-20T10:16:45.977Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:38:55.829Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AIGOErosQK\\/amixer_contents_output_hifiwalker_v1.3_hw1.5.txt\",\"added\":\"2026-03-20T10:16:37.159Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:51.460Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/step3.jpg\",\"added\":\"2026-03-20T10:16:45.998Z\",\"depth\":2}","{\"seedId\":1,\"started\":\"2026-03-20T12:40:53.870Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.rockbox.org\\/realwiki\\/pub\\/Main\\/AgptekRocker\\/step4.jpg\",\"added\":\"2026-03-20T10:16:46.001Z\",\"depth\":2}"]}}
{"timestamp":"2026-03-20T12:41:52.999Z","logLevel":"info","context":"general","message":"Waiting 2 seconds before moving on to next page","details":{"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.649Z","logLevel":"info","context":"fetch","message":"Direct fetch successful","details":{"url":"https://download.rockbox.org/bootloader/aigo/EROSK-v13-patched.upt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.650Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":2,"page":"https://download.rockbox.org/bootloader/aigo/EROSK-v13-patched.upt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.735Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v18-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.737Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v18-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.737Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":57835520,"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v18-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.788Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v20-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.788Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v20-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.789Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":58480640,"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v20-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.848Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v21-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.848Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v21-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.848Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":58480640,"url":"https://download.rockbox.org/bootloader/aigo/EROSQ-v21-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.889Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/agptek/H3-20180905-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.890Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/agptek/H3-20180905-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.890Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":61390848,"url":"https://download.rockbox.org/bootloader/agptek/H3-20180905-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.936Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v11-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.938Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v11-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:53.938Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":58089472,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v11-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.050Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v12-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.051Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v12-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.051Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":58724352,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v12-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.109Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v13-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.110Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v13-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.111Z","logLevel":"warn","context":"recorder","message":"Async fetch: possible response size mismatch","details":{"type":"Recorder","size":0,"expected":58941440,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v13-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.162Z","logLevel":"warn","context":"recorder","message":"takeReader interrupted","details":{"size":0,"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v14-patched.upt","type":"exception","message":"HTTP/2: \"GOAWAY\" frame received with code 0","stack":"SocketError: HTTP/2: \"GOAWAY\" frame received with code 0\n    at ClientHttp2Session.onHttp2SessionGoAway (/app/node_modules/undici/lib/dispatcher/client-h2.js:188:31)\n    at ClientHttp2Session.emit (node:events:524:28)\n    at Http2Session.onGoawayData (node:internal/http2/core:698:11)","page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}
{"timestamp":"2026-03-20T12:41:54.163Z","logLevel":"warn","context":"recorder","message":"Response truncated","details":{"url":"https://download.rockbox.org/bootloader/hifiwalker/H2-v14-patched.upt","canRetry":false,"page":"https://www.rockbox.org/realwiki/pub/Main/FiioM3K/m3k_alsa.txt","workerid":0}}

As You can notice at some point in time there is a lot of pending pages like "pending":41 (i have single worker) that are opened on direct download files and as such seems not respecting (fully) the --pageExtraDelay 2 or --waitUntil networkidle0

as an example the https://download.rockbox.org/bootloader/hifiwalker/H2-v11-patched.upt is 58 Mb and to get it it take a while.

So my understanding is that zimit/crawler queue this downloads and these are in progress and files downloads seems async thus even if i try hard the page is rejecting me after a while what i understand somehow (i have many downloads open in parallel).

Is there any strategy or parameter i missed to solve this problem ?

Thanks in Advance
Chris

[benoit74]

This is indeed an upstream crawler issue, which should better be reported upstream at https://github.com/webrecorder/browsertrix-crawler/ to have a chance to get a response. I don't have a clear understanding of which parameter could help you here (if any).

[clydzik]

Thanks. i will. I belive that by put more as --pageExtraDelay will help but this is difficult to accept such delays for 10k pages...