Connection impossible using NZBDAV

[pannal]

Using https://github.com/nzbdav-dev/nzbdav with nntp-proxy results in a "hanging" buffering connection independent of the nntp-proxy working mode.

nntp-proxy seems to be cycling available servers:

nntp-proxy  | 2025-11-08T05:12:15.107155Z  INFO nntp_proxy: NNTP proxy listening on 0.0.0.0:8119 (standard 1:1 mode)
nntp-proxy  | 2025-11-08T05:12:17.412774Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:36068 to backend BackendId(0) (redacted:8088)
nntp-proxy  | 2025-11-08T05:12:42.190598Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:43732 to backend BackendId(1) (redacted:8085)
nntp-proxy  | 2025-11-08T05:12:42.342913Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:43734 to backend BackendId(2) (redacted:8086)
nntp-proxy  | 2025-11-08T05:12:42.397806Z  WARN nntp_proxy::session::handlers::standard: Error reading from client 172.21.0.4:36068: Connection reset by peer (os error 104)
nntp-proxy  | 2025-11-08T05:12:42.397871Z  INFO nntp_proxy::proxy: Connection closed for client 172.21.0.4:36068: 0 bytes sent, 52 bytes received
nntp-proxy  | 2025-11-08T05:12:43.337475Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:43740 to backend BackendId(3) (redacted:8087)
nntp-proxy  | 2025-11-08T05:12:43.337855Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:43756 to backend BackendId(4) (redacted:8084)
nntp-proxy  | 2025-11-08T05:12:43.807019Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:52532 to backend BackendId(5) (redacted:8096)
nntp-proxy  | 2025-11-08T05:12:43.807357Z  INFO nntp_proxy::proxy: Routing client 172.21.0.4:52542 to backend BackendId(6) (redacted:8094)

[pannal]

It seems to work in --routing-mode standard, I'll report back!

[pannal]

Maybe some additional info that helps: This happened when using a plex watch together session, so five clients trying to access the same media at the exact same time.

[mjc]

fixed in #37 I think, can you check? per-command or hybrid mode should both work.

[pannal]

hybrid mode seems to work, but I haven't tested the actual original scenario, yet. I'll report back, thanks!

[pannal]

Hmm. Doesn't seem stable, honestly. Once NZBdav downloads something, any other connection to nntp-proxy seems to hang for a long time. Might not be your issue per se. I'll continue to investigate!

[pannal]

Yep, using the latest docker release, this is not stable. Switching over to a single provider connection in NZBdav fixes hanging playback requests. Using nntp-proxy breaks it once nntp-proxy downloads something and one requests playback (e.g. READ).

[mjc]

interesting, that sounds like something in my code. I'll take a look later this week

[mjc]

things have come a long way, can you give this another shot

[xbmc4lyfe]

@mjc I get a lot of timeouts, not sure what options to configure to fix that.

[xbmc4lyfe]

Posting some bugs I find along the way, I get insane performance out of his with my 6 usenet servers and have widdled down how to setup the best tiered config which solves my biggest bottleneck with NZBDAV: the checking of article health before the NZB starts streaming can take 5-10+ mins using nzbdav.

That goes down to under a minute using nntp-proxy but only when i use tiering correctly and claude identified I need to keep my fastest NNTP servers as tier 0. And not just use latency metrics but use the metrics you provide which identifies ACTUAL speed and suprisingly everyone except for news.newsgroupdirect.com is extremely slow on the actual fetching. Even other newsgroupdirect.com endpoints are slow.

But I am still running into a plethora of issues and they are all causing unique problems.

Really hoping I can submit some good reports that will make this issue.

So far my biggest finding is lower the # of conections that nzbdav uses against the local proxy to 10 -- even though I have over 300+ connections worth of usenet server. This is my current config.toml w/o server.

# Proxy Server Settings
[proxy]
host = "0.0.0.0"
port = 9119
threads = 0 # 12 cores a gaming CPU i7-8700k with HTT enabled
validate_yenc = true
log_file_level = "warn"

# Routing Configuration
[routing]
# Routing mode: "hybrid" (default), "per-command", or "stateful"
mode = "hybrid"
# Backend selection strategy: "least-loaded" (default) or "weighted-round-robin"
backend_selection = "least-loaded"
adaptive_precheck = true # this was identified as a solution by claude so turned on may try flipping it off

# Memory Configuration
[memory]
# TCP socket buffers for client and backend connections
socket_recv_buffer_size = 16777216
socket_send_buffer_size = 16777216
# added a 0 at the end
buffer_pool_size = 10485760
# added a 0 at the end saw big improvement with this (somehow even though nzbdav is only using say 25 connections this will trip)
buffer_pool_count = 1000
# Capture buffers used by cache ingest/response assembly
capture_pool_size = 10485760 # added zero at end
capture_pool_count = 160 # added zero at end

# Article Cache Configuration
[cache]
# beefy server so setting 12GB of RAM is nothing for it; and setting article cache up to hours so it buffers
# a huge amount into RAM thinking about increasing this to 20GB+
article_cache_capacity = "12gb"
article_cache_ttl_secs = 14400
store_article_bodies = true

[cache.disk]
# very fast nvme drive
path = "/nntp/data/articles"
capacity = "500gb" # thinking of going up even more
compression = "none" # no need already downloading compressed content
shards = 8 # could be an issue thinking of lowering to 1

[mjc]

for v0.5.0:

[proxy]
threads=1 # stay at 1 unless you  have like 40gbit+ internet access; the multithreading doesn't scale well yet.
validate_yenc = false # this isn't fully functional yet; it runs the code but doesn't do anything useful with it yet so it's just slowing things down. 

I'll investigate more once I get this retry bug sorted.

if your client has its own disk cache so I would probably disable the disk cache in this. I've done my best to minimize the cache-miss behavior but it's still a bit faster to have it off. use a bigger client cache instead.

as far as connections to the proxy from your client:

if your client does nntp pipelining (I read the source for sabnzbd, it does that. basically it will send more commands without waiting for a response first. nzbget does not), then you want the minimum number of connections to get the max speed.

if your client doesn't do pipelining then you can possibly benefit from oversubscribing. for example if you have 10 backend connections total (lets say 2 servers 5 conns each) and no pipelining in your client, then doing 20 connections to the proxy will turn into a pipeline depth of 1. that can give the backend more time to queue up articles to send.

I wouldn't be surprised if there's a bunch more disk and memory cache bugs, I haven't redone those since I changed nearly the entire non-cache path, in 0.5.0

[mjc]

The availability index is the thing that speeds things up the most btw; basically if the proxy knows which backends have which articles it makes much better routing decisions

[mjc]

try 0.5.1 with the disk and memory caches off.