fix(research): hard wall-clock fetch deadline that also cuts header and gzip drips

[wikithoughts]

What changed

Context: the review findings this task targets, F7 (quadratic _HTML_JUNK regex) and F8 (no total fetch deadline, response never closed), were already fixed on main by #31 (2e239f0). The review was written against 30ae508, before #31 landed. This PR closes the one gap #31 left, which its own _guarded_get docstring noted as a residual: the 30 s deadline was only checked between redirect hops and between body chunks. A server dripping one byte every few seconds can therefore hold the fetch thread indefinitely, with no per-read timeout ever firing, if it drips through either:

• its status line or headers (inside requests' header parse), or
• a gzip header FCOMMENT (urllib3's read1 loops over it internally).

passiveworkers/research.py:

• _FetchWatchdog: a daemon threading.Timer. At the deadline it calls shutdown(SHUT_RDWR) on a dup() of every socket the fetch opened. The dup is used because it stays valid after TLS wrapping detaches the original socket object.
  • Any recv that is still blocked returns at once, and fetch_extract raises TimeoutError.
  • A socket that connects after the deadline is cut immediately.
  • On every path, the timer is cancelled and the dups are closed.
• _watched_session: a per-fetch requests.Session. Its adapter swaps each pool instance's ConnectionCls for a subclass whose _new_conn registers the new socket. Because it subclasses whatever the pool already uses, the swap also works for HTTPS pools and for the CONNECT tunnel through an HTTPS_PROXY.
• _guarded_get: gains a session= parameter (default: the requests module). SSRF re-validation on every hop and allow_redirects=False are unchanged.
• Body read moved into _fetch_body: after the read loop, a watchdog-cut socket that looks like a clean EOF is never returned as a complete body.

tests/test_research.py:

• _patch_get helper: the existing fetch_extract tests now patch Session.get as well as requests.get.
• New tests, all against loopback stub servers with NO_PROXY pinned and no external network:
  • Header, gzip-comment and TLS-handshake drips each hit TimeoutError in under 3 s, with _FETCH_DEADLINE=0.5.
  • A socket tracked after the deadline is cut at once.
  • A real loopback page still fetches through the watched session, leaving no pw-fetch-deadline thread and no extra fd.

Findings addressed

• F7: already fixed by fix(research): linear-time HTML strip and a total fetch deadline #31. Timings are below; this PR doesn't change it.
• F8: makes the deadline a real wall clock covering the status line, headers and gzip-decoded body reads, not just the gaps between chunks.

Before / after timings (loopback, _FETCH_DEADLINE=0.5, stub server gives up after 4 s)

case	pre-#31 (30ae508)	main before this PR (f497835)	this branch
_strip_html, 40k unclosed <nav>	26 037 ms	15 ms	28 ms (same code; noise)
_strip_html, 200k lone <	18 318 ms	14 ms	18 ms
header drip	4.00 s (ConnectionError when server quit)	4.00 s (ConnectionError)	0.50 s TimeoutError
gzip-comment drip	4.00 s (ProtocolError)	4.00 s (ProtocolError)	0.50 s TimeoutError
TLS handshake drip	4.00 s (SSLError)	0.52 s ReadTimeout*	0.51 s TimeoutError

*On main, CPython already bounds a TLS handshake with a single deadline. The TLS case is kept only as a regression guard, not claimed as a fix.

New tests on main's research.py: 2 failed, 1 passed.

• The drip test fails on http_headers: 8.1 s, against a 3 s bound.
• The watchdog unit test fails because _FetchWatchdog doesn't exist on main.
• The real-page test passes on both versions; it guards the happy path.

Verification (rebased on af2e814)

• python -m py_compile $(git ls-files '*.py') → exit 0
• ruff check . && pytest tests/ -q --cov=passiveworkers --cov-fail-under=75 → exit 0: All checks passed!, 757 passed, 9 skipped, coverage 80.90%
• With bogus HTTP_PROXY/HTTPS_PROXY set, the new loopback tests still pass (2 passed).
• An adversarial read-only review pass found no Critical or High issues. Its proxy-env test note is fixed (NO_PROXY pinned).
• No UI touched, so no browser check is needed.

Not verified / residuals

• urllib3 1.26.x: not exercised. The installed version is 2.8.0, and pyproject only pins requests>=2.32. HTTPConnection._new_conn and HTTPConnectionPool.ConnectionCls exist there, but this is untested.
• requests versions without HTTPAdapter.get_connection_with_tls_context: the swap would not run, and fetches would quietly fall back to main's between-chunk deadline. Releases before 2.32.2 lack this hook; I believe 2.32.0 and 2.32.1 were yanked but did not confirm. The header-drip test fails loudly if the hook isn't called.
• SOCKS proxies: not run (PySocks isn't installed). The reviewer read urllib3/contrib/socks.py and expects the subclassing to work.
• Timing edge: a fetch whose last read lands at the exact deadline, or an HTTP error raised after it, is reported as TimeoutError. This is a safe failure.
• Connection close timing: a dup keeps each TCP connection open, so the server doesn't see a FIN, until the fetch ends (at most 30 s).
• Remaining unbounded phases: DNS resolution (getaddrinfo) and per-address connect retries are still outside the watchdog. No socket exists yet at that point.

Follow-up: test (3.10) abort in CI run 38124448503 (head 324a242)

Root cause. The abort predates this PR, and the fetch watchdog is not involved. CI aborted with exit 134 in the scoped coverage step: Fatal Python error: _enter_buffered_busy ... possibly due to daemon threads. The two lines printed just before it come from net/baseline.py:42 and net/coordinator_app.py:413, and both run only in the coordinator's pw-baseline daemon thread. Every POST /jobs in the suite starts that thread. It polls for up to 900 s, tries Ollama on loopback with the default PW_BASELINE_LOCAL_MODEL, and then prints. One still running when the suite ended printed into interpreter shutdown, and 3.10 aborted. The watchdog never writes to stdout.

What changed (on top of a merge of origin/main, which brings in #49):

• 0b21205 passiveworkers/research.py: _FetchWatchdog.close() now cancels and joins the pw-fetch-deadline timer, so it never outlives fetch_extract. Previously it only cancelled it.
• 0b21205 tests/test_research.py: new test_fetch_watchdog_thread_never_outlives_fetch_extract. It checks threading.enumerate() right after fetch_extract returns, on success, on error and when the deadline fires. The real-socket happy-path test now asserts directly instead of polling for 1 s. Both fail against the cancel-only close(): 3/3 runs gave 1 failed, 1 passed.
• 324a242 tests/conftest.py: new autouse fixture _no_real_baseline_threads. It refuses to start a pw-baseline thread while the coordinator's current _baseline_async is the real one. It raises RuntimeError, which takes fix(net): cap coordinator request bodies, bound baseline threads, index hot lookups, throttle rate-limit sweep #49's own "can't start a thread" path in _start_baseline and releases the slot. Emptying PW_BASELINE_LOCAL_MODEL is not enough, because the thread still starts, polls and prints. A module-level patch is undone by the importlib.reload() calls many tests make, so the guard sits on Thread.start. fix(net): cap coordinator request bodies, bound baseline threads, index hot lookups, throttle rate-limit sweep #49's tests that swap in their own _baseline_async still get real threads, and they pass.

Verification (local, wikiclaw-1):

• Declared gate on Python 3.12.3: ruff check . && pytest tests/ -q --cov=passiveworkers --cov-fail-under=75 → exit 0. All checks passed!, 771 passed, 9 skipped, coverage 80.75%.
• python -m py_compile $(git ls-files '*.py') → exit 0.
• CI whole-project floor on Python 3.10.21 (uv, .[all,dev] like CI): pytest tests/ -q --cov=passiveworkers --cov-report=term-missing --cov-fail-under=75 → exit 0, 786 passed, 1 skipped, 83.45%.
• CI scoped floor (the step that aborted) on Python 3.10.21: pytest tests/ -q --cov=passiveworkers.doctor --cov=passiveworkers.net.agent --cov=passiveworkers.operator --cov=passiveworkers.mcp_server --cov-report=term-missing --cov-fail-under=75 → exit 0, 786 passed, 1 skipped, 80.38%. The log has 0 [baseline] lines and 0 Fatal Python lines.
• Session-end thread probe on 3.10: after the change, 0 pw-baseline and 0 pw-fetch-deadline threads at exit and 0 [baseline] lines. Before it, [baseline] lines printed during the suite.

Not verified:

• The crash itself did not reproduce locally. Three runs of the 3.10 scoped step on the old head all exited 0, so this is a timing race, and the fix rests on the CI log's evidence plus the probe. CI on the new head has not been watched (per run rules).
• At session end about 170 idle pw-reaper daemon threads (net/_store_jobs.py) are still alive. They print only when a reaper pass fails, so they are a latent version of the same hazard. That code is owned by PR fix(ledger): roll the in-memory ledger back when escrow persistence fails #32 and was left untouched.

🤖 Generated with Claude Code