The issue
Running Trawl as a forward proxy for Jackett (MITM_PROXY_ENABLED=true). Some indexers that go through the browser tiers (Cloudflare challenge solve) failed with this error in Jackett:
Exception (thepiratebay): The archive entry was compressed using an unsupported compression method.: The archive entry was compressed using an unsupported compression method.
What is happening is .NET's HttpClient is trying to gunzip a body that's already plain text, because the response still carried a stale Content-Encoding: gzip header from the upstream server, even though the browser (Tier 3/4) had already decompressed the body before handing it back.
The solution:
In the below file...
File: apps/api/src/proxy/responsePolicy.ts
I found the first line (highlighted red) and replaced it with the new line below (highlighted green)...
Line:
- if (useRenderedHtml && TRANSFORMED_BODY_HEADERS.has(lower)) continue
+ if (TRANSFORMED_BODY_HEADERS.has(lower)) continue
The header-stripping was only conditional on useRenderedHtml (i.e. only applied for HTML pages where rendered DOM replaces the raw response), but result.body for non-HTML responses is also already browser-decoded, so the same headers need stripping there too.
I patched this locally in my Docker container and the previously-failing indexers (TPB, RBG) now return results with no error.
The issue
Running Trawl as a forward proxy for Jackett (
MITM_PROXY_ENABLED=true). Some indexers that go through the browser tiers (Cloudflare challenge solve) failed with this error in Jackett:What is happening is .NET's
HttpClientis trying to gunzip a body that's already plain text, because the response still carried a staleContent-Encoding: gzipheader from the upstream server, even though the browser (Tier 3/4) had already decompressed the body before handing it back.The solution:
In the below file...
File:
apps/api/src/proxy/responsePolicy.tsI found the first line (highlighted red) and replaced it with the new line below (highlighted green)...
Line:
The header-stripping was only conditional on
useRenderedHtml(i.e. only applied for HTML pages where rendered DOM replaces the raw response), butresult.bodyfor non-HTML responses is also already browser-decoded, so the same headers need stripping there too.I patched this locally in my Docker container and the previously-failing indexers (TPB, RBG) now return results with no error.