refactor(config): standardize configuration naming

.env.example

@@ -16,7 +16,7 @@ BROWSER_ACQUIRE_TIMEOUT_MS=15000
 # Recycle a browser after this many temporary challenge contexts. Set 0 to disable.
 BROWSER_RECYCLE_AFTER_CONTEXTS=8
 # Maximum Firefox content processes per browser.
-BROWSER_CONTENT_PROCESSES=2
+BROWSER_MAX_CONTENT_PROCESSES=2
 # Browsers in the headful sub-pool, used only for DataDome escalations. Warmed on first use
 # and additional to BROWSER_POOL_SIZE (~380 MB each). Set 1 to scrape DataDome targets.
 BROWSER_HEADFUL_POOL_SIZE=0
@@ -26,41 +26,45 @@ BROWSER_STALL_TIMEOUT_MS=180000
 BROWSER_CLOSE_TIMEOUT_MS=10000
 # Maximum time to wait for a browser launch.
 BROWSER_LAUNCH_TIMEOUT_MS=90000
-# Optional Chrome executable override.
-CHROME_EXECUTABLE=
-
 # Optional viewport screenshots (browser tiers only)
 SCREENSHOT_SETTLE_MS=3000
 SCREENSHOT_TIMEOUT_MS=10000
 SCREENSHOT_JPEG_QUALITY=60
 SCREENSHOT_MAX_BYTES=4000000
 
-# Optional browser diagnostics (browser tiers only)
-CAPTURE_MAX_CONSOLE_ENTRIES=500
-CAPTURE_MAX_NETWORK_ENTRIES=1000
-CAPTURE_MAX_STRING_CHARS=2000
-CAPTURE_MAX_TOTAL_CHARS=1000000
-CAPTURE_MAX_REDIRECT_ENTRIES=50
-CAPTURE_SIZES_TIMEOUT_MS=2000
-CAPTURE_MAX_RESPONSE_PATTERNS=10
+# Optional console and network diagnostics (browser tiers only)
+DIAGNOSTICS_MAX_CONSOLE_ENTRIES=500
+DIAGNOSTICS_MAX_NETWORK_ENTRIES=1000
+DIAGNOSTICS_MAX_STRING_CHARS=2000
+DIAGNOSTICS_MAX_TOTAL_CHARS=1000000
+DIAGNOSTICS_SIZE_TIMEOUT_MS=2000
+
+# Optional main-document redirect chain (browser tiers only)
+REDIRECT_MAX_ENTRIES=50
+REDIRECT_MAX_URL_CHARS=2000
+REDIRECT_MAX_TOTAL_CHARS=1000000
+
+# Optional matched response-body capture (browser tiers only)
+CAPTURE_MAX_PATTERNS=10
 CAPTURE_MAX_RESPONSES=5
-CAPTURE_MAX_RESPONSE_BYTES=5242880
-CAPTURE_MAX_RESPONSE_TOTAL_BYTES=10485760
+CAPTURE_MAX_BODY_BYTES=5242880
+CAPTURE_MAX_TOTAL_BYTES=10485760
 CAPTURE_MAX_READ_BYTES=10485760
+CAPTURE_MAX_METADATA_CHARS=2000
 CAPTURE_BODY_TIMEOUT_MS=5000
 CAPTURE_SETTLE_MS=15000
 CAPTURE_MAX_SETTLE_MS=60000
-CAPTURE_SETTLE_IDLE_FLOOR_MS=5000
+CAPTURE_IDLE_FLOOR_MS=5000
 
 # Sessions
-# Redis stores solved cookies and browser sessions.
+# Redis stores solved cookies and browser sessions. Leave empty to disable the cache.
 REDIS_URL=redis://localhost:6379
+# Lifetime of Redis-cached sessions in seconds.
+REDIS_SESSION_TTL_SECONDS=3600
 # Maximum time for each Redis connection attempt.
 REDIS_CONNECT_TIMEOUT_MS=5000
 # Delay between background reconnect attempts. Set 0 to disable retry.
 REDIS_RETRY_DELAY_MS=5000
-# Lifetime of cached sessions in seconds.
-SESSION_TTL_SECONDS=3600
 
 # Upstream proxies
 # Tier 3 datacenter proxy: HTTP, authenticated HTTP, or SOCKS5.
@@ -82,18 +86,18 @@ RESIDENTIAL_PROXY_LIST_FILE=
 
 # HTTP/HTTPS challenge-bypassing proxy
 # Enables the general proxy listener.
-MITM_PROXY_ENABLED=false
+MITM_ENABLED=false
 # Proxy address and port. Use 127.0.0.1 for local-only access.
-MITM_PROXY_HOST=0.0.0.0
-MITM_PROXY_PORT=8192
+MITM_HOST=0.0.0.0
+MITM_PORT=8192
 # Persistent directory for the generated CA certificate and key.
-MITM_PROXY_CA_DIR=/data/proxy-ca
+MITM_CA_DIR=/data/proxy-ca
 # Highest allowed escalation tier. Leave empty to allow Tier 4.
-MITM_PROXY_MAX_TIER=
+MITM_MAX_TIER=
 # Skip the direct Tier 0 probe and route requests through the scraper. Disables Tier 0 streaming.
-MITM_PROXY_ALWAYS_SCRAPE=false
+MITM_ALWAYS_SCRAPE=false
 # Logs proxied requests. Errors are always logged.
-MITM_PROXY_DEBUG=false
+MITM_DEBUG=false
 
 # CAPTCHA audio
 # Optional Whisper-compatible speech-to-text endpoint.

CHANGELOG.md

@@ -7,12 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+- Reorganized runtime configuration into concise subsystem namespaces. Console/network limits now use `DIAGNOSTICS_*`, redirect-chain limits use `REDIRECT_*`, response-body limits remain under `CAPTURE_*`, forward-proxy settings use `MITM_*`, and Redis session expiry is `REDIS_SESSION_TTL_SECONDS`. `BROWSER_MAX_CONTENT_PROCESSES` now makes the process cap explicit; the unused `CHROME_EXECUTABLE` entry was removed. These names replace the previous environment variables without compatibility aliases; the [configuration migration guide](apps/docs/deployment/configuration-migration.md) contains the complete mapping.
+- An empty or unset `REDIS_URL` now disables the optional session cache without attempting a localhost connection or requiring a separate enable flag.
+- Supplied Compose variants now explicitly pass every supported runtime tuning variable from `.env` into the container instead of silently ignoring screenshot, diagnostics, redirect, response-capture, and CAPTCHA settings.
+- Renamed internal byte/character limits to `STREAM_THRESHOLD_BYTES` and `MCP_HTML_MAX_CHARS`, and namespaced the Camoufox font-retention build argument as `CAMOUFOX_KEEP_SPOOFED_OS_FONTS`.
+
 ### Added
-- Optional `MITM_PROXY_ALWAYS_SCRAPE=true` mode skips the forward proxy's direct Tier 0 probe and routes ordinary HTTP requests immediately into the existing scraper ladder for sites where the probe itself triggers a temporary ban (#93). WebSocket relays remain direct; the mode is off by default and documented as unsuitable for general media/download traffic because it bypasses Tier 0 streaming.
+- Optional `MITM_ALWAYS_SCRAPE=true` mode skips the forward proxy's direct Tier 0 probe and routes ordinary HTTP requests immediately into the existing scraper ladder for sites where the probe itself triggers a temporary ban (#93). WebSocket relays remain direct; the mode is off by default and documented as unsuitable for general media/download traffic because it bypasses Tier 0 streaming.
 - **DataDome support.** Detect Device Check, slider CAPTCHA and `t=bv` hard blocks from challenge markers and `x-dd-b`. Device Check uses a dedicated waiter and an optional headful Xvfb pool; the slider is reported as `datadome-captcha-required`. Enable startup-warmed capacity with `BROWSER_HEADFUL_POOL_SIZE=1` (off by default).
 - **AWS WAF Challenge support.** Detect the documented `202` Challenge and `405` CAPTCHA responses from their `x-amzn-waf-action` header, with a conservative two-marker HTML fallback. Silent challenges use a dedicated browser waiter for the domain-matching `aws-waf-token`; interactive CAPTCHA is surfaced as `aws-waf-captcha-required` for a future solver.
 - Optional response-body capture: `captureResponses` on `POST /scrape` takes URL patterns (a substring, or a glob when the pattern contains `*` or `?`) and returns the matching responses' bodies in `ScrapeResult.capturedResponses`, so a page that ships an empty shell and loads its content over a background request is still readable. `settleTimeout` holds the page open after load waiting for a match and `waitForSelector` ends that window early. Off by default — without patterns no listener is attached. Pattern count, body count, per-body bytes and total bytes are bounded and tunable via `CAPTURE_*`; a body over its budget comes back trimmed and flagged `truncated`, a binary or unknown content type comes back base64, and a body that cannot be read carries its own `error` rather than failing the scrape.
-- Optional console, network and redirect-chain capture: `consoleLogs`, `networkLogs` and `redirectChain` on `POST /scrape` return the page's console messages, per-request resource timings, and the URLs the main document walked (`ScrapeResult.consoleLogs` / `networkLogs` / `redirectChain`), captured by the browser tiers (2-4). Each flag is independent and off by default — without it no listener is attached and nothing is buffered. Entry counts, string lengths and total captured characters are bounded and tunable via `CAPTURE_*`; anything past a cap is dropped whole rather than truncated, and a capture failure leaves the field unset rather than failing the scrape.
+- Optional console, network and redirect-chain capture: `consoleLogs`, `networkLogs` and `redirectChain` on `POST /scrape` return the page's console messages, per-request resource timings, and the URLs the main document walked (`ScrapeResult.consoleLogs` / `networkLogs` / `redirectChain`), captured by the browser tiers (2-4). Each flag is independent and off by default — without it no listener is attached and nothing is buffered. Diagnostics use `DIAGNOSTICS_*` limits and redirect chains use independent `REDIRECT_*` limits; anything past a cap is dropped whole rather than truncated, and a capture failure leaves the field unset rather than failing the scrape.
 - Optional viewport screenshot: `screenshot: true` on `POST /scrape` returns a base64 JPEG of the viewport in `ScrapeResult.screenshot`, captured by the browser tiers (2-4) immediately before the HTML read so image and markup describe the same moment. Off by default; a stock request attaches nothing and does no extra work. Settle wait, capture timeout, JPEG quality, and maximum image size are bounded and tunable via `SCREENSHOT_*`, and a capture failure leaves the field unset rather than failing the scrape.
 
 ### Fixed

README.md

@@ -143,15 +143,15 @@ cookie isn't portable. For those indexers, enable TRAWL's forward proxy and add
 Prowlarr as an **HTTP proxy**:
 
 ```env
-MITM_PROXY_ENABLED=true
-MITM_PROXY_PORT=8192
-MITM_PROXY_CA_DIR=/data/proxy-ca   # persist the CA (mount a volume)
-MITM_PROXY_MAX_TIER=4              # cap escalation (e.g. 3 to stay off residential)
-MITM_PROXY_ALWAYS_SCRAPE=false     # opt in to bypass the proxy's direct Tier 0 probe
+MITM_ENABLED=true
+MITM_PORT=8192
+MITM_CA_DIR=/data/proxy-ca   # persist the CA (mount a volume)
+MITM_MAX_TIER=4              # cap escalation (e.g. 3 to stay off residential)
+MITM_ALWAYS_SCRAPE=false     # opt in to bypass the proxy's direct Tier 0 probe
 ```
 
 By default the listener binds `0.0.0.0` so clients on a Docker bridge network can reach
-it; set `MITM_PROXY_HOST=127.0.0.1` to restrict it to loopback on a bare-metal host.
+it; set `MITM_HOST=127.0.0.1` to restrict it to loopback on a bare-metal host.
 
 1. Install the proxy's CA into the client's trust store so it accepts the per-host certs:
    `curl http://<trawl-host>:8191/proxy-ca.crt` → add to the Prowlarr container's CA store
@@ -175,7 +175,7 @@ supported traffic, limitations, CA installation, and client examples.
 ### Installing the proxy CA certificate
 
 The proxy self-generates a root CA on first run. Its certificate and private key are persisted
-under `MITM_PROXY_CA_DIR` (default `/data/proxy-ca`). Per-host certificates are minted and cached
+under `MITM_CA_DIR` (default `/data/proxy-ca`). Per-host certificates are minted and cached
 in memory while TRAWL runs; they do not need separate installation because they are signed by the
 persistent root. Every client that uses the proxy must trust that root. Without it, HTTPS fails with
 `ERR_CERT_AUTHORITY_INVALID` (browsers) or `PKIX path building failed` (Java).
@@ -297,7 +297,7 @@ Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like "*TRAWL
 
 Every installation method has a symmetric removal path. Search your trust store for
 `TRAWL MITM Proxy CA` (the CA's CN) and delete that entry. The CA certificate and key also live at
-`<MITM_PROXY_CA_DIR>/ca.crt` and `ca.key` on the TRAWL host. Deleting either causes TRAWL to
+`<MITM_CA_DIR>/ca.crt` and `ca.key` on the TRAWL host. Deleting either causes TRAWL to
 generate a new root on its next start, so existing clients must install the new certificate.
 
 ## Tiers
@@ -393,9 +393,9 @@ for pool and mounted-file examples.
 | `BROWSER_POOL_SIZE`              | `3`                      | Warm Camoufox Firefox instances                                                     |
 | `BROWSER_ACQUIRE_TIMEOUT_MS`     | `15000`                  | How long `acquire()` polls for a free browser before HTTP 429 is returned           |
 | `BROWSER_RECYCLE_AFTER_CONTEXTS` | `8`                      | Rolling-replace after this many Tier 3/4 contexts; set `0` to disable               |
-| `BROWSER_CONTENT_PROCESSES`      | `2`                      | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU  |
-| `SESSION_TTL_SECONDS`            | `3600`                   | Redis session cache TTL (seconds)                                                   |
-| `REDIS_URL`                      | `redis://localhost:6379` | Redis connection string                                                             |
+| `BROWSER_MAX_CONTENT_PROCESSES`  | `2`                      | Cap Firefox content processes per browser (`dom.ipc.processCount`); lowers RAM/CPU  |
+| `REDIS_SESSION_TTL_SECONDS`      | `3600`                   | Redis session cache TTL (seconds)                                                   |
+| `REDIS_URL`                      | —                        | Redis connection string; empty or unset disables the session cache                   |
 | `REDIS_CONNECT_TIMEOUT_MS`       | `5000`                   | Maximum time for each Redis connection attempt                                      |
 | `REDIS_RETRY_DELAY_MS`           | `5000`                   | Delay before reconnecting after startup failure; `0` disables retry                 |
 | `PROXY_URL`                      | —                        | Optional Tier 3 HTTP or SOCKS5 proxy, or comma-separated pool                       |
@@ -404,13 +404,17 @@ for pool and mounted-file examples.
 | `RESIDENTIAL_PROXY_LIST_FILE`    | —                        | File containing one Tier 4 proxy URL per line                                       |
 | `STT_URL`                        | —                        | Local Whisper endpoint for reCAPTCHA (optional)                                     |
 | `PORT`                           | `8191`                   | API listen port                                                                     |
-| `MITM_PROXY_ENABLED`             | `false`                  | Enable the challenge-bypassing HTTP/HTTPS proxy                                     |
-| `MITM_PROXY_PORT`                | `8192`                   | Forward-proxy listen port                                                           |
-| `MITM_PROXY_HOST`                | `0.0.0.0`                | Bind address; `127.0.0.1` for loopback-only                                         |
-| `MITM_PROXY_CA_DIR`              | `/data/proxy-ca`         | Persistent root CA certificate and private-key directory                            |
-| `MITM_PROXY_MAX_TIER`            | `4`                      | Cap escalation used by the proxy (e.g. `3` to stay off residential)                 |
-| `MITM_PROXY_ALWAYS_SCRAPE`       | `false`                  | Skip proxy Tier 0; disables the direct media/large-file streaming path               |
-| `MITM_PROXY_DEBUG`               | `false`                  | Log one line per proxied request (errors are always logged)                         |
+| `MITM_ENABLED`                   | `false`                  | Enable the challenge-bypassing HTTP/HTTPS proxy                                     |
+| `MITM_PORT`                      | `8192`                   | Forward-proxy listen port                                                           |
+| `MITM_HOST`                      | `0.0.0.0`                | Bind address; `127.0.0.1` for loopback-only                                         |
+| `MITM_CA_DIR`                    | `/data/proxy-ca`         | Persistent root CA certificate and private-key directory                            |
+| `MITM_MAX_TIER`                  | `4`                      | Cap escalation used by the proxy (e.g. `3` to stay off residential)                 |
+| `MITM_ALWAYS_SCRAPE`             | `false`                  | Skip proxy Tier 0; disables the direct media/large-file streaming path               |
+| `MITM_DEBUG`                     | `false`                  | Log one line per proxied request (errors are always logged)                         |
+
+Upgrading from an earlier release requires renaming several environment variables. See the
+[configuration migration guide](apps/docs/deployment/configuration-migration.md) for the complete
+old-to-new mapping and Redis opt-in behavior.
 
 ## Stack
 

apps/api/Dockerfile

@@ -57,8 +57,8 @@ ARG TARGETARCH
 # MacIntel -> "macos"), and Camoufox spoofs that OS's font list with it. Dropping those
 # bundles therefore leaves a browser advertising fonts whose files are gone: every glyph
 # renders as a tofu box, and the fingerprint is self-inconsistent for anti-bot scoring.
-# Set KEEP_SPOOFED_OS_FONTS=1 to keep them (+891MB) when rendered output matters.
-ARG KEEP_SPOOFED_OS_FONTS=0
+# Set CAMOUFOX_KEEP_SPOOFED_OS_FONTS=1 to keep them (+891MB) when rendered output matters.
+ARG CAMOUFOX_KEEP_SPOOFED_OS_FONTS=0
 RUN --mount=type=secret,id=GITHUB_TOKEN,env=GITHUB_TOKEN \
     case "$TARGETARCH" in \
       amd64) ZIP="camoufox-152.0.4-beta.28-lin.x86_64.zip"; SHA256="924f3109ccd6d47cd6a0384d67a345fadf975d48b6319f8dbbd5954c588982bd" ;; \
@@ -73,7 +73,7 @@ RUN --mount=type=secret,id=GITHUB_TOKEN,env=GITHUB_TOKEN \
     rm /tmp/camoufox.zip && \
     printf '{"version":"152.0.4","release":"beta.28"}\n' > /opt/camoufox/version.json && \
     chmod -R 755 /opt/camoufox && \
-    if [ "$KEEP_SPOOFED_OS_FONTS" != "1" ]; then \
+    if [ "$CAMOUFOX_KEEP_SPOOFED_OS_FONTS" != "1" ]; then \
       rm -rf /opt/camoufox/fonts/macos /opt/camoufox/fonts/windows; \
     fi
 

apps/api/Dockerfile.baseline

@@ -57,8 +57,8 @@ ARG TARGETARCH
 # MacIntel -> "macos"), and Camoufox spoofs that OS's font list with it. Dropping those
 # bundles therefore leaves a browser advertising fonts whose files are gone: every glyph
 # renders as a tofu box, and the fingerprint is self-inconsistent for anti-bot scoring.
-# Set KEEP_SPOOFED_OS_FONTS=1 to keep them (+891MB) when rendered output matters.
-ARG KEEP_SPOOFED_OS_FONTS=0
+# Set CAMOUFOX_KEEP_SPOOFED_OS_FONTS=1 to keep them (+891MB) when rendered output matters.
+ARG CAMOUFOX_KEEP_SPOOFED_OS_FONTS=0
 RUN --mount=type=secret,id=GITHUB_TOKEN,env=GITHUB_TOKEN \
     case "$TARGETARCH" in \
       amd64) ZIP="camoufox-152.0.4-beta.28-lin.x86_64.zip"; SHA256="924f3109ccd6d47cd6a0384d67a345fadf975d48b6319f8dbbd5954c588982bd" ;; \
@@ -73,7 +73,7 @@ RUN --mount=type=secret,id=GITHUB_TOKEN,env=GITHUB_TOKEN \
     rm /tmp/camoufox.zip && \
     printf '{"version":"152.0.4","release":"beta.28"}\n' > /opt/camoufox/version.json && \
     chmod -R 755 /opt/camoufox && \
-    if [ "$KEEP_SPOOFED_OS_FONTS" != "1" ]; then \
+    if [ "$CAMOUFOX_KEEP_SPOOFED_OS_FONTS" != "1" ]; then \
       rm -rf /opt/camoufox/fonts/macos /opt/camoufox/fonts/windows; \
     fi