Releases: Edgaras0x4E/web-loader-engine
Releases · Edgaras0x4E/web-loader-engine
v0.1.4
User Agent Rotation & Browser Log Control - Added configurable user agents and a way to silence warnings.
- New
USER_AGENT_ROTATIONenv var with strategiesoff(default),round_robin,random- rotates per request - Provide the pool via
USER_AGENT_POOL(inline,|- or newline-separated) orUSER_AGENT_POOL_FILE(path to a file, one UA per line,#comments supported). The file takes precedence when both are set DEFAULT_USER_AGENToverrides the hardcoded default used when rotation is off and no header is setx-user-agentheader now accepts special values:rotateforces rotation even whenUSER_AGENT_ROTATION=off, anddefaultforces the configured default- Precedence: explicit header → rotation (if enabled) → configured default. Empty pool safely falls back to the default with a warning at startup
- New
BROWSER_LOG_LEVELenv var (defaulterror) silences chromiumoxide's noisyWS Invalid messagewarnings emitted when Chromium sends CDP events the driver doesn't yet model. Acceptsoff,error,warn,info,debug,trace- operates independently ofRUST_LOG
v0.1.3
Chromium Egress Proxy Support - Chromium now honors HTTPS_PROXY/HTTP_PROXY/NO_PROXY from the environment so the browser's outbound traffic can be routed through an egress proxy.
- On launch, if
HTTPS_PROXY(orHTTP_PROXYas fallback) is set, Chromium is started with--proxy-server=<url> - If
NO_PROXYis set, its value is translated to Chrome's bypass-list syntax and passed via--proxy-bypass-list=<list>(commas → semicolons,*.domain→.domain) - When no proxy env vars are set, behavior is unchanged — dev/local runs need no configuration
- The Rust HTTP client (reqwest) already honors these vars natively, so direct HTTP fetches and browser fetches now share the same egress path
v0.1.2
Screenshot Delivery Fix - Screenshot URLs returned by the API are now actually reachable.
- Fixed issue where
/loadresponses advertised ascreenshot_urlthat returned 404 when fetched - Saved screenshots are now served directly from the configured
SCREENSHOT_DIR - Safe by design: path-traversal attempts (e.g.
/screenshots/../etc/passwd) return 404 - Respects the same API key authentication as the rest of the API when one is configured
v0.1.1
Browser Pool Resilience - Fixed critical issue where dead browser connections would cause requests to hang indefinitely.
- Added automatic browser health detection with 5-second timeout on page creation
- Implemented connection error detection for
Ws(AlreadyClosed)and related WebSocket errors - Auto-recovery: dead browsers are now automatically recreated on connection failure
- Request-level retry logic (up to 3 retries) for transient connection errors
- Health endpoint now exposes
healthystatus andrecreation_countfor monitoring
Health response now includes:
{
"status": "ok",
"version": "0.1.1",
"browser_pool": {
"available": 10,
"total": 10,
"healthy": true,
"recreation_count": 1
}
}Monitor recreation_count increasing to track browser recovery events.