Proxy config not honored — non-Bing engines time out behind an HTTP proxy (USE_PROXY/PROXY_URL/HTTPS_PROXY ineffective)

[Jonasyu0723]

Summary

On a network that requires an HTTP proxy for outbound access, wigolo does not route its engine scraping or page fetching through the proxy, even when configured via USE_PROXY/PROXY_URL, the config file, or standard HTTPS_PROXY/HTTP_PROXY/ALL_PROXY env vars.

Result: all non-Bing engines time out, search silently degrades to Bing-only, and fetching pages that are only reachable through the proxy returns empty content.

Environment

• wigolo 0.2.1 (npm global install, /Users/hansel/.local/bin/wigolo)
• Node v22.23.0
• macOS 26.5.1 (arm64)
• Proxy: HTTP proxy at 127.0.0.1:9674 (macOS system proxy; scutil --proxy shows HTTPEnable=1 HTTPProxy=127.0.0.1 HTTPPort=9674, same for HTTPS)
• Local SearXNG sidecar bootstrapped successfully (~/.wigolo/searxng, venv on Python 3.11 — note: system python3 is 3.9, which fails the curl_cffi>=0.16 requirement; worked around with a shim)

Proxy works outside wigolo

$ curl -x http://127.0.0.1:9674 -s -o /dev/null -w "%{http_code}" https://www.google.com
302
$ curl -x http://127.0.0.1:9674 -s -o /dev/null -w "%{http_code}" https://duckduckgo.com
200

Direct (no proxy) access to those hosts returns 000 on this network, confirming the proxy is required and functional.

What wigolo does instead

Search:

$ wigolo search "python asyncio best practices" --max-results 3
Search: "python asyncio best practices" (3 results, 43474ms, engines: bing)

  Warning: Multi-engine search is unavailable; using fallback engines (lower quality). Reason: unknown.
  To retry: `npx wigolo warmup --force`. For details: `npx wigolo doctor`.

Debug log (LOG_LEVEL=debug LOG_FORMAT=text):

INFO  [search] orchestrator dispatching engines vertical=general engineCount=4 hasDateBound=false
DEBUG [search] scraping bing query=...
DEBUG [search] scraping duckduckgo query=...
DEBUG [search] querying wikipedia opensearch query=... language=en
DEBUG [search] querying marginalia query=...
INFO  [search] pool degraded below floor, running recovery wave primaryHealthy=1 dispatched=4 recoveryEngines=mojeek
DEBUG [search] scraping mojeek query=...
WARN  [search] multi-query engine search failed engine=duckduckgo error=The operation was aborted due to timeout
WARN  [search] multi-query engine search failed engine=duckduckgo query="... guide" error=The operation was aborted due to timeout
INFO  [hybrid] fallback signal fired; running searxng signals=all_engines_failed

Fetch of a proxy-only-reachable page:

$ wigolo fetch "https://www.google.com" --json
title: ""    markdown length: 0

Non-proxied, locally reachable pages (e.g. Chinese sites) fetch fine — so fetch itself works; it just does not use the proxy.

Configuration attempts that do not fix it

1. Config file / headless set

   wigolo config --set USE_PROXY=true
   wigolo config --set PROXY_URL=http://127.0.0.1:9674
   

   Persisted in ~/.wigolo/config.json ({"settings": {..., "searchBackend": "hybrid", "llmProvider": "ollama"}}), and wigolo config --plain reads back PROXY_URL. Engines still time out.

2. Daemon started with proxy env

   USE_PROXY=true PROXY_URL=http://127.0.0.1:9674 wigolo serve
   

   wigolo health → Status: healthy, Search engine: active, searxng: active. Search via CLI still shows Bing-only.

3. Standard proxy env vars on the CLI invocation

   HTTPS_PROXY=http://127.0.0.1:9674 HTTP_PROXY=http://127.0.0.1:9674 \
   ALL_PROXY=socks5://127.0.0.1:9674 wigolo search "test query" --max-results 3
   

   Same outcome (Bing-only, ~43s per query).

4. hybrid + explicit sidecar URL

   WIGOLO_SEARCH=hybrid SEARXNG_URL=http://127.0.0.1:8888 wigolo search ... --language=zh
   

   Logs show the hybrid fallback firing correctly (provider":"hybrid","searxngAvailable":true, then fallback signal fired; running searxng), but the underlying engines still time out, so the fallback cannot help.

Expected behavior

Either:

• USE_PROXY/PROXY_URL (and/or standard HTTPS_PROXY/HTTP_PROXY) is honored for engine HTTP requests, the SearXNG sidecar, and browser-engine fetches; or
• wigolo explicitly reports that proxy configuration is unsupported on these code paths (so users can stop hunting for the right knob).

Impact

On proxy-required networks (China, corporate egress proxies, etc.) the search feature is effectively Bing-only with 20–120 s per query and noticeably worse long-tail/Chinese recall, and any proxy-only-reachable page cannot be fetched — which removes most of the tool's value for exactly the users most likely to want a local, keyless web layer.

Happy to test a patch or provide more logs (LOG_LEVEL=debug captures are available on request).