From 8671f4fb6c5ed105f5eb0b84bd46ca4f875af8ee Mon Sep 17 00:00:00 2001 From: Jason McCartney Date: Sat, 17 Oct 2026 18:15:18 -0700 Subject: [PATCH] Redact credentials from requests sent to external backends Add opt-in egress redaction. With security.redactEgress (default false), the gateway replaces credentials in request text with [secret] before forwarding to a backend whose baseUrl host is outside the local network. backends..redact overrides the decision in either direction. - Patterns ported from Tessera's Steward redactor, plus URL passwords (mongodb+srv://, postgres://, redis://, ...); sk- already covers OpenRouter sk-or-v1- keys. Labelled values keep their label, need 6+ characters, and skip unquoted values that read as code. - Applied once, after protocol translation, in a fetchModelUpstream wrapper used by every JSON model route, and to the OpenRouter/Replicate audio and video provider adapters. Chat, Responses and Anthropic bridges, streaming or not, share it. - Internal: loopback, RFC1918, link-local, 100.64/10, fc00::/7, .ts.net, .local, .internal and single-label hosts, and federated LLooM gateways. - Embeddings and transcriptions are never rewritten. The caller's body is not mutated, and only a per-request count is logged. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 + SECURITY.md | 1 + docs/architecture.md | 58 ++++ package.json | 4 +- src/config.mjs | 9 + src/egress-redaction.mjs | 239 ++++++++++++++++ src/server.mjs | 75 +++--- test/egress-redaction.test.mjs | 479 +++++++++++++++++++++++++++++++++ 8 files changed, 830 insertions(+), 37 deletions(-) create mode 100644 src/egress-redaction.mjs create mode 100644 test/egress-redaction.test.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 77f87ff..6b8cf8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ All notable changes to LLooM will be documented in this file. The format follows ### Added +- Opt-in egress redaction (`security.redactEgress`, per-backend `redact`). Credentials such as API keys, tokens, PEM private keys, JWTs, URL passwords and labelled secrets in request text are replaced with `[secret]` before the gateway forwards a Chat Completions, Responses, Anthropic Messages or JSON media request to a backend outside the local network. Local, private, tailnet and federated LLooM backends are left alone, and embeddings are never rewritten. See [egress redaction](docs/architecture.md#egress-redaction). + - Built-in reservations and a job log (`lloom schedule`, `reserve`, `renew`, `release`, `job`; `/gateway/schedule`). Expiring leases on configured resources run route and suspend actions, possibly on other cluster nodes, and undo them in reverse with health-gated restore. There are queues, renewal and heartbeat-extended leases, stalled-job detection, and an event feed for boards such as Tessera. Schedule refusals now reach the CLI as redacted messages. - Atlas SparkQwen RC16 two-Spark recipe (`linux-nvidia-dgx-spark-2x-qwen38-flash-next-atlas`, profile `8x262k`) with a fail-closed pin gate and a digest-only image installer, and explicit per-node `${fabricHca}` settings for distributed recipes. Both Atlas recipes pin the published images with exact rail selection; missing settings and ambiguous discovery fail clearly. diff --git a/SECURITY.md b/SECURITY.md index e35b57c..d1f0400 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,6 +24,7 @@ The maintainers will acknowledge reports as soon as practical, validate the issu - A signature proves which key signed a recipe pack. Trusting keys served by the same community host is equivalent to trusting that host and its TLS connection; use explicit local trusted keys for stronger publisher pinning. - Development keys and the checked-in public seed key are not production trust roots. - Model weights and external runtimes have their own licenses and security posture. LLooM does not make untrusted model code safe. +- Requests routed to cloud providers carry whatever text the client sent. Set `security.redactEgress` to scrub common credential formats from prompts before they leave the local network. It is a pattern-based safety net, not a guarantee; see [egress redaction](docs/architecture.md#egress-redaction). - Browser management requests must come from the gateway's own origin. First-run setup additionally requires a local session token and applies only reviewed plans. Keep the setup terminal and tokens private. See [the browser security boundary](docs/browser-experience.md#local-security). - The public community MVP is read-only by design. Proposals arrive through reviewed pull requests; anonymous recipe and benchmark uploads are disabled in production. - Remote community feeds must use HTTPS and a locally pinned public signing key. Do not treat a signing key downloaded from the same remote host as an independent trust root. diff --git a/docs/architecture.md b/docs/architecture.md index 19e9744..8ab680a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -87,12 +87,70 @@ See [OpenRouter provider routing](https://openrouter.ai/docs/guides/routing/prov | `security.allowWildcardCors` | `false` | Non-loopback CORS origin is `null` unless enabled | | `security.apiKeys` | `["sk-lloom-local"]` in default config | Inference Bearer / `x-api-key` values | | `security.adminApiKeys` | `[]` | When non-empty, admin routes require these keys even on loopback | +| `security.redactEgress` | `false` | Scrub credentials from request text sent to external backends (see below) | | `logging.requestLog` | `false` | Append NDJSON request lines to `~/.lloom/logs/requests.ndjson` (or `LLOOM_REQUEST_LOG=1`) | | `logging.metricsPersistence` | `true` | Persist aggregate gateway metrics across restarts (or disable with `LLOOM_METRICS_PERSISTENCE=0`) | | `logging.metricsPath` | `null` | Override `~/.lloom/metrics-history.json` (or set `LLOOM_METRICS_PATH`) | Prefer binding `server.host` to `127.0.0.1`. If you expose the gateway on a LAN or public interface, set `allowNonLoopbackBind`, strong API keys, keep `allowRemoteAdmin` false unless you intentionally operate remote admin, and treat install/start endpoints as privileged. `GET /gateway/security` returns public auth metadata (no secrets) for dashboards. +### Egress redaction + +With `security.redactEgress: true`, the gateway replaces credentials in a +request's text with `[secret]` before forwarding it to an external backend, +such as DeepSeek or OpenRouter. It is off by default. + +```json +{ + "security": { "redactEgress": true }, + "backends": { + "deepseek": { "type": "openai", "baseUrl": "https://api.deepseek.com/v1", "apiKeyEnv": "DEEPSEEK_API_KEY" }, + "lab-proxy": { "type": "openai", "baseUrl": "https://llm.lab.example/v1", "redact": false } + } +} +``` + +**Which backends.** A backend is external unless its `baseUrl` host is +loopback (`127.0.0.0/8`, `::1`, `localhost`), RFC 1918 (`10/8`, `172.16/12`, +`192.168/16`), link-local, CGNAT/tailnet (`100.64.0.0/10`), IPv6 ULA +(`fc00::/7`), a name ending in `.ts.net`, `.local`, `.internal` or +`.localhost`, or an unqualified single-label name such as `ennspark03`. The +host is not resolved, so a public name that points at a private address still +counts as external. Backends with `audioProvider` or `videoProvider` are +always external, because those adapters call OpenRouter or Replicate directly. +A federated LLooM gateway (`lloom-node-*`) is internal: the downstream gateway +applies its own policy when it egresses. `backends..redact: true | false` +overrides the automatic decision in either direction, and `true` applies even +when `security.redactEgress` is off. + +**What is scrubbed.** The outgoing JSON body after protocol translation, so +Chat Completions, Responses and Anthropic Messages (both bridged to Chat +Completions), streaming or not, share one code path. The scrubbed fields are +message `content` (strings and text parts), assistant `tool_calls` arguments, +`role: tool` results, replayed `reasoning_content`, and `prompt`, `input`, +`instructions` and `system`, which also covers JSON image, video, speech and +audio-generation prompts. Image, audio and file parts are left as they are. +Embeddings are never rewritten, because changing their input changes the +vectors, and neither are transcriptions, whose payload is audio. Multipart +uploads (image edits and voice-clone speech) and web search queries are not +scrubbed. + +**Patterns.** PEM private keys; `sk-` keys, including Anthropic, OpenAI +project and OpenRouter `sk-or-v1-` keys; `lloom_*` keys; GitHub, Slack, AWS +access key, Google API, Replicate, Hugging Face, Jina and xAI tokens; JWTs; +the password in `scheme://user:password@host` URLs such as `mongodb+srv://`, +`postgres://` and `redis://`; and labelled values such as `Bearer …`, +`API_KEY=…`, `password: …` or `"client_secret": "…"`, where the label is kept. +A labelled value needs at least six characters, so `token budget: 500` stays +intact. Unquoted values that read as code, such as `getToken()`, +`process.env.API_KEY`, `$TOKEN` or a word without digits or symbols, are left +alone. This catches common credential formats and is not a guarantee: a +letters-only password after an unquoted label passes through. + +The caller's request is never modified. Matched values are never logged; when +anything is replaced, the gateway logs one line per request with the number of +replacements and the backend ID. + ## API Surface LLooM currently fronts these local contracts: diff --git a/package.json b/package.json index 9073cee..028bb58 100644 --- a/package.json +++ b/package.json @@ -65,7 +65,7 @@ "check": "npm run check:cluster && npm run check:js && npm run check:entity-stagger && npm run check:python && npm run check:shell && npm run check:spark-deploy", "check:cluster": "node --check src/cluster.mjs && node --check test/cluster.test.mjs && node --check test/cluster-availability.test.mjs && node --check test/cluster-readiness.test.mjs && node --check src/cluster-deployment.mjs && node --check src/cluster-deployment-cli.mjs && node --check src/ssh-deployment-transport.mjs && node --check test/cluster-deployment.test.mjs && node --check test/cluster-deployment-cli.test.mjs && node --check test/cluster-deployment-e2e.test.mjs", "check:entity-stagger": "node --check scripts/entity-stagger-benchmark.mjs && node --check test/entity-stagger-benchmark.test.mjs", - "check:js": "node --check bin/lloom.mjs && node --check bin/lloom-host.mjs && node --check src/backend-catalog.mjs && node --check src/benchmarks.mjs && node --check src/bootstrap.mjs && node --check src/client-integrations.mjs && node --check src/community-client.mjs && node --check src/community-site.mjs && node --check src/config.mjs && node --check src/dashboard.mjs && node --check src/doctor.mjs && node --check src/host-memory.mjs && node --check src/host-server.mjs && node --check src/init.mjs && node --check src/installer.mjs && node --check src/interchange.mjs && node --check src/machine-profile.mjs && node --check src/managed-environment.mjs && node --check src/model-acquisition.mjs && node --check src/model-intake.mjs && node --check src/model-removal.mjs && node --check src/onboarding.mjs && node --check src/recipe-index.mjs && node --check src/recipe-pack-export.mjs && node --check src/recipe-pack.mjs && node --check src/registry.mjs && node --check src/route-control.mjs && node --check src/recipes.mjs && node --check src/resource-fit.mjs && node --check src/request-scheduler.mjs && node --check src/rate-limit.mjs && node --check src/runtime-manager.mjs && node --check src/runtime-policy.mjs && node --check src/server.mjs && node --check src/deployment-fence.mjs && node --check test/deployment-fence.test.mjs && node --check src/schedule.mjs && node --check src/schedule-cli.mjs && node --check test/schedule.test.mjs && node --check test/schedule-integration.test.mjs && node --check src/setup.mjs && node --check src/setup-status.mjs && node --check src/process-control.mjs && node --check src/security.mjs && node --check src/standby.mjs && node --check src/tts-catalog.mjs && node --check src/voice-profiles.mjs && node --check src/protocol/text.mjs && node --check src/protocol/reasoning-normalize.mjs && node --check src/protocol/reasoning-effort.mjs && node --check src/protocol/responses.mjs && node --check src/protocol/anthropic.mjs && node --check src/protocol/sse.mjs && node --check src/protocol/stream-anthropic.mjs && node --check src/protocol/stream-responses.mjs && node --check src/protocol/index.mjs && node --check scripts/apply-spark-route-catalog.mjs && node --check scripts/capture-live-metrics.mjs && node --check scripts/merge-live-metrics.mjs && node --check scripts/run-qualitative-chat-benchmark.mjs && node --check scripts/run-local-decision-benchmark.mjs && node --check scripts/check-community-deploy.mjs && node --check scripts/generate-client-configs.mjs && node --check scripts/check-interchange.mjs && node --check scripts/check-package.mjs && node --check scripts/release-lib.mjs && node --check scripts/build-release.mjs && node --check scripts/deploy-spark.mjs && node --check test/audio-generations.test.mjs && node --check test/community-host.test.mjs && node --check test/dashboard-status.test.mjs && node --check test/ds4fv-recipe.test.mjs && node --check test/dspark-hotfixes.test.mjs && node --check test/glm53-exl3-recipe.test.mjs && node --check test/host-memory.test.mjs && node --check test/metrics-persistence.test.mjs && node --check test/model-acquisition.test.mjs && node --check test/model-failover.test.mjs && node --check test/qwen38-vllm-recipe.test.mjs && node --check test/reasoning-normalize.test.mjs && node --check test/reasoning-effort.test.mjs && node --check test/resource-fit.test.mjs && node --check test/route-control.test.mjs && node --check test/spark-route-catalog.test.mjs && node --check test/runtime-watchdog.test.mjs && node --check test/tts-catalog.test.mjs && node --check test/voice-profiles.test.mjs && node --check test/chatterbox-recipe.test.mjs && node --check test/hear-recipe.test.mjs && node --check test/smoke.mjs && node --check src/runtime-supervisor.mjs && node --check test/runtime-supervisor.test.mjs && node --check test/comfyui-media.test.mjs && node --check test/spark-audio-recipes.test.mjs && node --check src/runtime-residency.mjs && node --check test/ltx-pipelines.test.mjs && node --check test/ltx-pipelines-recipe.test.mjs && node --check src/video-contracts.mjs && node --check src/dashboard-scene.mjs && node --check src/dashboard-presence.mjs && node --check src/dashboard-presence-client.mjs && node --check src/first-run.mjs && node --check src/first-run-page.mjs && node --check src/browser-setup.mjs && node --check src/runtime-preferences.mjs && node --check src/installation-jobs.mjs && node --check src/model-installation.mjs && node --check src/dashboard-installation.mjs && node --check src/runtime-memory-safety.mjs && node --check src/dashboard-memory.mjs && node --check src/runtime-memory-usage.mjs && node --check src/config-profiles.mjs && node --check test/config-profiles.test.mjs && node --check src/runtime-policy-config.mjs && node --check test/runtime-policy-config.test.mjs && node --check src/audio-providers.mjs && node --check src/protocol/openrouter-provider.mjs && node --check test/audio-providers.test.mjs && node --check test/openrouter-provider.test.mjs && node --check backends/atlas-sparkglm/verify-pins.mjs && node --check test/atlas-sparkglm-recipe.test.mjs && node --check backends/atlas-sparkqwen/verify-pins.mjs && node --check test/atlas-sparkqwen-recipe.test.mjs && node --check src/head-preparation.mjs && node --check src/head-transfer.mjs && node --check src/service-control.mjs && node --check src/gateway-relay.mjs && node --check src/client-retarget.mjs && node --check src/head-promotion.mjs && node --check src/head-promotion-transfer.mjs && node --check src/gateway-client.mjs && node --check test/cli-diagnostics.test.mjs && node --check src/service-install.mjs && node --check test/service-install.test.mjs && node --check src/cluster-deployment.mjs && node --check src/cluster-deployment-cli.mjs && node --check src/ssh-deployment-transport.mjs && node --check src/node-release-agent.mjs && node --check src/node-gateway-adapter.mjs && node --check src/node-agent-cli.mjs && node --check test/cluster-deployment.test.mjs && node --check test/cluster-deployment-cli.test.mjs && node --check test/cluster-deployment-e2e.test.mjs && node --check test/node-release-agent.test.mjs && node --check test/node-gateway-adapter.test.mjs && node --check test/node-gateway-adapter-e2e.test.mjs && node --check test/deployment-bundle.test.mjs && node --check test/node-agent-cli.test.mjs && node --check test/ssh-deployment-transport.test.mjs && node --check scripts/build-deployment-bundle.mjs", + "check:js": "node --check bin/lloom.mjs && node --check bin/lloom-host.mjs && node --check src/backend-catalog.mjs && node --check src/benchmarks.mjs && node --check src/bootstrap.mjs && node --check src/client-integrations.mjs && node --check src/community-client.mjs && node --check src/community-site.mjs && node --check src/config.mjs && node --check src/dashboard.mjs && node --check src/doctor.mjs && node --check src/host-memory.mjs && node --check src/host-server.mjs && node --check src/init.mjs && node --check src/installer.mjs && node --check src/interchange.mjs && node --check src/machine-profile.mjs && node --check src/managed-environment.mjs && node --check src/model-acquisition.mjs && node --check src/model-intake.mjs && node --check src/model-removal.mjs && node --check src/onboarding.mjs && node --check src/recipe-index.mjs && node --check src/recipe-pack-export.mjs && node --check src/recipe-pack.mjs && node --check src/registry.mjs && node --check src/route-control.mjs && node --check src/recipes.mjs && node --check src/resource-fit.mjs && node --check src/request-scheduler.mjs && node --check src/rate-limit.mjs && node --check src/runtime-manager.mjs && node --check src/runtime-policy.mjs && node --check src/server.mjs && node --check src/deployment-fence.mjs && node --check test/deployment-fence.test.mjs && node --check src/schedule.mjs && node --check src/schedule-cli.mjs && node --check test/schedule.test.mjs && node --check test/schedule-integration.test.mjs && node --check src/setup.mjs && node --check src/setup-status.mjs && node --check src/process-control.mjs && node --check src/security.mjs && node --check src/egress-redaction.mjs && node --check test/egress-redaction.test.mjs && node --check src/standby.mjs && node --check src/tts-catalog.mjs && node --check src/voice-profiles.mjs && node --check src/protocol/text.mjs && node --check src/protocol/reasoning-normalize.mjs && node --check src/protocol/reasoning-effort.mjs && node --check src/protocol/responses.mjs && node --check src/protocol/anthropic.mjs && node --check src/protocol/sse.mjs && node --check src/protocol/stream-anthropic.mjs && node --check src/protocol/stream-responses.mjs && node --check src/protocol/index.mjs && node --check scripts/apply-spark-route-catalog.mjs && node --check scripts/capture-live-metrics.mjs && node --check scripts/merge-live-metrics.mjs && node --check scripts/run-qualitative-chat-benchmark.mjs && node --check scripts/run-local-decision-benchmark.mjs && node --check scripts/check-community-deploy.mjs && node --check scripts/generate-client-configs.mjs && node --check scripts/check-interchange.mjs && node --check scripts/check-package.mjs && node --check scripts/release-lib.mjs && node --check scripts/build-release.mjs && node --check scripts/deploy-spark.mjs && node --check test/audio-generations.test.mjs && node --check test/community-host.test.mjs && node --check test/dashboard-status.test.mjs && node --check test/ds4fv-recipe.test.mjs && node --check test/dspark-hotfixes.test.mjs && node --check test/glm53-exl3-recipe.test.mjs && node --check test/host-memory.test.mjs && node --check test/metrics-persistence.test.mjs && node --check test/model-acquisition.test.mjs && node --check test/model-failover.test.mjs && node --check test/qwen38-vllm-recipe.test.mjs && node --check test/reasoning-normalize.test.mjs && node --check test/reasoning-effort.test.mjs && node --check test/resource-fit.test.mjs && node --check test/route-control.test.mjs && node --check test/spark-route-catalog.test.mjs && node --check test/runtime-watchdog.test.mjs && node --check test/tts-catalog.test.mjs && node --check test/voice-profiles.test.mjs && node --check test/chatterbox-recipe.test.mjs && node --check test/hear-recipe.test.mjs && node --check test/smoke.mjs && node --check src/runtime-supervisor.mjs && node --check test/runtime-supervisor.test.mjs && node --check test/comfyui-media.test.mjs && node --check test/spark-audio-recipes.test.mjs && node --check src/runtime-residency.mjs && node --check test/ltx-pipelines.test.mjs && node --check test/ltx-pipelines-recipe.test.mjs && node --check src/video-contracts.mjs && node --check src/dashboard-scene.mjs && node --check src/dashboard-presence.mjs && node --check src/dashboard-presence-client.mjs && node --check src/first-run.mjs && node --check src/first-run-page.mjs && node --check src/browser-setup.mjs && node --check src/runtime-preferences.mjs && node --check src/installation-jobs.mjs && node --check src/model-installation.mjs && node --check src/dashboard-installation.mjs && node --check src/runtime-memory-safety.mjs && node --check src/dashboard-memory.mjs && node --check src/runtime-memory-usage.mjs && node --check src/config-profiles.mjs && node --check test/config-profiles.test.mjs && node --check src/runtime-policy-config.mjs && node --check test/runtime-policy-config.test.mjs && node --check src/audio-providers.mjs && node --check src/protocol/openrouter-provider.mjs && node --check test/audio-providers.test.mjs && node --check test/openrouter-provider.test.mjs && node --check backends/atlas-sparkglm/verify-pins.mjs && node --check test/atlas-sparkglm-recipe.test.mjs && node --check backends/atlas-sparkqwen/verify-pins.mjs && node --check test/atlas-sparkqwen-recipe.test.mjs && node --check src/head-preparation.mjs && node --check src/head-transfer.mjs && node --check src/service-control.mjs && node --check src/gateway-relay.mjs && node --check src/client-retarget.mjs && node --check src/head-promotion.mjs && node --check src/head-promotion-transfer.mjs && node --check src/gateway-client.mjs && node --check test/cli-diagnostics.test.mjs && node --check src/service-install.mjs && node --check test/service-install.test.mjs && node --check src/cluster-deployment.mjs && node --check src/cluster-deployment-cli.mjs && node --check src/ssh-deployment-transport.mjs && node --check src/node-release-agent.mjs && node --check src/node-gateway-adapter.mjs && node --check src/node-agent-cli.mjs && node --check test/cluster-deployment.test.mjs && node --check test/cluster-deployment-cli.test.mjs && node --check test/cluster-deployment-e2e.test.mjs && node --check test/node-release-agent.test.mjs && node --check test/node-gateway-adapter.test.mjs && node --check test/node-gateway-adapter-e2e.test.mjs && node --check test/deployment-bundle.test.mjs && node --check test/node-agent-cli.test.mjs && node --check test/ssh-deployment-transport.test.mjs && node --check scripts/build-deployment-bundle.mjs", "check:python": "python3 -m py_compile backends/dspark-vllm/apply-patch-pack.py backends/dspark-vllm/packs/miaai-dsv4flash-d1b76251-defaults/patches/*.py backends/dspark-vllm/packs/miaai-ds4fv-f5665e8/patches/*.py backends/dspark-vllm/packs/miaai-ds4fv-f5665e8/patches/vision_exp/*.py backends/qwen38-vllm/apply-ple-fp8-patch.py backends/qwen38-sglang/*.py backends/glm53-exl3/*.py backends/mlx-audio/lloom_audio_server.py backends/hear/lloom_hear_server.py backends/chatterbox/lloom_chatterbox_server.py patches/apply_mtplx_longctx_fix.py backends/comfyui-media/bridge/*.py backends/comfyui-media/graphs/*.py backends/comfyui-media/build/*.py backends/comfyui-media/install.py backends/comfyui-media/derive_h3_turbo.py backends/qwen-image-diffusers/*.py backends/ace-step-diffusers/*.py backends/ace-step-diffusers/tests/*.py backends/hear/test/test_hear_security.py backends/spark-audio/install.py backends/spark-audio/lloom_audio_cuda_server.py backends/spark-audio/test_lloom_audio_cuda_server.py backends/ltx-pipelines/*.py src/darwin-memory-usage.py", "check:shell": "bash -n backends/dspark-vllm/entrypoint.sh backends/dspark-vllm/runtime/build-hardened-image.sh backends/dspark-vllm/packs/miaai-dsv4flash-d1b76251-defaults/patches/*.sh backends/dspark-vllm/packs/miaai-ds4fv-f5665e8/apply-runtime.sh backends/dspark-vllm/packs/miaai-ds4fv-f5665e8/patches/*.sh backends/qwen38-vllm/entrypoint.sh backends/qwen38-sglang/entrypoint.sh backends/glm53-exl3/entrypoint.sh backends/mlx-audio/install.sh backends/hear/install.sh backends/chatterbox/install.sh clients/examples/lloom-claude clients/examples/lloom-codex clients/examples/lloom-hermes clients/examples/lloom-opencode clients/examples/lloom-zero scripts/remote-install-spark.sh scripts/remote-stage-spark-worker.sh backends/atlas-sparkglm/install.sh backends/atlas-sparkqwen/install.sh", "check:spark-deploy": "node --check test/spark-deploy.test.mjs && node test/spark-deploy.test.mjs", @@ -74,7 +74,7 @@ "community:check": "node scripts/check-community-deploy.mjs", "package:check": "node scripts/check-package.mjs", "test:adoption": "node test/resource-fit.test.mjs && node test/model-acquisition.test.mjs && node test/dashboard-status.test.mjs", - "test:unit": "node --test test/deployment-fence.test.mjs test/cluster-deployment.test.mjs test/cluster-deployment-cli.test.mjs test/cluster-deployment-e2e.test.mjs test/node-release-agent.test.mjs test/node-gateway-adapter.test.mjs test/node-gateway-adapter-e2e.test.mjs test/deployment-bundle.test.mjs test/node-agent-cli.test.mjs test/ssh-deployment-transport.test.mjs && node test/runtime-health-ownership.test.mjs && node test/long-prefill-transport.test.mjs && node test/sparkglm-managed.test.mjs && node test/protocol.test.mjs && node test/protocol-stream.test.mjs && node test/stream-metrics.test.mjs && node test/responses-codex.test.mjs && node test/security.test.mjs && node test/community-host.test.mjs && node test/ds4fv-recipe.test.mjs && node test/dspark-hotfixes.test.mjs && node test/glm53-exl3-recipe.test.mjs && node test/host-memory.test.mjs && node test/config-profiles.test.mjs && node test/runtime-policy.test.mjs && node --test test/runtime-policy-config.test.mjs && node --test test/request-scheduler.test.mjs test/adaptive-priority.test.mjs test/rate-limit.test.mjs test/runtime-queue.test.mjs test/foreground-priority.test.mjs test/federated-priority.test.mjs test/runtime-supervisor.test.mjs && node test/interchange.test.mjs && node test/server-resilience.test.mjs && node --test test/standby.test.mjs && node test/metrics-persistence.test.mjs && node test/model-failover.test.mjs && node test/qwen38-vllm-recipe.test.mjs && node test/reasoning-normalize.test.mjs && node test/reasoning-effort.test.mjs && node test/route-control.test.mjs && node test/spark-route-catalog.test.mjs && node test/chat-lane-canary.test.mjs && node test/runtime-watchdog.test.mjs && node test/tts-catalog.test.mjs && node test/voice-profiles.test.mjs && node test/chatterbox-recipe.test.mjs && node test/hear-recipe.test.mjs && npm run test:adoption && node --test test/performance-sampler.test.mjs && node --test test/audio-generations.test.mjs && node test/video-providers.test.mjs && node --test test/web-functions.test.mjs && node --test test/config-mutation.test.mjs && node --test test/model-maintenance.test.mjs && node --test test/model-maintenance-control.test.mjs && node --test test/route-maintenance-concurrency.test.mjs && node --test test/model-maintenance-integration.test.mjs && node --test test/schedule.test.mjs test/schedule-integration.test.mjs && node test/comfyui-media.test.mjs && node test/qwen-image-diffusers.test.mjs && node test/ace-step-diffusers.test.mjs && node test/runtime-residency.test.mjs && node test/spark-audio-recipes.test.mjs && node --test test/video-contracts.test.mjs && node test/ltx-pipelines-recipe.test.mjs && npm run test:ux && npm run test:memory-safety && node --test test/dashboard-memory.test.mjs test/runtime-memory-usage.test.mjs && node --test test/speech-stream.test.mjs && node --test test/audio-providers.test.mjs && node test/openrouter-provider.test.mjs && node test/atlas-sparkglm-recipe.test.mjs && node test/atlas-sparkqwen-recipe.test.mjs && node --test test/head-preparation.test.mjs test/head-transfer.test.mjs test/service-control.test.mjs && node --test test/gateway-relay.test.mjs test/client-retarget.test.mjs && node --test test/head-profile-acceptance.test.mjs test/head-promotion.test.mjs test/head-promotion-transfer.test.mjs test/head-transfer-acceptance.test.mjs && node --test test/cli-diagnostics.test.mjs test/init-backend-timeout.test.mjs test/recipes-root-cli.test.mjs && node --test test/service-install.test.mjs", + "test:unit": "node --test test/deployment-fence.test.mjs test/cluster-deployment.test.mjs test/cluster-deployment-cli.test.mjs test/cluster-deployment-e2e.test.mjs test/node-release-agent.test.mjs test/node-gateway-adapter.test.mjs test/node-gateway-adapter-e2e.test.mjs test/deployment-bundle.test.mjs test/node-agent-cli.test.mjs test/ssh-deployment-transport.test.mjs && node test/runtime-health-ownership.test.mjs && node test/long-prefill-transport.test.mjs && node test/sparkglm-managed.test.mjs && node test/protocol.test.mjs && node test/protocol-stream.test.mjs && node test/stream-metrics.test.mjs && node test/responses-codex.test.mjs && node test/security.test.mjs && node --test test/egress-redaction.test.mjs && node test/community-host.test.mjs && node test/ds4fv-recipe.test.mjs && node test/dspark-hotfixes.test.mjs && node test/glm53-exl3-recipe.test.mjs && node test/host-memory.test.mjs && node test/config-profiles.test.mjs && node test/runtime-policy.test.mjs && node --test test/runtime-policy-config.test.mjs && node --test test/request-scheduler.test.mjs test/adaptive-priority.test.mjs test/rate-limit.test.mjs test/runtime-queue.test.mjs test/foreground-priority.test.mjs test/federated-priority.test.mjs test/runtime-supervisor.test.mjs && node test/interchange.test.mjs && node test/server-resilience.test.mjs && node --test test/standby.test.mjs && node test/metrics-persistence.test.mjs && node test/model-failover.test.mjs && node test/qwen38-vllm-recipe.test.mjs && node test/reasoning-normalize.test.mjs && node test/reasoning-effort.test.mjs && node test/route-control.test.mjs && node test/spark-route-catalog.test.mjs && node test/chat-lane-canary.test.mjs && node test/runtime-watchdog.test.mjs && node test/tts-catalog.test.mjs && node test/voice-profiles.test.mjs && node test/chatterbox-recipe.test.mjs && node test/hear-recipe.test.mjs && npm run test:adoption && node --test test/performance-sampler.test.mjs && node --test test/audio-generations.test.mjs && node test/video-providers.test.mjs && node --test test/web-functions.test.mjs && node --test test/config-mutation.test.mjs && node --test test/model-maintenance.test.mjs && node --test test/model-maintenance-control.test.mjs && node --test test/route-maintenance-concurrency.test.mjs && node --test test/model-maintenance-integration.test.mjs && node --test test/schedule.test.mjs test/schedule-integration.test.mjs && node test/comfyui-media.test.mjs && node test/qwen-image-diffusers.test.mjs && node test/ace-step-diffusers.test.mjs && node test/runtime-residency.test.mjs && node test/spark-audio-recipes.test.mjs && node --test test/video-contracts.test.mjs && node test/ltx-pipelines-recipe.test.mjs && npm run test:ux && npm run test:memory-safety && node --test test/dashboard-memory.test.mjs test/runtime-memory-usage.test.mjs && node --test test/speech-stream.test.mjs && node --test test/audio-providers.test.mjs && node test/openrouter-provider.test.mjs && node test/atlas-sparkglm-recipe.test.mjs && node test/atlas-sparkqwen-recipe.test.mjs && node --test test/head-preparation.test.mjs test/head-transfer.test.mjs test/service-control.test.mjs && node --test test/gateway-relay.test.mjs test/client-retarget.test.mjs && node --test test/head-profile-acceptance.test.mjs test/head-promotion.test.mjs test/head-promotion-transfer.test.mjs test/head-transfer-acceptance.test.mjs && node --test test/cli-diagnostics.test.mjs test/init-backend-timeout.test.mjs test/recipes-root-cli.test.mjs && node --test test/service-install.test.mjs", "test:cluster": "node test/cluster.test.mjs && node --test test/cluster-availability.test.mjs test/cluster-readiness.test.mjs", "test": "npm run test:cluster && npm run test:unit && npm run test:entity-stagger && npm run smoke && npm run community:check", "test:entity-stagger": "node test/entity-stagger-benchmark.test.mjs", diff --git a/src/config.mjs b/src/config.mjs index 9987205..412f73a 100644 --- a/src/config.mjs +++ b/src/config.mjs @@ -199,6 +199,14 @@ function validateConfig(config, sourcePath, env) { errors.push('server.foregroundIdleMs must be an integer between 0 and 3600000'); } errors.push(...validateRequestScheduling(config.server?.requestScheduling)); + if (config.security?.redactEgress != null && typeof config.security.redactEgress !== 'boolean') { + errors.push('security.redactEgress must be a boolean'); + } + for (const [backendId, backend] of Object.entries(config.backends ?? {})) { + if (backend?.redact != null && typeof backend.redact !== 'boolean') { + errors.push(`backends.${backendId}.redact must be a boolean`); + } + } for (const [index, model] of (config.models ?? []).entries()) { if (!model?.id) errors.push(`models[${index}] is missing id`); @@ -614,6 +622,7 @@ export async function loadConfig( allowWildcardCors: false, allowNonLoopbackBind: false, publicTelemetry: false, + redactEgress: false, apiKeys: [], adminApiKeys: [], ...asObject(expanded.security) diff --git a/src/egress-redaction.mjs b/src/egress-redaction.mjs new file mode 100644 index 0000000..3caeaef --- /dev/null +++ b/src/egress-redaction.mjs @@ -0,0 +1,239 @@ +/** + * Egress redaction: scrub credentials out of request text before the gateway + * forwards it to a backend outside the local network. + * + * Opt-in through `security.redactEgress` (every external backend) or + * `backends..redact` (one backend, either direction). Matches become + * `[secret]`; labelled values keep their label. Patterns follow Tessera's + * Steward redactor. Matched values are never returned or logged, only counted. + */ + +import { isIP } from 'node:net'; +import { isFederatedGatewayBackend } from './cluster.mjs'; +import { isLoopbackAddress } from './security.mjs'; + +const SECRET = '[secret]'; + +// Whole-token credential formats. No character class admits a quote or a +// backslash, so redacting inside JSON-encoded text keeps it valid JSON. +const SECRET_PATTERNS = [ + // Also covers OpenRouter `sk-or-v1-…` keys. + /\bsk-(?:ant-|proj-)?[A-Za-z0-9_-]{16,}/g, + /\blloom_[a-z]+_[0-9a-f]{16,}/g, + /\b(?:gh[pousr]_[A-Za-z0-9]{20,}|github_pat_[A-Za-z0-9_]{20,})/g, + /\bxox[abposr]-[A-Za-z0-9-]{10,}/g, + /\bAKIA[0-9A-Z]{16}\b/g, + /\bAIza[0-9A-Za-z_-]{35}/g, + /\b(?:r8|hf|jina|xai)_[A-Za-z0-9]{20,}/g, + /\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/g +]; + +// PEM private keys, BEGIN through END. Scanned with two linear searches rather +// than one lazy regex, which goes quadratic over many unterminated BEGIN lines. +const PEM_BEGIN = /-----BEGIN [A-Z ]*PRIVATE KEY-----/g; +const PEM_END = /-----END [A-Z ]*PRIVATE KEY-----/g; + +function replacePrivateKeys(text, replacement) { + let output = ''; + let cursor = 0; + PEM_BEGIN.lastIndex = 0; + for (let begin = PEM_BEGIN.exec(text); begin; begin = PEM_BEGIN.exec(text)) { + PEM_END.lastIndex = PEM_BEGIN.lastIndex; + if (!PEM_END.exec(text)) break; + output += text.slice(cursor, begin.index) + replacement(); + cursor = PEM_END.lastIndex; + PEM_BEGIN.lastIndex = cursor; + } + return cursor ? output + text.slice(cursor) : text; +} + +// scheme://user:password@host keeps scheme, user and host. +const URL_CREDENTIAL = /\b([a-z][a-z0-9+.-]*:\/\/[^\s:@/?#"'\\]*:)([^\s@/?#"'\\]+)@/gi; + +// A credential label (bearer, *api_key, *token, *secret, *password, …) and its +// value. The value is 6+ characters, so "token: 500" and prose stay intact. +const LABELLED_VALUE = + /\b((?:authorization:\s*)?bearer\s+|[A-Z0-9_]*(?:(?:api|secret|private|access)[_-]?key|secret[_-]?access[_-]?key|token|secret|password|passwd)["']?\s*[:=]\s*["']?)(?!\[secret\])([^\s"'\\]{6,})/gi; + +const TRAILING_PUNCTUATION = /[,;)\]}]+$/; + +// An unquoted labelled value that reads as code or a placeholder rather than a +// literal: an expression, a variable reference, a member path, or a word with +// no digit or symbol (`getToken`, `string`, `None`, `YOUR_API_KEY`). +function looksLikeCode(value) { + return ( + /[()[\]{}<>]/.test(value) || + /^[$=<>!&|%]/.test(value) || + /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+$/.test(value) || + /^[A-Za-z_.$-]+$/.test(value) + ); +} + +/** Replace credentials in one string. Returns the text and a match count. */ +export function redactSecrets(text) { + if (typeof text !== 'string' || !text) return { text, count: 0 }; + let count = 0; + const replacement = () => { + count += 1; + return SECRET; + }; + let output = replacePrivateKeys(text, replacement); + for (const pattern of SECRET_PATTERNS) output = output.replace(pattern, replacement); + output = output.replace(URL_CREDENTIAL, (match, prefix, password) => { + if (password === SECRET) return match; + count += 1; + return `${prefix}${SECRET}@`; + }); + output = output.replace(LABELLED_VALUE, (match, label, rawValue) => { + const tail = rawValue.match(TRAILING_PUNCTUATION)?.[0] ?? ''; + const value = rawValue.slice(0, rawValue.length - tail.length); + const quoted = /["']$/.test(label); + if (value.length < 6 || (!quoted && looksLikeCode(value))) return match; + count += 1; + return `${label}${SECRET}${tail}`; + }); + return { text: output, count }; +} + +// Text-bearing keys inside messages, content parts, tool calls and Responses +// items. Image, audio and file parts carry none of them, so they pass through. +const NESTED_TEXT_FIELDS = [ + 'content', + 'text', + 'arguments', + 'output', + 'input', + 'reasoning', + 'reasoning_content', + 'tool_calls', + 'function' +]; +// Top-level request fields: Chat/Anthropic messages, legacy and media prompts, +// Responses/speech input, and instructions/system text. +const BODY_TEXT_FIELDS = ['messages', 'prompt', 'input', 'instructions', 'system']; + +function redactValue(value, scrub) { + if (typeof value === 'string') return scrub(value); + if (Array.isArray(value)) { + let changed = false; + const next = value.map((entry) => { + const updated = redactValue(entry, scrub); + changed ||= updated !== entry; + return updated; + }); + return changed ? next : value; + } + return value && typeof value === 'object' ? redactFields(value, NESTED_TEXT_FIELDS, scrub) : value; +} + +function redactFields(object, fields, scrub) { + let next = object; + for (const field of fields) { + if (!Object.hasOwn(object, field)) continue; + const updated = redactValue(object[field], scrub); + if (updated === object[field]) continue; + if (next === object) next = { ...object }; + next[field] = updated; + } + return next; +} + +/** + * Redact the model-visible text of an outgoing JSON request body. The caller's + * object is never mutated; it is returned as is when nothing matched. + */ +export function redactEgressBody(body) { + if (!body || typeof body !== 'object' || Array.isArray(body)) return { body, count: 0 }; + let count = 0; + const scrub = (text) => { + const result = redactSecrets(text); + count += result.count; + return result.count ? result.text : text; + }; + const next = redactFields(body, BODY_TEXT_FIELDS, scrub); + return { body: next, count }; +} + +function ipv4Internal(address) { + const [a, b] = address.split('.').map(Number); + return ( + a === 127 || + a === 10 || + a === 0 || + (a === 172 && b >= 16 && b <= 31) || + (a === 192 && b === 168) || + (a === 169 && b === 254) || + (a === 100 && b >= 64 && b <= 127) + ); +} + +function ipv6Groups(address) { + let value = address; + const v4 = value.match(/(\d+\.\d+\.\d+\.\d+)$/); + if (v4) { + const [a, b, c, d] = v4[1].split('.').map(Number); + value = value.slice(0, -v4[1].length) + `${((a << 8) | b).toString(16)}:${((c << 8) | d).toString(16)}`; + } + const [head, tail] = value.split('::'); + const left = head ? head.split(':') : []; + const right = tail ? tail.split(':') : []; + const fill = value.includes('::') ? Array(8 - left.length - right.length).fill('0') : []; + return [...left, ...fill, ...right].map((group) => parseInt(group, 16)); +} + +function ipv6Internal(address) { + const groups = ipv6Groups(address); + if (groups.slice(0, 5).every((group) => group === 0) && groups[5] === 0xffff) { + const [hi, lo] = groups.slice(6); + return ipv4Internal(`${hi >> 8}.${hi & 0xff}.${lo >> 8}.${lo & 0xff}`); + } + return ( + groups.every((group, index) => group === (index === 7 ? 1 : 0)) || // ::1 + groups.every((group) => group === 0) || // :: + (groups[0] & 0xfe00) === 0xfc00 || // ULA fc00::/7 + (groups[0] & 0xffc0) === 0xfe80 // link-local fe80::/10 + ); +} + +/** + * True for loopback, RFC1918, link-local, CGNAT/tailnet and IPv6 ULA + * addresses, `.ts.net`, `.local`, `.internal` and `.localhost` names, and + * unqualified single-label hostnames. Everything else is external. + */ +export function isInternalHost(hostname) { + const host = String(hostname ?? '') + .trim() + .toLowerCase() + .replace(/^\[|\]$/g, '') + .split('%')[0] + .replace(/\.$/, ''); + if (!host) return false; + if (isLoopbackAddress(host)) return true; + const family = isIP(host); + if (family === 4) return ipv4Internal(host); + if (family === 6) return ipv6Internal(host); + if (!host.includes('.')) return true; + return ['.ts.net', '.local', '.internal', '.localhost'].some((suffix) => host.endsWith(suffix)); +} + +function isExternalBackend(backend) { + // Provider adapters call their fixed cloud origin whatever baseUrl says. + if (backend?.audioProvider || backend?.videoProvider) return true; + try { + return !isInternalHost(new URL(String(backend?.baseUrl ?? '')).hostname); + } catch { + return true; + } +} + +/** + * Whether requests to this backend are redacted. `backends..redact` wins + * in either direction; otherwise `security.redactEgress` applies to external + * backends, except a federated LLooM gateway, which applies its own policy. + */ +export function shouldRedactEgress(config, backendId, backend) { + if (typeof backend?.redact === 'boolean') return backend.redact; + if (config?.security?.redactEgress !== true) return false; + if (backendId && isFederatedGatewayBackend(config, backendId)) return false; + return isExternalBackend(backend); +} diff --git a/src/server.mjs b/src/server.mjs index 37ffde7..82fd7f9 100644 --- a/src/server.mjs +++ b/src/server.mjs @@ -136,9 +136,11 @@ import { securityPublicStatus } from './security.mjs'; import { ClusterCoordinator, currentNodeId, isFederatedGatewayBackend, runtimePlacement } from './cluster.mjs'; +import { redactEgressBody, shouldRedactEgress } from './egress-redaction.mjs'; import { createDeploymentFence, DeploymentFenceError, isDeploymentFencePath } from './deployment-fence.mjs'; const JSON_TYPE = 'application/json; charset=utf-8'; +const UNREDACTED_EGRESS_PATHS = new Set(['/v1/embeddings', '/v1/audio/transcriptions']); const SSE_TYPE = 'text/event-stream; charset=utf-8'; const execFileAsync = promisify(execFile); const packageRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); @@ -3758,6 +3760,31 @@ export function createLloomServer( return { 'x-lloom-request-class': requestClass }; } + // The single egress policy for model requests. It runs after protocol + // translation, so Chat, Responses, Anthropic and media routes, streaming or + // not, share it. Embeddings and transcriptions are never rewritten: changing + // their input changes the vectors, and a transcription payload is audio. + function egressBody(resolved, path, body) { + if (UNREDACTED_EGRESS_PATHS.has(path) || !shouldRedactEgress(config, resolved.model.backend, resolved.backend)) { + return body; + } + const result = redactEgressBody(body); + if (result.count) { + logger.info?.(`egress redaction: replaced ${result.count} secret(s) for backend ${resolved.model.backend}`); + } + return result.body; + } + + function fetchModelUpstream(req, resolved, { path, body, ...options }) { + return fetchUpstream({ + ...options, + headers: inferenceGatewayHeaders(req, resolved), + backend: resolved.backend, + path, + body: egressBody(resolved, path, body) + }); + } + async function handleOpenAIChat(req, res, { bodyOverride = null } = {}) { const body = bodyOverride ?? (await readJson(req)); return recordModelRequestWithFailover( @@ -3789,9 +3816,7 @@ export function createLloomServer( const timeout = setTimeout(() => timeoutController.abort(), DEPLOYMENT_CANARY_TIMEOUT_MS); const canarySignal = AbortSignal.any([signal, timeoutController.signal]); try { - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/chat/completions', signal: canarySignal, body: { @@ -3849,9 +3874,7 @@ export function createLloomServer( clearTimeout(timeout); } } - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/chat/completions', signal, body: { @@ -3901,9 +3924,7 @@ export function createLloomServer( }, async ({ signal, timing, watchdog }) => { watchdog.arm(); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/images/generations', signal, dispatcher: longRunningMediaDispatcher, @@ -4029,15 +4050,13 @@ export function createLloomServer( if (resolved.backend.videoProvider) { const upstream = await generateProviderVideo({ backend: resolved.backend, - body: { ...body, model: resolved.model.upstreamModel }, + body: egressBody(resolved, '/v1/videos/generations', { ...body, model: resolved.model.upstreamModel }), signal, timeoutMs: resolved.backend.timeoutMs ?? 600000 }); return proxyRawResponse(res, upstream, { signal, timing, corsConfig: config }); } - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/videos/generations', signal, dispatcher: longRunningMediaDispatcher, @@ -4085,7 +4104,7 @@ export function createLloomServer( backend: resolved.backend, fetchFn: undiciFetch, dispatcher: upstreamDispatcher ?? longRunningMediaDispatcher, - body: { ...body, model: resolved.model.upstreamModel }, + body: egressBody(resolved, '/v1/audio/generations', { ...body, model: resolved.model.upstreamModel }), signal, timeoutMs: resolved.backend.timeoutMs ?? 600000 }); @@ -4093,9 +4112,7 @@ export function createLloomServer( // assembled artifact is written as a buffered binary response. return proxyRawResponse(res, upstream, { signal, timing, corsConfig: config }); } - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/audio/generations', signal, dispatcher: longRunningMediaDispatcher, @@ -4136,9 +4153,7 @@ export function createLloomServer( }, async (resolved, { signal, timing, watchdog, hasNext }) => { watchdog.arm(); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/embeddings', signal, body: { @@ -4198,9 +4213,7 @@ export function createLloomServer( }, async (resolved, { signal, timing, watchdog, hasNext }) => { watchdog.arm(); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { // Absolute URL: `fetchUpstream` passes it through untouched, which is // what keeps the request off the backend's `/v1` base path. path: tokenizeUpstreamUrl(resolved.backend), @@ -4296,9 +4309,7 @@ export function createLloomServer( }, async ({ signal, timing, progress, watchdog }) => { watchdog.arm(); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/audio/speech', signal, body: normalized @@ -4580,9 +4591,7 @@ export function createLloomServer( }, async ({ signal, timing, watchdog }) => { watchdog.arm(); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/audio/transcriptions', signal, body: { @@ -4666,9 +4675,7 @@ export function createLloomServer( async (resolved, { signal, timing, watchdog, progress, hasNext }) => { watchdog.arm(); const normalizedRequest = prepareStructuredOutputForBackend(responsesToOpenAIChat(body, resolved), resolved); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/chat/completions', signal, body: normalizedRequest.body @@ -4728,9 +4735,7 @@ export function createLloomServer( }, async (resolved, { signal, timing, watchdog, progress, hasNext }) => { watchdog.arm(); - const upstream = await fetchUpstream({ - headers: inferenceGatewayHeaders(req, resolved), - backend: resolved.backend, + const upstream = await fetchModelUpstream(req, resolved, { path: '/v1/chat/completions', signal, body: anthropicMessagesToOpenAI(body, resolved) diff --git a/test/egress-redaction.test.mjs b/test/egress-redaction.test.mjs new file mode 100644 index 0000000..ecf8ac1 --- /dev/null +++ b/test/egress-redaction.test.mjs @@ -0,0 +1,479 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import { test } from 'node:test'; +import { Agent, MockAgent, getGlobalDispatcher, setGlobalDispatcher } from 'undici'; + +import { loadConfig } from '../src/config.mjs'; +import { isInternalHost, redactEgressBody, redactSecrets, shouldRedactEgress } from '../src/egress-redaction.mjs'; +import { createLloomServer } from '../src/server.mjs'; + +// Fake credentials are assembled at runtime so no literal in this file looks +// like a live key to secret scanners. +const join = (...parts) => parts.join(''); +const FAKE = { + openai: join('sk-', 'proj-', 'A1b2C3d4E5f6G7h8I9j0K1l2'), + anthropic: join('sk-', 'ant-', 'api03-', 'Zy9Xw8Vu7Ts6Rq5Po4Nm3'), + openrouter: join('sk-', 'or-', 'v1-', '0123456789abcdef'.repeat(4)), + deepseek: join('sk-', '9f8e7d6c5b4a39281706f5e4'), + lloom: join('lloom_', 'live_', '0123456789abcdef0123'), + github: join('gh', 'p_', 'A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8'), + githubPat: join('github_', 'pat_', '11ABCDEFG0123456789_abcdefghij'), + slack: join('xo', 'xb-', '1234567890-abcdefghij'), + aws: join('AK', 'IA', 'IOSFODNN7EXAMPLE'), + google: join('AI', 'za', 'SyA1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q'), + replicate: join('r8', '_', 'A1b2C3d4E5f6G7h8I9j0K1'), + huggingface: join('hf', '_', 'A1b2C3d4E5f6G7h8I9j0K1'), + jina: join('jina', '_', 'A1b2C3d4E5f6G7h8I9j0K1'), + xai: join('xai', '_', 'A1b2C3d4E5f6G7h8I9j0K1'), + jwt: join('eyJ', 'hbGciOiJIUzI1NiJ9', '.', 'eyJzdWIiOiIxMjM0In0', '.', 'dBjftJeZ4CVPmB92K27uhbUJU1p1r_wW1gFWFOEjXk') +}; +const PEM = join( + '-----BEGIN ', + 'RSA PRIVATE KEY-----\n', + 'MIIEowIBAAKCAQEA0Z3VS5JJcds3xfn/ygWyF8PbnGy0AHB7MhgHcTz6sE2I2yPB\n', + 'aFDrBz9vFqU4yQ==\n', + '-----END ', + 'RSA PRIVATE KEY-----' +); + +function redacted(text) { + return redactSecrets(text).text; +} + +test('known credential formats are replaced whole', () => { + for (const [name, secret] of Object.entries(FAKE)) { + const result = redactSecrets(`before ${secret} after`); + assert.equal(result.text, 'before [secret] after', name); + assert.equal(result.count, 1, name); + } +}); + +test('the sk- pattern already covers OpenRouter sk-or-v1 keys', () => { + assert.equal(redacted(`OpenRouter key ${FAKE.openrouter}.`), 'OpenRouter key [secret].'); +}); + +test('PEM private keys are replaced from BEGIN through END', () => { + const result = redactSecrets(`key:\n${PEM}\nend of file`); + assert.equal(result.text, 'key:\n[secret]\nend of file'); + assert.equal(result.count, 1); + // Public material is not a secret. + const cert = '-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----'; + assert.equal(redacted(cert), cert); +}); + +test('unterminated PEM headers scan in linear time', () => { + const text = `${'-----BEGIN PRIVATE KEY-----\n'.repeat(40000)}no end`; + const started = Date.now(); + assert.equal(redactSecrets(text).count, 0); + assert.ok(Date.now() - started < 1000); +}); + +test('URL credentials keep scheme, user and host', () => { + assert.equal( + redacted('MONGO_URI=mongodb+srv://app:hunter22@cluster0.example.net/db?retryWrites=true'), + 'MONGO_URI=mongodb+srv://app:[secret]@cluster0.example.net/db?retryWrites=true' + ); + assert.equal(redacted('postgres://admin:p%40ss@db:5432/main'), 'postgres://admin:[secret]@db:5432/main'); + assert.equal(redacted('redis://:s3cret@cache:6379'), 'redis://:[secret]@cache:6379'); + assert.equal( + redacted(`git clone https://x-access-token:${FAKE.github}@github.com/o/r.git`), + 'git clone https://x-access-token:[secret]@github.com/o/r.git' + ); + // No password, a port, or an email address is not a URL credential. + for (const text of ['https://example.com:8443/path', 'ssh://git@github.com/o/r', 'mail me at a@b.co']) { + assert.equal(redacted(text), text); + } +}); + +test('labelled values keep the label', () => { + const cases = [ + ['Authorization: Bearer abc123def456ghi', 'Authorization: Bearer [secret]'], + ['curl -H "Authorization: bearer abc123def456"', 'curl -H "Authorization: bearer [secret]"'], + ['export DB_PASSWORD=hunter22', 'export DB_PASSWORD=[secret]'], + ['OPENAI_API_KEY="abc123-xyz789"', 'OPENAI_API_KEY="[secret]"'], + ['api-key: 9f8e7d6c5b', 'api-key: [secret]'], + ['passwd = Tr0ub4dor&3', 'passwd = [secret]'], + ["client_secret: 'whatever-it-is'", "client_secret: '[secret]'"], + ['{"password": "correcthorse", "user": "jo"}', '{"password": "[secret]", "user": "jo"}'], + ['token=abc123def456,', 'token=[secret],'], + ['X-API-KEY: 0a1b2c3d4e', 'X-API-KEY: [secret]'], + ["SECRET_KEY = 'django-insecure-abc'", "SECRET_KEY = '[secret]'"], + [join('AWS_SECRET_ACCESS_KEY=', 'wJalrXUtnFEMI/K7MDENG/bPxRfi'), 'AWS_SECRET_ACCESS_KEY=[secret]'] + ]; + for (const [input, expected] of cases) assert.equal(redacted(input), expected, input); +}); + +test('a known token behind a label is counted once', () => { + const result = redactSecrets(`Authorization: Bearer ${FAKE.openai}\nOPENAI_API_KEY=${FAKE.openai}`); + assert.equal(result.text, 'Authorization: Bearer [secret]\nOPENAI_API_KEY=[secret]'); + assert.equal(result.count, 2); +}); + +test('ordinary prose and code stay intact', () => { + const untouched = [ + 'the token budget is 500', + 'The token budget: 500 tokens per request.', + 'max_tokens: 4096', + 'Set max_completion_tokens = 2048 and temperature = 0.2.', + 'Use a password manager and rotate your API key regularly.', + 'The bearer of this letter is a friend.', + 'A secret: the cake is a lie.', + 'tokenizer: "o200k_base"', + 'const token = getToken();', + 'const apiKey = process.env.OPENAI_API_KEY;', + 'api_key = os.environ["OPENAI_API_KEY"]', + "password = request.form['password']", + 'secret = os.urandom(32)', + 'token = jwt.sign(payload, secret)', + 'token: string;', + 'password: Optional[str] = None', + 'api_key=None', + 'headers = {"Authorization": f"Bearer {token}"}', + 'Authorization: Bearer ${TOKEN}', + 'password: ${{ secrets.DB_PASSWORD }}', + 'TOKEN=$GITHUB_TOKEN', + 'if (token === expected) return;', + 'self.secret_key = settings.SECRET_KEY', + 'sk-learn is fine, task-oriented too', + 'see https://example.com/docs#api-key' + ]; + for (const text of untouched) { + const result = redactSecrets(text); + assert.equal(result.text, text, text); + assert.equal(result.count, 0, text); + } +}); + +test('redaction inside JSON-encoded tool arguments keeps the JSON valid', () => { + const args = JSON.stringify({ + command: `export DB_PASSWORD=hunter22\\x && curl -H "Authorization: Bearer ${FAKE.openai}"`, + env: { API_KEY: 'abc123def456' }, + pem: PEM + }); + const result = redactSecrets(args); + const parsed = JSON.parse(result.text); + assert.equal(parsed.command.includes('hunter22'), false); + assert.equal(parsed.command.includes(FAKE.openai), false); + assert.equal(parsed.pem, '[secret]'); + assert.ok(result.count >= 3); +}); + +test('non-strings pass through', () => { + assert.deepEqual(redactSecrets(undefined), { text: undefined, count: 0 }); + assert.deepEqual(redactSecrets(''), { text: '', count: 0 }); +}); + +test('redactEgressBody scrubs every text field the model would see', () => { + const image = { type: 'image_url', image_url: { url: `data:image/png;base64,${FAKE.openai}` } }; + const audio = { type: 'input_audio', input_audio: { data: FAKE.openai, format: 'wav' } }; + const body = { + model: 'cloud/model', + instructions: `use ${FAKE.deepseek}`, + system: [{ type: 'text', text: `key ${FAKE.anthropic}` }], + messages: [ + { role: 'system', content: `secret ${FAKE.lloom}` }, + { role: 'user', content: [{ type: 'text', text: `my key is ${FAKE.openrouter}` }, image, audio] }, + { + role: 'assistant', + content: null, + reasoning_content: `they pasted ${FAKE.github}`, + tool_calls: [ + { + id: 'call_1', + type: 'function', + function: { name: 'shell', arguments: JSON.stringify({ cmd: `echo ${FAKE.aws}` }) } + } + ] + }, + { role: 'tool', tool_call_id: 'call_1', content: `AWS_SECRET=${FAKE.aws}` } + ], + tools: [{ type: 'function', function: { name: 'shell', parameters: { type: 'object' } } }] + }; + const before = structuredClone(body); + const result = redactEgressBody(body); + assert.deepEqual(body, before, 'caller body must not be mutated'); + const out = result.body; + assert.equal(out.instructions, 'use [secret]'); + assert.equal(out.system[0].text, 'key [secret]'); + assert.equal(out.messages[0].content, 'secret [secret]'); + assert.equal(out.messages[1].content[0].text, 'my key is [secret]'); + assert.equal(out.messages[1].content[1], image, 'image parts are untouched'); + assert.equal(out.messages[1].content[2], audio, 'audio parts are untouched'); + assert.equal(out.messages[2].reasoning_content, 'they pasted [secret]'); + assert.deepEqual(JSON.parse(out.messages[2].tool_calls[0].function.arguments), { cmd: 'echo [secret]' }); + assert.equal(out.messages[2].tool_calls[0].function.name, 'shell'); + assert.equal(out.messages[3].content, 'AWS_SECRET=[secret]'); + assert.equal(out.tools, body.tools); + assert.equal(out.model, 'cloud/model'); + assert.equal(result.count, 7); +}); + +test('redactEgressBody covers prompt and input shapes', () => { + assert.deepEqual(redactEgressBody({ prompt: `a ${FAKE.xai}` }).body, { prompt: 'a [secret]' }); + assert.deepEqual(redactEgressBody({ prompt: [`a ${FAKE.xai}`, 'b'] }).body, { prompt: ['a [secret]', 'b'] }); + assert.deepEqual(redactEgressBody({ input: `say ${FAKE.jina}` }).body, { input: 'say [secret]' }); + const items = redactEgressBody({ + input: [ + { role: 'user', content: [{ type: 'input_text', text: `k=${FAKE.replicate}` }] }, + { type: 'function_call', call_id: 'c', name: 'f', arguments: `{"t":"${FAKE.huggingface}"}` }, + { type: 'function_call_output', call_id: 'c', output: `got ${FAKE.google}` }, + { type: 'input_image', image_url: `https://example.com/${FAKE.openai}.png` } + ] + }).body.input; + assert.equal(items[0].content[0].text, 'k=[secret]'); + assert.equal(items[1].arguments, '{"t":"[secret]"}'); + assert.equal(items[2].output, 'got [secret]'); + assert.equal(items[3].image_url, `https://example.com/${FAKE.openai}.png`); +}); + +test('redactEgressBody returns the same object when nothing matches', () => { + const body = { model: 'm', messages: [{ role: 'user', content: 'the token budget is 500' }] }; + const result = redactEgressBody(body); + assert.equal(result.body, body); + assert.equal(result.count, 0); + for (const value of [null, undefined, 'text', [1, 2]]) assert.equal(redactEgressBody(value).body, value); +}); + +test('isInternalHost classifies local, private and tailnet hosts', () => { + const internal = [ + 'localhost', + 'api.localhost', + '127.0.0.1', + '127.9.9.9', + '[::1]', + '::1', + '10.0.0.5', + '172.16.0.1', + '172.31.255.254', + '192.168.1.20', + '169.254.10.10', + '100.64.0.1', + '100.127.255.254', + 'fc00::1', + 'fd12:3456::1', + '[fe80::1]', + '::ffff:10.1.2.3', + '::ffff:a01:203', + 'ennspark03', + 'ennspark03.tail1234.ts.net', + 'nas.local', + 'gpu.internal', + 'NAS.LOCAL.' + ]; + const external = [ + 'api.deepseek.com', + 'openrouter.ai', + '8.8.8.8', + '172.32.0.1', + '172.15.0.1', + '100.128.0.1', + '100.63.255.255', + '192.169.0.1', + '2606:4700::1111', + '::ffff:8.8.8.8', + 'ts.net.example.com', + 'evil-ts.net', + '' + ]; + for (const host of internal) assert.equal(isInternalHost(host), true, host); + for (const host of external) assert.equal(isInternalHost(host), false, host); +}); + +test('shouldRedactEgress honours the global switch, overrides and federation', () => { + const cloud = { type: 'openai', baseUrl: 'https://api.deepseek.com/v1' }; + const lan = { type: 'openai', baseUrl: 'http://192.168.1.20:8000/v1' }; + const tailnet = { type: 'openai', baseUrl: 'http://ennspark03:8000/v1' }; + const config = (redactEgress, extra = {}) => ({ security: { redactEgress }, backends: {}, ...extra }); + + assert.equal(shouldRedactEgress(config(undefined), 'cloud', cloud), false, 'off by default'); + assert.equal(shouldRedactEgress(config(false), 'cloud', cloud), false); + assert.equal(shouldRedactEgress(config(true), 'cloud', cloud), true); + assert.equal(shouldRedactEgress(config(true), 'lan', lan), false); + assert.equal(shouldRedactEgress(config(true), 'tailnet', tailnet), false); + assert.equal(shouldRedactEgress(config(true), 'bad', { baseUrl: 'not a url' }), true, 'unknown hosts fail safe'); + + // A per-backend override wins in either direction. + assert.equal(shouldRedactEgress(config(false), 'lan', { ...lan, redact: true }), true); + assert.equal(shouldRedactEgress(config(true), 'cloud', { ...cloud, redact: false }), false); + + // Provider adapters call their fixed cloud origin whatever baseUrl says. + assert.equal(shouldRedactEgress(config(true), 'music', { ...lan, audioProvider: 'openrouter' }), true); + assert.equal(shouldRedactEgress(config(true), 'video', { videoProvider: 'replicate' }), true); + + // Another LLooM gateway applies its own policy when it egresses. + const federated = config(true, { + cluster: { nodes: { edge: { endpoint: 'https://edge.example.com', proxy: { models: ['m'] } } } } + }); + const federatedBackend = { type: 'openai', baseUrl: 'https://edge.example.com/v1' }; + assert.equal(shouldRedactEgress(federated, 'lloom-node-edge', federatedBackend), false); + assert.equal(shouldRedactEgress(federated, 'lloom-node-edge', { ...federatedBackend, redact: true }), true); + assert.equal(shouldRedactEgress(federated, 'other', federatedBackend), true); +}); + +test('config defaults redactEgress to false and validates both keys', async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'lloom-egress-config-')); + const write = async (extra) => { + const file = path.join(dir, `config-${Math.random().toString(16).slice(2)}.json`); + await fs.writeFile( + file, + JSON.stringify({ + backends: { cloud: { type: 'openai', baseUrl: 'https://api.deepseek.com/v1' } }, + models: [{ id: 'm', backend: 'cloud' }], + ...extra + }) + ); + return file; + }; + try { + const defaults = await loadConfig(await write({}), { env: {} }); + assert.equal(defaults.security.redactEgress, false); + const enabled = await loadConfig(await write({ security: { redactEgress: true } }), { env: {} }); + assert.equal(enabled.security.redactEgress, true); + await assert.rejects( + loadConfig(await write({ security: { redactEgress: 'yes' } }), { env: {} }), + /security\.redactEgress must be a boolean/ + ); + await assert.rejects( + loadConfig(await write({ backends: { cloud: { type: 'openai', baseUrl: 'https://x.example/v1', redact: 1 } } }), { + env: {} + }), + /backends\.cloud\.redact must be a boolean/ + ); + } finally { + await fs.rm(dir, { recursive: true, force: true }); + } +}); + +// --------------------------------------------------------------------------- +// Gateway integration: the outgoing upstream body is captured through a mocked +// undici dispatcher, so no request leaves the machine. +// --------------------------------------------------------------------------- + +const CHAT_REPLY = { + id: 'chatcmpl-1', + object: 'chat.completion', + choices: [{ index: 0, message: { role: 'assistant', content: 'ok' }, finish_reason: 'stop' }], + usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 } +}; +const STREAM_REPLY = [ + 'data: {"id":"c","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"ok"},"finish_reason":null}]}', + '', + 'data: {"id":"c","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}', + '', + 'data: [DONE]', + '', + '' +].join('\n'); + +async function readBody(body) { + if (!body || typeof body[Symbol.asyncIterator] !== 'function') return body; + const chunks = []; + for await (const chunk of body) chunks.push(chunk); + return Buffer.concat(chunks).toString('utf8'); +} + +async function withGateway({ redactEgress, backend = {} }, fn) { + const previous = getGlobalDispatcher(); + const mockAgent = new MockAgent({ agent: new Agent({ headersTimeout: 60000, bodyTimeout: 60000 }) }); + mockAgent.disableNetConnect(); + mockAgent.enableNetConnect((host) => typeof host === 'string' && host.startsWith('127.0.0.1')); + setGlobalDispatcher(mockAgent); + const seen = []; + const pool = mockAgent.get('https://api.deepseek.com'); + // One upstream reply per gateway call, shaped for the route and stream flag. + const expectUpstream = (route, stream) => + pool.intercept({ path: route, method: 'POST' }).reply( + 200, + async (opts) => { + seen.push({ route, body: JSON.parse(await readBody(opts.body)) }); + if (route === '/v1/embeddings') return { object: 'list', data: [{ index: 0, embedding: [0.1] }] }; + if (route === '/v1/audio/transcriptions') return { text: 'ok' }; + return stream ? STREAM_REPLY : CHAT_REPLY; + }, + { headers: { 'content-type': stream ? 'text/event-stream' : 'application/json' } } + ); + const app = createLloomServer( + { + server: { host: '127.0.0.1', port: 0 }, + security: { allowMissingAuth: true, apiKeys: [], redactEgress }, + defaults: { chatModel: 'deepseek-chat' }, + backends: { deepseek: { type: 'openai', baseUrl: 'https://api.deepseek.com/v1', ...backend } }, + models: [ + { id: 'deepseek-chat', backend: 'deepseek', upstreamModel: 'deepseek-chat', kind: 'chat' }, + { id: 'deepseek-embed', backend: 'deepseek', upstreamModel: 'embed', kind: 'embedding' }, + { id: 'deepseek-stt', backend: 'deepseek', upstreamModel: 'stt', kind: 'audio_transcription' } + ], + runtimes: {} + }, + { logger: { error() {}, warn() {}, info() {}, log() {} }, upstreamDispatcher: mockAgent } + ); + await new Promise((resolve) => app.server.listen(0, '127.0.0.1', resolve)); + const base = `http://127.0.0.1:${app.server.address().port}`; + const post = async (route, body, headers = {}) => { + const passthrough = ['/v1/embeddings', '/v1/audio/transcriptions'].includes(route); + expectUpstream(passthrough ? route : '/v1/chat/completions', body.stream === true); + const res = await fetch(`${base}${route}`, { + method: 'POST', + headers: { 'content-type': 'application/json', ...headers }, + body: JSON.stringify(body) + }); + const text = await res.text(); + assert.equal(res.status, 200, text); + return text; + }; + try { + await fn({ post, seen }); + } finally { + await new Promise((resolve) => app.server.close(resolve)); + setGlobalDispatcher(previous); + await mockAgent.close(); + } +} + +const userSecret = `deploy with ${FAKE.openai}`; + +test('gateway redacts chat, Responses and Anthropic requests sent to an external backend, not embeddings or transcriptions', async () => { + await withGateway({ redactEgress: true }, async ({ post, seen }) => { + await post('/v1/chat/completions', { model: 'deepseek-chat', messages: [{ role: 'user', content: userSecret }] }); + await post('/v1/chat/completions', { + model: 'deepseek-chat', + stream: true, + messages: [{ role: 'user', content: userSecret }] + }); + await post('/v1/responses', { model: 'deepseek-chat', instructions: userSecret, input: userSecret }); + await post( + '/v1/messages', + { model: 'deepseek-chat', max_tokens: 16, system: userSecret, messages: [{ role: 'user', content: userSecret }] }, + { 'anthropic-version': '2023-06-01' } + ); + await post('/v1/embeddings', { model: 'deepseek-embed', input: userSecret }); + await post('/v1/audio/transcriptions', { + model: 'deepseek-stt', + file: 'data:audio/wav;base64,AAAA', + prompt: userSecret + }); + + const chat = seen.filter((entry) => entry.route === '/v1/chat/completions'); + assert.equal(chat.length, 4); + for (const { body } of chat) { + const outbound = JSON.stringify(body.messages); + assert.equal(outbound.includes(FAKE.openai), false, outbound); + assert.match(outbound, /deploy with \[secret\]/); + } + const embedding = seen.find((entry) => entry.route === '/v1/embeddings'); + assert.equal(embedding.body.input, userSecret, 'embeddings are never rewritten'); + const transcription = seen.find((entry) => entry.route === '/v1/audio/transcriptions'); + assert.equal(transcription.body.prompt, userSecret, 'transcriptions are never rewritten'); + }); +}); + +test('gateway leaves external requests untouched when redaction is off or overridden', async () => { + for (const options of [{ redactEgress: false }, { redactEgress: true, backend: { redact: false } }]) { + await withGateway(options, async ({ post, seen }) => { + await post('/v1/chat/completions', { model: 'deepseek-chat', messages: [{ role: 'user', content: userSecret }] }); + assert.equal(seen[0].body.messages[0].content, userSecret); + }); + } +});