The mechanism
Router.previous_models is an instance attribute, not per-request state:
# router.py:579
self.previous_models: List = ...
log_retry appends to it and then attaches the shared list to the current request's metadata:
# router.py:6429-6434
if len(self.previous_models) > 3:
self.previous_models.pop(0)
self.previous_models.append(previous_model)
kwargs[_metadata_var]["previous_models"] = self.previous_modelsSo every request served by that pod after a fallback carries the last ≤4 fallback events from whichever members happened to trigger them — and each entry is built with previous_model[k] = v over the failed attempt's kwargs, which includes the header dicts.
Measured
Community primary, 2026-09-09, one scan of a 20-minute window (7,105 rows):
rows with metadata.previous_models carrying `bearer sk-` 11
model_groups affected: qwen3.6, deepseek-v4-flash-fallback (i.e. our own fallback chain)
A reviewer's independent 2-minute window found 4 rows, 4/4 carrying a token.
The affected groups are exactly the ones with a configured fallback, which is consistent with the mechanism rather than coincidental.
Why this is worse than #43
#43 is "the row contains the requester's own credential". That is bad, but the blast radius is the operator with DB access.
This is cross-member: member A's row can contain member B's live API key. And GET /spend/logs/ui/{request_id} returns proxy_server_request in full to the owner of the row (spend_management_endpoints.py:2109-2196, ownership checked on the row at :3560) — the ownership check protects the row, not the foreign credential inside it. /spend/logs/ui is in spend_tracking_routes (_types.py:581-591).
Exploitability is NOT confirmed and should be before this is prioritised: it depends on whether a community member key is permitted that route in our configuration. I have not tested it, because doing so needs a real member key and per feedback_e2e_test_pollutes_real_account that is not something to do casually. Confirm with a purpose-made key, not a member's.
Not closed by nan-devops#290
That PR scrubs the credential from secret_fields.raw_headers and proxy_server_request["headers"] in the pre-call hook, in place. Two reviewers argue it probably closes this one as a side effect — the scrub runs before the router attempts anything, log_retry copies by reference, so the entry should point at the already-cleaned dict. That is an argument, not a measurement. #290 states it as open for exactly that reason.
Verification after #290 deploys (this is the task):
- Force a fallback on community (the
deepseek-v4-flash → deepseek-v4-flash-fallback chain, or glm5.3 → qwen3.6) with a purpose-made key.
- Query, NULL-safe, projecting nothing but counts:
SELECT count(*) FILTER (WHERE "proxy_server_request"->'metadata'->'previous_models' IS NOT NULL) AS with_pm,
count(*) FILTER (WHERE ("proxy_server_request"->'metadata'->'previous_models')::text ILIKE '%bearer sk-%') AS leaking
FROM "LiteLLM_SpendLogs" WHERE "startTime" > now() - interval '15 minutes';
- If
leaking > 0, the scrub does not reach it and this needs its own fix — most likely scrubbing metadata.previous_models in the post-call path, or asking upstream why a Router instance attribute is attached to per-request metadata at all.
Regardless of the above
The shared-list design is worth reporting upstream on its own merits: previous_models mixes requests from different callers into one another's log rows even when there is no credential in them (prompts, model choices, timing). That is a tenancy leak in a multi-tenant proxy.
Evidence trail
The mechanism
Router.previous_modelsis an instance attribute, not per-request state:log_retryappends to it and then attaches the shared list to the current request's metadata:So every request served by that pod after a fallback carries the last ≤4 fallback events from whichever members happened to trigger them — and each entry is built with
previous_model[k] = vover the failed attempt's kwargs, which includes the header dicts.Measured
Community primary, 2026-09-09, one scan of a 20-minute window (7,105 rows):
A reviewer's independent 2-minute window found 4 rows, 4/4 carrying a token.
The affected groups are exactly the ones with a configured fallback, which is consistent with the mechanism rather than coincidental.
Why this is worse than #43
#43 is "the row contains the requester's own credential". That is bad, but the blast radius is the operator with DB access.
This is cross-member: member A's row can contain member B's live API key. And
GET /spend/logs/ui/{request_id}returnsproxy_server_requestin full to the owner of the row (spend_management_endpoints.py:2109-2196, ownership checked on the row at:3560) — the ownership check protects the row, not the foreign credential inside it./spend/logs/uiis inspend_tracking_routes(_types.py:581-591).Exploitability is NOT confirmed and should be before this is prioritised: it depends on whether a community member key is permitted that route in our configuration. I have not tested it, because doing so needs a real member key and per
feedback_e2e_test_pollutes_real_accountthat is not something to do casually. Confirm with a purpose-made key, not a member's.Not closed by nan-devops#290
That PR scrubs the credential from
secret_fields.raw_headersandproxy_server_request["headers"]in the pre-call hook, in place. Two reviewers argue it probably closes this one as a side effect — the scrub runs before the router attempts anything,log_retrycopies by reference, so the entry should point at the already-cleaned dict. That is an argument, not a measurement. #290 states it as open for exactly that reason.Verification after #290 deploys (this is the task):
deepseek-v4-flash→deepseek-v4-flash-fallbackchain, orglm5.3→qwen3.6) with a purpose-made key.leaking > 0, the scrub does not reach it and this needs its own fix — most likely scrubbingmetadata.previous_modelsin the post-call path, or asking upstream why a Router instance attribute is attached to per-request metadata at all.Regardless of the above
The shared-list design is worth reporting upstream on its own merits:
previous_modelsmixes requests from different callers into one another's log rows even when there is no credential in them (prompts, model choices, timing). That is a tenancy leak in a multi-tenant proxy.Evidence trail
secret_fields.raw_headers) #43 (the same column, single-member scope)