litellm-community: SpendLogs is 154 GB for a 2-day window — 148 GB of it is TOAST

[sre-helmcode]

Surfaced while investigating #44. Not the same problem: the pruner works, and this is what makes each of its runs expensive.

Measured 2026-09-10, community primary (litellm-community-pg-7)

LiteLLM_SpendLogs   heap 2721 MB   total 154 GB
pg_toast_10712991        148 GB    30,240,880 chunks   index 2163 MB
n_live_tup          1,013,094

~146 KB of TOAST per row, against a heap row of ~2.7 KB. The retention window is 2d, so this is the steady state, not a backlog.

Why

store_prompts_in_spend_logs: true on community puts the entire caller request body into proxy_server_request. Community request bodies are large by nature — real ones reach 27.8 MB, and after nan-devops#298 the edge cap is 256 MiB. Anything over the ~2 KB TOAST threshold gets pushed out-of-line and compressed, and a body of a few hundred KB becomes tens of TOAST chunks.

Why it matters beyond disk

heap_delete() calls heap_toast_delete(). Deleting one SpendLogs row is therefore ~30 further deletes in the TOAST table plus its index. That turns every retention sweep into a much larger write event than the row count suggests:

sweep	rows	TOAST chunk deletes
*/30 steady state	~13k	~390k
7 */6 steady state	~157k	~4.7M
SPEND_LOG_RUN_LOOPS=500 worst case	501k	~15M

That lands on an archiver sustaining ~21 GB/h to R2, and on a single synchronous replica (minSyncReplicas: 1) that must fsync every COMMIT — if it falls behind on replay, primary writes block with no timeout. nan-devops#302 caps a run at 50k rows for exactly this reason, but that bounds the symptom, not the cause.

Options, none of them free

1. Stop storing prompts on community (store_prompts_in_spend_logs: false). Kills the bloat at the source. Cost: loses the request body from SpendLogs, which is what the credential-scrub work in nan-devops#290 and issues LiteLLM community: member API keys are stored in cleartext in SpendLogs (secret_fields.raw_headers) #43/LiteLLM: metadata.previous_models leaks one member's credentials into another member's SpendLogs row #45 are all reasoning about — check with those before touching it.
2. Shorten retention below 2d. Linear, and cheap to try, but 2d is already short for support questions.
3. Range-partition LiteLLM_SpendLogs by startTime and drop whole partitions instead of deleting rows. Reclaims instantly and generates almost no WAL. LiteLLM 1.95.0 already implements this (SpendLogsPartitionManager, gated on use_spend_logs_partitioning) — community runs 1.83.14, which does not have it. This is the real fix and it arrives with the version bump.
4. Truncate what goes into the body before it is stored — a hook-side cap on the persisted proxy_server_request. Most invasive, and it interacts with the scrub work.

Suggested order

Measure first: how much of the 148 GB is a handful of enormous rows versus broad. pg_stat_statements and a bounded percentile over pg_column_size(proxy_server_request) on a sample, not a full scan — that is how this primary was taken down earlier in the same session. Then decide between (1) and (3), with (3) preferred if the 1.95 upgrade is close.

Related: #43 (member keys in cleartext in SpendLogs), #45 (metadata.previous_models leak), #46 (scrub follow-ups), nan-devops#302 (sweep cap and cadence).