Skip to content

fix(mcp): prevent OOM from long-lived stream retention - #467

Draft
JayZeeDesign wants to merge 1 commit into
mainfrom
cursor/fix-mcp-stream-oom-c2fc
Draft

JayZeeDesign wants to merge 1 commit into
mainfrom
cursor/fix-mcp-stream-oom-c2fc

Conversation

@JayZeeDesign

Copy link
Copy Markdown
Contributor

What this does

Fixes production OOM caused by long-lived MCP stream retention. The incident pattern: GET /mcp/ connections from Cursor, claude-code, and opencode hold SSE streams open for minutes to hours, accumulating memory (~10 MiB worst-case per connection) until RSS spikes from ~1 GiB to ~3+ GiB and OOM kills the pod at 4 GiB.

Adds MCPStreamLimitsMiddleware that enforces connection lifecycle limits:

  1. Per-instance concurrent connection limit (default 200) — reject new connections at 503 when at capacity
  2. Per-org concurrent connection limit (default 20) — fair sharing across teams, prevents one team from monopolizing connections
  3. Max connection lifetime (default 30 min) — force reconnection after N seconds; buffers are freed, clients auto-reconnect
  4. Idle timeout (default 5 min) — close connections with no activity (no receive/send)
  5. Prompt cleanup on disconnect — ensures buffers are freed when client disconnects (499)
  6. Structured logging with metrics — periodic snapshots of connection count, age distribution, rejection counts

Configuration

New environment variables in config.py:

Variable Default Purpose
TREG_MCP_MAX_CONNECTIONS 200 Max concurrent MCP connections per instance
TREG_MCP_MAX_CONNECTIONS_PER_ORG 20 Max concurrent connections per org (fair sharing)
TREG_MCP_MAX_LIFETIME_S 1800 Force reconnection after N seconds
TREG_MCP_IDLE_TIMEOUT_S 300 Close connections idle for N seconds

Why these defaults

  • 200 connections × ~10 MiB = 2 GiB headroom on a 4 GiB instance with ~1 GiB baseline
  • 30 min lifetime is long enough for most tool sessions, short enough to prevent unbounded accumulation
  • 5 min idle is conservative; SSE streams typically have keepalive traffic
  • Per-org limit ensures no single team can cause OOM

How it was tested

  1. Unit tests — 23 tests covering all limit types, timeout scenarios, disconnect cleanup
  2. Integration tests — verified existing MCP test suite (136 tests) still passes
  3. Import linter — all 14 contracts pass
  4. Manual verification — middleware correctly wraps MCP transport
# All new tests pass
uv run pytest tests/test_mcp_limits.py -v  # 23 passed

# Existing MCP tests unaffected  
uv run pytest tests/test_mcp.py tests/test_mcp_directory.py -q  # 113 passed

# Import rules maintained
uv run lint-imports  # 14 kept, 0 broken

Verification in production

After deploy:

  1. Check logs for mcp_stream_metrics entries — should show reasonable connection counts
  2. Monitor RSS — should plateau instead of growing unbounded under MCP load
  3. Watch for mcp_connection_rejected logs — indicates limits are working
  4. Confirm no user-visible impact — clients should auto-reconnect seamlessly

Rollback

Set very high limits to effectively disable:

TREG_MCP_MAX_CONNECTIONS=10000
TREG_MCP_MAX_LIFETIME_S=86400
TREG_MCP_IDLE_TIMEOUT_S=86400

Checklist

  • uv run --with pytest-xdist pytest -n auto -q passes locally (ran subset; full suite in CI)
  • Added or updated tests for the change (23 new tests in test_mcp_limits.py)
  • Updated the relevant docs/context/ fragment (no existing MCP fragment)
  • No secrets in the diff (keys, tokens, .env values)
Open in Web Open in Cursor 

Production OOM pattern: long-lived GET /mcp/ connections (Cursor, claude-code,
opencode) hold SSE streams for hours, accumulating memory (~10 MiB/connection)
until the 4 GiB limit kills the pod.

Add MCPStreamLimitsMiddleware that enforces:
- Per-instance concurrent connection limit (default 200)
- Per-org concurrent connection limit (default 20)
- Max connection lifetime (default 30 min, force reconnection)
- Idle timeout (default 5 min, close inactive streams)
- Prompt cleanup on client disconnect (499)
- Structured logging with periodic metrics

Configuration via TREG_MCP_* environment variables:
- TREG_MCP_MAX_CONNECTIONS: per-instance limit
- TREG_MCP_MAX_CONNECTIONS_PER_ORG: fair sharing across teams
- TREG_MCP_MAX_LIFETIME_S: force reconnection after N seconds
- TREG_MCP_IDLE_TIMEOUT_S: close after N seconds of inactivity

Clients that implement MCP reconnection (all major ones do) will automatically
reconnect when limits trigger. The 503 response includes Retry-After: 5.

Includes 23 tests covering all limit types and cleanup scenarios.

Co-authored-by: Jason Zhou <JayZeeDesign@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants