物理真机卡死过2次 gpt5.6分析的结果

[chentong0317]

结论：192.168.0.151 这次“卡死”是 CC Hub 的 Node 进程内存泄漏/失控，耗尽整机内存和交换分区，触发 Linux 全局 OOM。
关键证据：
2026-08-09 11:40:50：内核杀掉 CC Hub 的 node-MainThread。匿名常驻内存：29,626,620 KiB，约 28.2 GiB
交换分区：0 KiB 可用
Docker 退出码：137
随后 Docker 自动重启容器

同一容器在前一天也发生过完全相同的问题：2026-08-08 19:44:57
Node 内存约 30,038,260 KiB，约 28.6 GiB
同样是 OOM、退出码 137

OOM 前 Redis 的 bio_aof 线程处于 D 状态超过 245 秒，多个 Docker healthcheck 超时；这是内存耗尽后的 I/O 堵塞表现，不是最初原因。
当前容器没有内存限制：Memory=0，因此 Node 可以继续增长并拖垮宿主机。
PostgreSQL、Redis 本身内存都只有几百 MiB以内，持久化 replay 数据约 124 MiB，不足以解释这次故障。
磁盘根分区虽已使用 92%，但仍有约 36 GiB 可用，不是磁盘写满导致。
最可疑的配置路径是 CC Hub 当前启用的：
STORE_SESSION_RESPONSE_BODY=true
并且使用了请求重放/响应保存功能。结合两次都在容器启动后约十几分钟至二十多分钟增长到 30 GiB，基本可以判断是 CC Hub 0.9.2 的某个请求/流式响应处理路径存在持续内存增长。现有日志没有 heap dump，所以还不能精确指认具体 JavaScript 对象。
目前容器已自动恢复，CC Hub 内存约 380 MiB，但如果不限制内存或关闭响应体保存，同样的 OOM 很可能再次发生。

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• 内存泄露 #1283

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[chentong0317]

这是0.9.2版本出现的问题

[chentong0317]

CC Hub v0.9.2 production OOM and host-wide freeze

Suggested title

v0.9.2 Node process grows to ~30 GiB and triggers host-wide OOM; repeated exit 137

Summary

After upgrading CC Hub to v0.9.2, the application container repeatedly grows until the host runs out of RAM and swap. The host becomes effectively unresponsive, Docker healthchecks time out, Redis AOF I/O blocks, and Linux kills the CC Hub Node process. Docker then restarts the container automatically.

The same failure happened twice within about 16 hours. Downgrading only to v0.9.1 is not expected to avoid it because the memory-amplification paths were introduced with the v0.9.0 proxy features and remain present in v0.9.1/v0.9.2.

Environment

• Host: 192.168.0.151
• Hostname: bh
• OS/kernel: Ubuntu 22.04, Linux 6.8.0-111-generic
• Hardware: Inspur SA5212M4
• RAM: approximately 31 GiB
• Swap: approximately 10 GiB
• CC Hub image before rollback: ghcr.io/ding113/claude-code-hub:latest
• Actual image version before rollback: 0.9.2
• Image source revision: 4b634e1d396889e22756458f0b82db74f4453df9
• Node version in v0.9.2: v26.5.1
• Database: PostgreSQL 18
• Cache: Redis 7 Alpine with AOF enabled
• Docker restart policy: unless-stopped
• Container memory limit: none (HostConfig.Memory=0)

Sensitive values such as ADMIN_TOKEN and DB_PASSWORD are intentionally omitted.

Failure evidence

First occurrence

At 2026-08-08 19:44:57 +08:00, the kernel reported:

oom-kill: ... task_memcg=/system.slice/docker-15b1e2a5...scope,
task=node-MainThread,pid=778971,uid=1000
Out of memory: Killed process 778971 (node-MainThread)
total-vm:60718300kB, anon-rss:30038260kB, file-rss:1536kB

Docker then recorded an automatic restart with exit code 137:

restarting container ... exitCode=137 manualRestart=false restartCount=1

Second occurrence

At 2026-08-09 11:40:50 +08:00, the same container failed again:

memcached invoked oom-killer
Free swap  = 0kB
oom-kill: ... task_memcg=/system.slice/docker-15b1e2a5...scope,
task=node-MainThread,pid=1698679,uid=1000
Out of memory: Killed process 1698679 (node-MainThread)
total-vm:60661952kB, anon-rss:29626620kB, file-rss:0kB
oom_reaper: reaped process 1698679 (node-MainThread)

Docker then recorded:

restarting container ... exitCode=137 manualRestart=false restartCount=2

This is a host-level global OOM, not a normal application-level crash.

Host-wide symptoms before the OOM

The kernel and system services reported secondary pressure symptoms:

INFO: task bio_aof blocked for more than 122 seconds
INFO: task bio_aof blocked for more than 245 seconds

The blocked thread was Redis AOF fdatasync() waiting in the ext4 journal commit path. Docker healthchecks for multiple containers timed out. systemd-oomd also killed GNOME-related processes before the final global OOM. The host itself did not reboot.

Relevant configuration

The deployment used the following relevant settings:

STORE_SESSION_MESSAGES=false
STORE_SESSION_RESPONSE_BODY=true

ENABLE_REQUEST_REPLAY was not explicitly set in .env; in the v0.9.2 image configuration schema its default is enabled. The v0.9.x proxy path also enables the stream gate, replay, discovery/shadow attempts, and related cache/affinity features by default or through runtime settings.

Excluded causes

• PostgreSQL and Redis were not the dominant memory consumers. At the time of diagnosis they were each using only hundreds of MiB.
• The persistent replay_payloads table contained about 1,479 rows and occupied about 124 MiB, far below the approximately 30 GiB Node RSS.
• Redis memory after recovery was about 33 MiB.
• In the recent request window there was no evidence of a traffic spike large enough to explain 30 GiB: about 20 requests in 20 minutes, maximum messages_count=62, maximum input_tokens=30,860.
• The root filesystem was at 92% usage but still had about 36 GiB free; /www had substantial free space. There was no disk-full error.
• The host had been running for about 89 days, so this was not caused by a host reboot.

Most likely root cause

The production behavior is consistent with the memory-amplification paths described in upstream PR #1405, titled fix(proxy): prevent shared mutation and replay memory retention.

That PR identifies two independent retention paths:

1. Concurrent shadow/hedged attempts deep-clone large request messages and ArrayBuffer payloads. Shared nested request state can also be mutated by rectifiers.
2. ReplaySpool retains accumulated and queued payload batches while Redis/PostgreSQL writes are blocked. Abort/disable races can release the concurrency quota before fenced cleanup has actually completed.

The observed facts match this diagnosis:

• Node RSS repeatedly reaches about 30 GiB.
• Redis AOF becomes blocked while the Node process is under pressure.
• The failure repeats after automatic container restart.
• The problem appeared after moving to the v0.9.x proxy feature set.

The exact retaining object cannot be proven from the production logs alone because no heap snapshot was captured before the OOM. However, the upstream PR is a direct code-level match for the observed failure and includes focused regression tests for replay cleanup and request isolation.

Version analysis

• v0.9.0 introduced Stream Gate, Request Replay, bounded streaming discovery, prefix affinity, and cache-effectiveness paths.
• v0.9.1 contains those same paths.
• v0.9.2 is only two commits ahead of v0.9.1; its implementation change adds DeepSeek reasoning_content to the stream-frame classifier. It does not contain the memory-retention fix.
• PR #1405 is still open and has not been released.

Therefore, the issue may have become visible immediately after the v0.9.2 upgrade, but the likely regression area is the v0.9.x replay/shadow proxy implementation rather than the single reasoning_content classifier line added by v0.9.2.

Workaround and rollback result

The deployment was pinned to ghcr.io/ding113/claude-code-hub:v0.8.10 and only the app container was recreated. PostgreSQL and Redis were left running.

After rollback:

Health API: HTTP 200
Reported version: 0.8.10
Database: up, latency approximately 2 ms
Redis: up, latency approximately 1 ms
Proxy: up, latency approximately 1 ms
App memory shortly after startup: approximately 300 MiB

The v0.8.10 image was already present on the host, so no image download was required.

Requested maintainer action

• Please merge or otherwise release the fix from PR fix(proxy): prevent shared mutation and replay memory retention #1405.
• Please publish a patched tag that includes the request copy-on-write and ReplaySpool cleanup changes.
• Please add a production regression test that runs replay/shadow workloads long enough to detect RSS growth, not only short functional success.
• Please consider a default container memory limit or an internal bounded-memory failure mode so one proxy process cannot exhaust the host.
• Please document the interaction between STORE_SESSION_RESPONSE_BODY, Request Replay, shadow attempts, and PostgreSQL/Redis backpressure.

Reproduction notes

The exact request sequence has not yet been reduced to a minimal public fixture. In production, the failure reproduced twice with ordinary proxy traffic after the v0.9.2 deployment. A useful reproduction should include:

• OpenAI Responses or OpenAI-compatible streaming requests with large conversation histories;
• concurrent shadow/hedge attempts;
• Request Replay enabled;
• deliberately slow or blocked Redis/PostgreSQL writes;
• client aborts and provider fallback during an active stream;
• RSS/heap/external/ArrayBuffer sampling until cleanup completes.

[ding113]

hi，可尝试在系统设置中关闭 Replay 功能，并开启高性能模式使用
相关问题也正在复现当中

[vantis-zh]

+1 退回0.91也有内存泄漏导致重启问题