Server Freezes (2026-07-08)

[Cibershield]

========================================================================

1. Summary

========================================================================

The production server froze at times (total blocks of

1-2 min, unanswered services). Cause: sessions of Claude Code (CLI

claude) running like root that were left open and grew in memory

Until the RAM runs out and fires the kernel's OOM killer. The report of

Today ("agot blocked") corresponded to the SAME event of 02:44,

Not to a new one. From ~03:00 the server is stable.

========================================================================

2. TIMELINE (2026-07-08)

========================================================================

01:52 Start root session running claude v2.1.204.

02:44 That session scales to 10.7 GB (peak 13.3 GB) -> OOM GLOBAL.

The kernel kills claude; systemd-journald dies by watchdog;

RCU stalls -> TOTAL FREEZING of the server.

02:54 Memory limits are applied in user-0.slice (mitigation).

02:45-04:19 Another session claude reaches 5.1 GB and is CONTAINED by

The limit, without taking down the server -> the mitigation worked.

03:00 onwards Zero OOM, zero watchdog, zero stalls. Stable.

========================================================================

3. ROOT CAUSE

========================================================================

Claude Code v2.1.204 memory leak in long sessions: a single

Process reached 10.7 GB. The server had no mattress (swap) or limits

From memory, so a single uncontrolled session knocked down EVERYTHING

(Including Docker, journald and production services).

It was NOT a problem of Nomina, SD Invoice, Postgres or the

Docker containers: all healthy during the incident.

Note: claude update confirms that 2.1.204 is ALREADY the latest version, it is

Say, there is still no patch for the leak. The protection falls on the

Limits + swap + monitoring described below.

========================================================================

4. MEASURES APPLIED

========================================================================

A) 4 GB swap (/swapfile, persisted in /etc/fstab). Santee mattress

Peaks of memory.

B) Limit added in user-0.slice (all root sessions):

MemoryHigh=6G, MemoryMax=8G. Does not affect Docker/containers

(Run in system.slice, without limit).

C) Limit PER SESSION: drop-in

/Etc/systemd/system/session-.scope.d/50-memory.conf with

MemoryMax=5G. Each SSH/login session is bumped; a claude

Wild dies only at 5 GB before dragging the rest.

WARNING: the legitimate use observed was ~5.1 GB, so this

5 GB cap can kill legitimate claude sessions in tasks

Heavy (value expressly chosen by the user). If it is cut

Legitimate work, raise the value with:

Systemctl set-property... (or edit the drop-in) and daemon-reload.