fix(anvil): survive OOM-truncated state + bound memory growth - #11
Conversation
The internal anvil devnet (rpc-dev) crash-looped ~4900 times until Caddy
served only 502s. Root cause: with a fixed block-time anvil's in-memory
chain history grows without bound; on the ~3.8 GB / no-swap VPS it reached
~3.7 GB and the global OOM-killer killed it mid state-snapshot, truncating
/var/lib/anvil/state.json to 0 bytes. anvil's --state then refuses to parse
the empty file ("EOF while parsing a value at line 1 column 0", exit 2),
wedging the unit permanently.
Three layered fixes in the anvil role:
- anvil-state-guard.sh as ExecStartPre: quarantines an empty/corrupt
state.json (moves it aside) so a truncated snapshot self-heals to a fresh
chain instead of crash-looping. A devnet can lose state; it can't be down.
- --prune-history (default 5000) bounds in-memory history so the box can't
OOM in the first place. --transaction-block-keeper is exposed too (off).
- Optional cgroup MemoryHigh/MemoryMax in the unit, set for the rpc-dev VPS
in group_vars (2G/2560M), so a runaway is confined to its own slice rather
than letting the global OOM-killer pick Caddy or sshd.
Verified: template renders to valid systemd for flags on/off; ansible-lint
(production), yamllint, shellcheck, KICS (0 high/critical) all pass; molecule
converge+verify boots the new unit (anvil up with --prune-history, chain-id
0x7a69, loopback-only, 401 gate intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces memory limits and a state-resilience guard for the Anvil service to prevent and recover from OOM crashes. Feedback on the state-guard script highlights that using jq to validate large state files on resource-constrained hosts could trigger further OOM events, suggesting a lightweight trailing-character check instead. Additionally, it is recommended to use a static filename for corrupt state files rather than timestamps to prevent disk space exhaustion during repeated crash loops.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
Pull request overview
This PR hardens the anvil Ansible role to prevent a devnet outage mode where an OOM-killed anvil leaves an empty/partial state.json, causing --state to fail and systemd to crash-loop indefinitely. It adds a pre-start guard to self-heal from truncated snapshots, bounds in-memory growth, and optionally contains memory usage via systemd cgroup limits (with host-specific ceilings applied for the devnet VPS).
Changes:
- Add an
ExecStartPreguard script that quarantines empty/invalidstate.jsonso anvil can start a fresh chain instead of crash-looping. - Enable bounded history retention via
--prune-historyby default, with an optional--transaction-block-keeperknob. - Support optional systemd
MemoryHigh/MemoryMaxlimits (enabled foranvil_devnet) to avoid global OOM impacts.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| ansible/roles/anvil/templates/anvil.service.j2 | Adds ExecStartPre state guard, optional --prune-history/--transaction-block-keeper, and optional cgroup memory ceilings. |
| ansible/roles/anvil/templates/anvil-state-guard.sh.j2 | New guard script to detect/quarantine empty or invalid JSON snapshots before anvil starts. |
| ansible/roles/anvil/tasks/main.yml | Installs the guard script before the unit so ExecStartPre always has a valid target, and restarts anvil when it changes. |
| ansible/roles/anvil/defaults/main.yml | Introduces defaults for pruning/history knobs, memory ceiling knobs, and the guard script path. |
| ansible/inventory/group_vars/anvil_devnet.yml | Sets concrete MemoryHigh/MemoryMax ceilings for the constrained devnet VPS and documents the incident context. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address review on the ExecStartPre guard. It runs inside the unit's cgroup, so it is subject to the same MemoryMax as anvil — validating a large snapshot with jq could itself be OOM-killed and re-wedge startup, the exact failure the guard exists to prevent. Replace the jq parse with an O(1) check: file is non-empty and its last non-whitespace byte is a JSON close token (} or ]); a truncated write won't end that way. This also drops the jq dependency. Quarantine to a static '.corrupt' name instead of a timestamped one so a repeated crash/OOM loop overwrites a single file rather than accumulating snapshots and exhausting disk on the constrained VPS; the latest sample still suffices for forensics. Drops the date dependency too. Verified: shellcheck clean; behavioural test covers no-file/empty/truncated/ valid (valid snapshot preserved, empty and truncated quarantined). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What happened
The internal anvil devnet behind
rpc-dev.decdn.orgwas returning HTTP 502 for every authenticated RPC call. Caddy, TLS, and basic-auth were all healthy — anvil itself was down.anvil.servicewas crash-looping (~4,900 restarts). Root cause, in sequence:--block-time 2it grows without bound.anon-rss ~3.7 GBand the global OOM-killer killed it at09:47:48— mid--state-intervalsnapshot./var/lib/anvil/state.jsonto 0 bytes.--staterefuses to parse an empty file (EOF while parsing a value at line 1 column 0, exit 2) → systemd restarts → same failure → permanent loop.No backup existed and the file was already empty, so the chain state was unrecoverable regardless (acceptable for a devnet).
The fix — three layers in the
anvilroleanvil-state-guard.shasExecStartPre— quarantines an empty/corruptstate.json(moves it aside) so anvil starts a fresh chain instead of wedging--prune-history(default5000) bounds in-memory history;--transaction-block-keeperexposed too (off by default)MemoryHigh/MemoryMax, set for rpc-dev ingroup_vars(2G/2560M)All knobs default to safe, generic values in
roles/anvil/defaults/main.yml; the host-specific memory ceilings live ininventory/group_vars/anvil_devnet.ymlwith the incident noted.Verification
ansible-lint(production profile) ✅ ·yamllint✅ ·shellcheck✅ · KICS 0 critical / 0 high ✅molecule converge+verify✅ — boots the new unit: anvil up with--prune-history, chain-id0x7a69, loopback-only binding, 401 auth gate intact.Out of scope / ops follow-up
state.json+ restart on rpc-dev) was applied separately by the operator — this PR is the durable prevention.decdn-node-1andanvil-vpsat the samerpc-dev.decdn.org, which contradicts the "keep on a separate host" guidance — flagging separately.🤖 Generated with Claude Code