Set GOTMPDIR per task so interrupted Go builds don't leak /tmp#36
Open
e-jung wants to merge 2 commits into
Open
Set GOTMPDIR per task so interrupted Go builds don't leak /tmp#36e-jung wants to merge 2 commits into
e-jung wants to merge 2 commits into
Conversation
… /tmp Go's GOTMPDIR is unset, so every go build/test creates numbered /tmp/go-build* dirs. Go cleans them on a clean exit but LEAVES THEM when interrupted (signal, timeout, OOM, full disk), accumulating and filling the disk over time. Give each task its own temp root at /tmp/fm-<id>/ with Go's build temp nested at gotmp/. fm-spawn creates the dir (Go won't mkdir GOTMPDIR), exports GOTMPDIR into the crewmate pane so the agent and child processes inherit it, and records tasktmp= in meta. fm-teardown reads tasktmp= and removes the whole root on cleanup, deterministically. GOTMPDIR (not TMPDIR) is the targeted knob: TMPDIR is too broad (affects every program's temp). The nested root is extensible: other per-task temp can live under /tmp/fm-<id>/ later. Backward compat: tasks spawned before this change have no tasktmp= in meta; teardown tolerates the empty value as a no-op. The daily fm-disk-cleanup.sh cron remains a safety net for any pre-fix stray dirs.
The structural grep -F assertions deliberately match literal $TASK_TMP in the fm-spawn source; add per-line shellcheck disable=SC2016 (the codebase's existing pattern, e.g. bin/fm-spawn.sh) so CI lint passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Go's
GOTMPDIRis unset, so everygo build/go testin a crewmate pane creates numbered temp dirs (/tmp/go-build*). Go cleans these on a clean exit but leaves them behind when a build is interrupted — signal, timeout, OOM, or full disk. Over time these accumulate and fill the disk on the firstmate VPS.Fix
Give each task its own Go temp dir that teardown cleans deterministically.
bin/fm-spawn.shcreates a per-task temp root/tmp/fm-<id>/with Go's build temp nested at/tmp/fm-<id>/gotmp/(Go won'tmkdirGOTMPDIR, so it must exist first). Itexport GOTMPDIR=...into the crewmate's pane shell so the agent and every child process inherit it, and recordstasktmp=in the task's meta.bin/fm-teardown.shreadstasktmp=from meta andrm -rfs the whole root on cleanup, so interrupted builds can never outlive the task.GOTMPDIR(notTMPDIR) is the targeted knob —TMPDIRis too broad (it redirects every program's temp, not just Go's). The nested root is extensible: other per-task temp can live under/tmp/fm-<id>/later, and teardown cleans one deterministic path.Backward compatibility
Tasks spawned before this change have no
tasktmp=line in their meta. Teardown reads the value with an empty-tolerant grep and treats the empty string as a no-op ([ -n "$TASK_TMP" ] && rm -rf ...), so pre-existing tasks tear down exactly as before.Safety net
The daily
~/.local/bin/fm-disk-cleanup.shcron (rm -rf /tmp/go-build* ...) remains in place and sweeps any pre-fix stray dirs or anything missed in edge cases.Tests
tests/fm-gotmp.test.sh(TAP style) covers:fm-spawncreatesgotmp/and recordstasktmp=in meta (structural contract + behavioral simulation).fm-teardownremoves the dir pointed to bytasktmp=(runs the real teardown via symlink against a fake root — exercises actual code, sources nothing).fm-teardownskips gracefully whentasktmp=is absent (backward compat).fm-teardownskips gracefully whentasktmp=points to a nonexistent dir.All existing tests (
fm-spawn-batch,fm-wake-queue) still pass.🤖 AI disclosure: Human-reviewed