Write JSON, CSV and the sidecar atomically

[jehrr]

The audit's second finding, worked. Real, and the half it does not mention is the one that matters most.

The defect, reproduced

open(path, "w") truncates before a single byte is written, so a crash, a kill or a full disk partway through json.dump leaves a shorter file where a complete one was. Measured here, by writing a good run and then interrupting the next one mid-dump:

truncating (before)   good=204292  after=    28  INVALID JSON
atomic (this change)  good=204292  after=204292  valid        strays=none

That breaks the same promise save already keeps when it refuses to overwrite good output with an empty result — by a different route, with the previous good run destroyed by the attempt to replace it rather than by its outcome.

The sidecar is the part the audit did not mention and the part that matters most. <out>.meta.json is the file a consumer branches on, so a truncated one beside good rows reads as a broken run over data that is fine.

Lifted, not written

All three writers now go through one _atomic: temp file in the target's own directory (os.replace is atomic only within one filesystem — a temp file under /tmp can be on another, where the rename degrades to an interruptible copy), flush + fsync before the rename, and the temp file unlinked on any exception including KeyboardInterrupt.

It is taken from a sibling rather than written here, because the one part that is easy to get wrong has been got wrong across the family. NamedTemporaryFile creates its file 0600 and a rename keeps that, so an output nobody else can read is the silent default.

Measured 2026-10-09 by calling each sibling's writer and stat-ing the file it produced, not by grepping for a umask line:

	repos
leave the output 0600	8 of 11
hardcode 0644	1
derive it from the umask	2

This takes the third kind: a new file gets what open() would have given it, and a target somebody tightened deliberately keeps its own mode.

Verification

Five faults planted, suite required to go RED on a named check:

planted fault	caught by
write_json back to a truncating open	the previous good file survives an interrupted write (+1)
the sidecar back to a truncating open	write_run_meta() writes through _atomic
the mode left as NamedTemporaryFile's 0600	a new output is readable, not 0600 (got 0o600 under umask 022)
the temp file not cleaned up on failure	...and no half-written temp file is left beside it
newline="" dropped on the CSV	(see below)

The suite also carries its own control inline: it asserts that a truncating writer really does destroy the file before asserting that this one does not, so "the file survived" cannot quietly become a property of the test.

The fifth control stayed green, and that is reported rather than smoothed over. Dropping newline="" changes nothing on Linux — the doubled carriage return only appears where os.linesep is \r\n — so the behavioural check cannot fail on CI however it is written. That is a hole in the suite, not a pass. The byte-level assertion is kept as a sanity check and labelled as one, and the guard that can actually fail is now the wiring: write_csv() must hand newline="" to _atomic.

What is NOT changed, and why

• No run identifier for the JSON/CSV/sidecar set. The atomic rename is right and is taken. The transaction id is solving a problem this contract already solves by construction: a failed run writes no sidecar at all, and save refuses to overwrite good output with an empty result. Adding an id would add a mechanism to enforce something that is enforced by not writing — and a second thing to keep consistent is a second thing that can disagree.

  What the audit is right about underneath that recommendation is the failure window, and os.replace closes it: each file is either the previous run's or this one's, never half of either.

• CSV formula neutralisation is not included. It is a known family TODO and the audit did not raise it. Measured on a real 213-row run before deciding: 0 of 3,620 string cells are formula-shaped. It is worth shipping on its own merits and is not worth bundling into a fix for a different problem.

🤖 Generated with Claude Code