Repository navigation
fix(exports): close failed Parquet output streams - #227
Conversation
Miracle656
left a comment
There was a problem hiding this comment.
Right fix, and it cleans up after itself rather than only closing the handle.
A failed Parquet export previously left the write stream open and the partial file on disk. Both matter and for different reasons: the open handle is a descriptor leak that accumulates silently until the process hits its limit — at which point unrelated work starts failing, which is a miserable thing to debug — and the orphaned partial file is worse than useless, because a half-written Parquet is still a plausible-looking file that someone or something may later pick up.
output.destroy() then fs.promises.unlink(output.path) with the ENOENT swallowed is the correct order and the correct tolerance: a stream that never created its file must not turn a real export error into a confusing cleanup error on the way out.
The test asserting the error still reaches the error handler is the one I would have asked for — cleanup code that accidentally swallows the original failure is a common way to make an outage undiagnosable.
726 tests pass, tsc --noEmit clean, verified against current main. Integration tests is pre-existing red on main.
Merging.
Summary
If a Parquet export fails while querying rows, appending a row, or closing the writer, the handler attempts to unlink the temporary file while its output descriptor remains open. Retain ownership of the underlying stream, finish pending writes, and wait for descriptor closure before removing the file and forwarding the original error. Run the truncation probe before opening the file.
The regression suite uses real temporary files and the real Parquet library, with controlled database and writer failures. It also reads a successful capped download back through the Parquet reader to check compatibility.
Related issue
Follow-up to the export endpoint in #149 and export validation in #199. No existing open issue or PR was found for writer cleanup on failure.
Type of change
Validation
Windows, Node.js 24.14.0, based on
4428d53:npm run db:generate(same schema and shared dependency installation)npx tsc --noEmitnpm run typechecknpm run buildnpm test: 54 suites, 650 tests passed, including coverage thresholds.Tests verify descriptors are closed and files are removed before an error response completes. No live database or RPC is used. The change covers file-generation failures; cancellation during the later download is outside this scope. AI assistance: implemented with Codex and validated locally.
Checklist
npx tsc --noEmitpassesnpm run buildpasses