Repository navigation
Close abandoned CSV readers; bump clojask-io to 1.0.7 - #85
Merged
Merged
Conversation
clojask-io 1.0.7 closes its reader when :data is fully consumed and
returns a :close function for consumers that stop early. Use both:
- dataframe construction opened three to four readers per call (header
probe, stat, output fn) and abandoned them all after at most one line.
It now realizes the io result once, reuses it for the header, size and
output function, and closes it.
- The stored read function gains a second arity returning
{:data ... :close ...} so partial consumers can reach the reader.
- preview fetches through that arity and closes in a finally, which also
covers every errorPredetect and print-df, including when a preview
throws (the errorPredetect failure path).
Readers abandoned when an Onyx job dies mid-compute still wait for GC;
full computes close on exhaustion via 1.0.7.
Test written first and seen failing (5 readers opened, 0 closed): counts
opened vs closed readers across construction, preview, and a failing
errorPredetect.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch 4 of the agreed plan, clojask side. clojask-io 1.0.7 (released today) closes its reader when
:datais fully consumed and returns a:closefunction for consumers that stop early. This PR bumps the dep and uses:closeat every site that abandons a reader partway — which turned out to be most of them:dataframeconstruction opened three to four readers per call (:clojask-ioprobe, header, stat, output fn) and abandoned each after at most one line. It now realizes the io result once, reuses it for the header row,:path,:sizeand:output, and closes it.{:data ... :close ...}so partial consumers can reach the reader; the 0-arity behavior is unchanged (Onyx input plugin, full computes).previewfetches through the new arity and closes in afinally. That coversprint-df, the publicpreview, and everyerrorPredetect— including when the previewed pipeline throws, which is exactly the errorPredetect failure path.Readers abandoned when an Onyx job dies mid-compute still linger until GC (nothing holds the
:closehandle across the job); full computes now close on exhaustion via 1.0.7. Thebindextensions still open stat-only readers without closing them — left as is, they build their own io maps and are out of scope here.Test written first and seen failing (5 readers opened, 0 closed): wraps
clojask-io.input/read-fileto count opened vs closed readers across construction, preview, and a failingerrorPredetect. Suite: 50 tests / 184 assertions, green locally on JDK 25.No version bump; rides with #83/#84 in a later 2.0.6.