Repository navigation
Run aggregate and group-by computes on the local engine - #89
Merged
Merged
Conversation
With clojask.engine=local, aggregate-only, group-by and group-aggregate computes no longer start Onyx. engine/start-aggre-only folds the rows in input order on the calling thread, engine/start-groupby writes the rows to their group files or the in-memory groups, and engine/start-aggre aggregates one group per pool task. They take the same arguments as the Onyx drivers they replace. Joins still run on Onyx, including their group-by phase. engine-test runs each of these computes on both engines and compares the sorted output, with Onyx made unreachable during the local run. It also checks that a failing aggregate surfaces as an ExecutionException on both engines and leaves no group files.
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 2 of replacing Onyx (stage 1 was #88). With
clojask.engine=local(orCLOJASK_ENGINE=local), aggregate-only, group-by and group-aggregate computes no longer start Onyx. The default engine is still Onyx.engine/start-aggre-onlyfolds the rows in input order on the calling thread and writes the result.engine/start-groupbywrites the rows to their group files or to the in-memory groups.engine/start-aggreaggregates one group per pool task.They take the same arguments as the Onyx drivers they replace;
DataFrame.computeAggreandcomputeGroupAggrepick one by the flag. Joins, including their group-by phase, still run on Onyx (stage 3).Tests
New
engine-testruns each of these computes on both engines and compares the sorted output, with Onyx made unreachable during the local run: aggregate-only, group-aggregate on disk and in memory, a multi-row aggregate with:select, group-by only, and a failing aggregate (stagedExecutionExceptionon both engines, no group files left). Written first; before the port only the local runs failed.lein test: 57 tests / 216 assertions green withCLOJASK_ENGINE=localand with the default engine (JDK 25).Benchmark
Synthetic 1.8M-row, 160 MB Compustat-like CSV, JDK 25, 8 cores, seconds, Onyx → local:
Group-aggregate on disk is dominated on both engines by opening the group file once per row.
Output is identical except for the last digits of group means (max relative difference 4e-15): Onyx feeds a group's rows in arbitrary order, so its disk and in-memory runs differ from each other; the local engine adds them in input order and its two runs match exactly.