Repository navigation
Fix select-col, delete-col and reorder-col; release 2.1.1 - #93
Merged
Merged
Conversation
select-col and delete-col had no effect, and reorder-col dropped earlier operations and moved filters and types onto other columns. All three now set the dataframe's output columns and their order, which compute, preview, print-df and joins follow, without touching operations, filters or types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes
select-col,delete-colandreorder-col, and releases 2.1.1.Bugs
select-colanddelete-coldid nothing. They recorded the deleted columns, but since 1.1.0 (January 2022) compute no longer read that record, so every column was still written andget-col-namesstill listed it.reorder-colrebuilt the column descriptions, which dropped every earlieroperateon existing columns. Earlier filters and types stayed at their old positions, so they applied to whichever column moved there. The result was silently wrong output (e.g. a filter onSalaryending up onDepartmentand returning no rows) or a rejected reorder with a misleading error. It also rejected columns created byoperate.Fix
The three functions now only choose which columns compute writes and in which order. They store a list of output columns on the dataframe, and rollback covers it. Operations, filters and types are not touched, so they stay on their columns.
get-col-names,compute,previewandprint-dffollow the chosen columns. An explicit:select/:excludepicks from them.operate, group-by, aggregate or join key can still use a deleted column, the same as with a column left out by:select.operateafter a delete or reorder is added at the end.reorder-collist that isn't every column exactly once, and any of the three aftergroup-byoraggregate(use:selectthere).preview :format trueapplied formatters by position, which became wrong once columns can be left out; it now applies them by column.The unused
setColInfo/setRowInfocode is removed.Tests
Five new tests in
regression_test.clj: reorder afterset-type,operate, filter and a formatter; delete and select with filters on deleted columns and new columns; aggregate on a deleted column; joins on both sides, swapped and with:select, on disk and in memory; and the rejected inputs. Against the old code they give 12 failures and 2 errors. Suite: 60 tests / 306 assertions, green on JDK 25.Version bumped to 2.1.1 in
project.cljand the README install snippets.