Repository navigation
Multi datasets per analysis support - #223
Merged
Merged
Conversation
added 14 commits
October 14, 2026 10:19
…atasets' list - runJaspResults gains multiDataSetJson: when the engine queues per-dataset slices, each queued .readDataSetRequestedNative call becomes one entry of datasets (keyed by dataset id, titles as attr(datasets, "dataSetNames")), passed to the analysis as the datasets argument. - While such a run is active, readDataSet*/readFullDataset/readDataSetHeader/ readDataSetByVariableTypes hard-stop: one 'current dataset' is meaningless when an analysis runs on several. The mode flag is set/restored around the run. Part of the jasp-desktop multi-dataset VariablesForm feature.
…analyses Encoded names embed their dataset id (DataSet::setupEncoderPrefix), so: - dataSetIdFromEncoded(): the dataset id out of 'JASPColumn_<dsId>_<n>' (type suffix and _For_Replacement postfix tolerated); NA for legacy/plain names, - dataSetNameFromEncoded(x, datasets): the dataset title via attr(datasets,'dataSetNames'), id-less names attributed to the primary (first) dataset, - getDataSetFor(x, datasets, default): routes an option value to its dataframe. Mirrors ColumnEncoder::dataSetIdFromEncoded in C++; pure R so analyses and tests work without an engine. Unit tests in test-multiDataSet-decoding.R.
…Encoded/getDataSetColumn In an aware run the options carry encoded column names (JASPColumn_<ds>_<n>.<type>) while the slice dataframes carry the originals - and during the analysis the engine's current encoder is whatever slice was read last, so decodeColNames alone cannot resolve them. Two ways to decode, in order: - .decodeColNamesForDataSet (new engine callback): decodes a value against the encoder of the dataset its name itself points to, - attr(datasets, 'dataSetEncodedNames'), snapshotted by runJaspResults right after each slice read while that dataset's encoder is still current (also works against engines without the callback), then the stripped value as-is. getDataSetColumn(x, datasets) = right dataset + right column in one call; unit tests extended.
For wrappers/R-syntax use there is no engine slice queue to read from, and none is needed: the datasets arrive as a named list, whose names are the ids and whose dataSetNames attribute carries the titles. So: - runJaspResults gains datasets = NULL: a directly supplied list switches on multi-dataset mode and is used as-is (ids normalised, titles defaulted to the names); the multiDataSetJson + queued-native-read path stays as the engine route. - runWrappedAnalysis forwards datasets (and stores every one of them in the syntax bridge via the new storeDataSets()). - getDataSetFor routes plain (non-encoded) names primary-first, then the first dataset that actually has the column (wrapper mode cannot encode per dataset), then default; tests updated accordingly.
… in getDataSetColumn - The `datasets <- NULL` initialisation line (from the original multiDataSet commit) nulled the new datasets parameter before the multi-mode check: the wrapper handout arrived as NULL and aware analyses silently ran their non-aware branch. Removed; the parameter default already covers it. - getDataSetColumn: when the routed dataset lacks the resolved column (bridge-encoded values without a usable id, e.g. the syntax bridge single-encoder form), scan all datasets for the resolved name before giving up.
Options and dataset slices arrive at R ENCODED by the same per-dataset ColumnEncoder (rbridge_readDataSetRequested encodes the slice headers just like the options), so analyses index datasets[[id]][[encodedValue]] as-is; result strings are decoded on the way out by Engine::sendString against every encoder the analysis references. - dataSetColumnFromEncoded and the attr(datasets,'dataSetEncodedNames') snapshot are gone: they decoded to ORIGINAL names to index dataframes that are (always were) encoded - the combination is what emptied the table cells in real runs. runJaspResults' queue branch keeps the slices exactly as the bridge hands them over. - dataSetIdFromEncoded now parses the real encoder format 'JASPColumn_<dsId>_<counter>_Encoded' (the old pattern, without the _Encoded postfix, never occurs - all ids parsed as NA and routing silently degraded to the primary dataset). - getDataSetFor/getDataSetColumn rewritten to direct encoded indexing with exact-name fallbacks (primary first, then scan - covers the plain-name jaspTools handover). Dangling ids no longer error on [[ by name. - testthat fixtures now model what the engine really delivers (encoded column names); added scan/lost-column and runJaspResults-shape cases.
…ne-parity - storeDataSets -> jaspSyntax::loadDataSets: every named dataset becomes a DataSet in the bridge workspace (title = list name, real id + column encoder), so the QML forms can SELECT them and the bridge can encode options per dataset exactly like the engine does. - runWrappedAnalysis aware branch uses jaspSyntax::loadQmlAndParseOptionsStatus and passes the returned multiDataSetJson to runJaspResults: the analysis gets its datasets through the same queued-slice branch a desktop run uses (encoded columns matching encoded options), instead of the raw R list whose plain names never lined up with anything the bridge produced. Parts travel as json strings (no R-side re-serialization). - jaspTools' direct datasets= handout branch is untouched.
…sliceKeyForDataSet routes encoded names across filter-keyed slices
…liceKeyForDataSet LOUD on multi-slice datasets (never silently first); primarySliceKey attr; docs to filter-keyed reality
CTO review 2026-10-09: single-column accessor whose fallback scanned every dataset and could silently serve a same-named column from the wrong one. Analysis authors index datasets[[key]][, cols] directly. The rlogic contract check now pins its absence. Rest of the slice family follows with the filter-id-only key space (Track B).
CTO decision 2026-10-09, enabled by the filter-id-only key space:
getSliceKey/getSlice/sliceDataSetId/sliceTitle/getDataSetFor/
dataSetIdFromEncoded/dataSetNameFromEncoded (and the .sliceKeyForDataSet
internals) all existed to bridge TWO key-spaces (dataset-id handout vs
filter-id queue) and to forgive missing selections - with one key-space
there is nothing left to bridge. The whole contract, documented in common.R:
datasets keyed by FILTER id - exactly what the form's
dataSetSelectionOption holds: datasets[[key]]
attr dataSetNames key -> dataset title (display only)
attr dataSetIds key -> dataset id
attr primarySliceKey the analysis' own slice
runJaspResults loses its datasets= parameter too: the R handout branch was
proved dead first (stop() in place, every suite green without touching it -
jaspTools sends multiDataSetJson since the bridge reroute, wrappers load
through the bridge too). storeDataSets is internal now (runWrappedAnalysis'
own boundary hop). The old accessors' cross-dataset name scans could silently
serve the wrong dataset; [[ on a wrong key yields NULL, visibly.
… updated - the token shape is C1's JASPColumn_D<dsId>_F<filterId>_<n>_Encoded minted by the per-filter encoder views (the pre-C1 JASPColumn_<dataSetId>_<counter>_Encoded + DataSet::setupEncoderPrefix description retired with the Phase 1 mint split); comment only, no R behavior change
JorisGoosen
force-pushed
the
multiDataSetAware
branch
from
October 19, 2026 09:41
c780e23 to
769e61e
Compare
JorisGoosen
commented
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the jasp-desktop multi-dataset VariablesForm feature.
See jasp-stats/jasp-desktop#6345