Summary
Add functionality to save/load precomputed data to/from disc. This includes precomputed representations for the full dataset, as well as mean, var of the train set used for standardization, and other runtime properties computed from the dataset itself.
Motivation
Currently, the dataset is precomputed over and over again when training models on the same dataset using the same representation. Sweeping and experimentation speed could be optimized by eliminating the need to precompute the dataset every time.
Proposal
The precomputed data must be uniquely identified and saved to the disc and (if available) loaded by the data pipeline before any data splitting.
Summary
Add functionality to save/load precomputed data to/from disc. This includes precomputed representations for the full dataset, as well as mean, var of the train set used for standardization, and other runtime properties computed from the dataset itself.
Motivation
Currently, the dataset is precomputed over and over again when training models on the same dataset using the same representation. Sweeping and experimentation speed could be optimized by eliminating the need to precompute the dataset every time.
Proposal
The precomputed data must be uniquely identified and saved to the disc and (if available) loaded by the data pipeline before any data splitting.