We need a flexible dataloader framework that by default supports both UCLCHEM and 3D-PDR datasets. The concept is that most expert users can define and simulate their own training dataset, then build a dataloader for it and run an hyperparameter optimization routine to find the best architecture for their specific usage.
The data generated by different astrochemical codes is fundamentally the same, but is organized differently. In the end it all reduces to a set of series with different external (simulation specific), input and output variables. We need to set a clear context for the "users".
- We need to define a standard format for loading the data. People use CSV and h5 for saving the data, we should specify the internal structure of the data (schema).
- Support existing datasets
- 3D-PDR h5 datasets:
- UCLCHEM (existing datasets)
- there is a new API in UCLCHEM 4.0 with
xarray
Existing on disk dataset structure and documentation
v1 Dataset (8,192 models)
header - 226 field names (8 physics + 215 species + 3 auxiliary)species - Array of 215 species namesmodel_ids - Array of model identifiers (strings)model_df - Initialization parameters array, shape (n_models, 3): [radfield_init, density_init, zeta_init]model_XXXX/pdr - Individual model datasets containing the actual data arrays
v2 Dataset (1,681 models)
header - 39 field names (6 physics + 33 species)species - Array of 33 species names (subset of full chemistry)model_ids - Array of model identifiers
zcrXX_uvYY/pdr - Individual model datasets containing the actual data arrays
v3 Dataset (300K+ models)
header - Full field names (8 physics + species + auxiliaries)species - Full species listmodel_ids - Array of model identifiersmodel_XXXXXX/pdr - Individual model datasets
Proposal for new unified UCLCHEM / UCLPDR / 3D-PDR dataset
...
The dataloader schema
Schema: necessary fields
Fields that are absolutely necessary, without which we cannot train.
- independent variable: time, position, visual extinction (Av).
shape T and float{32,64}
- chemical abundances: the density of each molecule/species at each independent variable point:
shape (T,N) and float{32,64}
Schema: additional fields
Additional fields that maybe used for alternative training modes.
- auxiliary variables: These can be anything that the user wants to provide as an additional input to the machine learning model.
shape (T,P) and float{32,64}
Schema: metadata fields (non-exhaustive)
Metadata that is often used for preprocessing or postprocessing.
- Name
- Description: qualitative description of which system is emulated.
- Physical parameter space covered
- Simulation framework information
- Software version
- Species included in simulation
- Reaction included in simulation with parameters
Open questions
- Should we save these datasets in some intermediate format that is effective for ML training
We need a flexible dataloader framework that by default supports both UCLCHEM and 3D-PDR datasets. The concept is that most expert users can define and simulate their own training dataset, then build a dataloader for it and run an hyperparameter optimization routine to find the best architecture for their specific usage.
The data generated by different astrochemical codes is fundamentally the same, but is organized differently. In the end it all reduces to a set of series with different external (simulation specific), input and output variables. We need to set a clear context for the "users".
xarrayExisting on disk dataset structure and documentation
v1 Dataset (8,192 models)
header- 226 field names (8 physics + 215 species + 3 auxiliary)species- Array of 215 species namesmodel_ids- Array of model identifiers (strings)model_df- Initialization parameters array, shape (n_models, 3): [radfield_init, density_init, zeta_init]model_XXXX/pdr- Individual model datasets containing the actual data arraysv2 Dataset (1,681 models)
header- 39 field names (6 physics + 33 species)species- Array of 33 species names (subset of full chemistry)model_ids- Array of model identifierszcrXX_uvYY/pdr- Individual model datasets containing the actual data arraysv3 Dataset (300K+ models)
header- Full field names (8 physics + species + auxiliaries)species- Full species listmodel_ids- Array of model identifiersmodel_XXXXXX/pdr- Individual model datasetsProposal for new unified UCLCHEM / UCLPDR / 3D-PDR dataset
...
The dataloader schema
Schema: necessary fields
Fields that are absolutely necessary, without which we cannot train.
shape T and float{32,64}shape (T,N) and float{32,64}Schema: additional fields
Additional fields that maybe used for alternative training modes.
shape (T,P) and float{32,64}Schema: metadata fields (non-exhaustive)
Metadata that is often used for preprocessing or postprocessing.
Open questions