-
Notifications
You must be signed in to change notification settings - Fork 3
Description
Clean Up create_empty_ts.smk for Unused Components
Description
The create_empty_ts.smk rule currently writes empty_ts for all components that require foreign_key='efficiency', even if those components are not listed in model_structure["components"]. While this behavior does not break functionality, it is not clean and can be misleading.
To improve clarity and maintainability, this behavior should be revised.
Steps to Take
- Filter
foreign_keys_efficiencyto exclude components not listed inmodel_structure["components"], following the approach used forforeign_keys_profile:# Drop the ones that are not listed in model_structure foreign_keys_efficiency = [ fk for fk in foreign_keys_efficiency if fk["efficiency"].split("-efficiency")[0] in model_structure["components"] ]
- Ensure that
empty_ts_load,empty_ts_feedin, andempty_ts_efficiencycan gracefully handle being empty dataframes when applicable.
Error Message Log
WARNING - Waiting at most 5 seconds for missing files.
MissingOutputException in rule create_empty_ts in file [...]/apipe/esys/snakemake_rules/create_empty_ts.smk, line 1:
Job 0 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
store/datasets/esys_raw/data/time_series/empty_ts_efficiencies.csv
Removing output files of failed job create_empty_ts since they might be corrupted:
store/datasets/esys_raw/data/time_series/empty_ts_load.csv, store/datasets/esys_raw/data/time_series/empty_ts_feedin.csv
WARNING - Removing output files of failed job create_empty_ts since they might be corrupted:
store/datasets/esys_raw/data/time_series/empty_ts_load.csv, store/datasets/esys_raw/data/time_series/empty_ts_feedin.csv
Shutting down, this might take some time.
WARNING - Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-11-28T134931.620653.snakemake.log
WARNING - Complete log: .snakemake/log/2024-11-28T134931.620653.snakemake.log
Acceptance Criteria
create_empty_ts.smkno longer writesempty_tsfiles for components not listed inmodel_structure["components"].empty_ts_load,empty_ts_feedin, andempty_ts_efficiencycan remain valid but empty when appropriate.- The pipeline runs cleanly without warnings or unnecessary outputs.