-
Notifications
You must be signed in to change notification settings - Fork 3
Add standardized sources and targets definitions across dataset modules #1373
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
amir9667
wants to merge
214
commits into
dev
Choose a base branch
from
features/#1283-fill-dataset-sources-and-targets
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+5,875
−3,376
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…executed The version is set after the execution of all tasks
You successfully defined the sources and targets inside the HeavyDutyTransport class. You updated the download_hgv_data function to use the new class attributes instead of the old config variable. You correctly kept the DATASET_CFG variable at the top of the file, as it's still needed for other parameters and constants. FOR INDIVIDUAL TRAVEL This commit refactors several dataset modules to standardize how input sources and output targets are defined, making the code more modular and easier to maintain. Key changes include: Replaced External Config Calls: Removed direct calls to config.datasets() for accessing source and target paths and tables. This reduces reliance on scattered YAML definitions. Centralized Dependencies in Classes: Defined all data dependencies directly within each dataset's main Python class using the DatasetSources and DatasetTargets objects. Separated Files and Tables: Following team feedback, inputs in DatasetSources were correctly categorized into .files (for CSVs) and .tables (for database tables) for better clarity. Updated Function Logic: All functions were updated to use the new class attributes (e.g., ClassName.sources.files['key']) instead of the old cfg dictionary. Improved Modularity: Updated function calls to pass the entire sources object where necessary, making the functions more independent and reusable. Affected datasets: power_plants substation_extraction chp heavy_duty_transport motorized_individual_travel FOR CHARGING INFRASTRUCTURE Key changes include: Defined sources and targets: Moved all input (URLs, tables) and output (files, tables) definitions into the MITChargingInfrastructure class using DatasetSources and DatasetTargets. This makes the dataset's dependencies explicit and self-contained. Added Dependencies from Docstring: In addition to the tracbev URL, source tables for mv_grid_districts and buildings were added based on the class docstring's Dependencies section. Removed Global Config: Deleted the global DATASET_CFG variable to eliminate reliance on the external YAML configuration for sources and targets. Updated Data Handling: Modified the get_tracbev_data function to use the new class attributes (MITChargingInfrastructure.sources.urls and MITChargingInfrastructure.targets.files) for all data access.
…ps://github.com/openego/eGon-data into features/#1283-fill-dataset-sources-and-targets
This reverts commit 678aa75.
…ps://github.com/openego/eGon-data into features/#1283-fill-dataset-sources-and-targets
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces definition and integration of class-level sources and targets for the affected dataset modules.
references to cfg have been removed and replaced with static, structured access through DatasetSources and DatasetTargets.
Key Changes
🧑💻 Contributor Checklist
Before requesting a review, make sure you've completed all of the following:
Optional:
🔍 Reviewer Checklist
During your review, please check the following:
- [ ] If metadata is pending: Is there an appropriate issue filed?