Development#5
Closed
EduardSala wants to merge 38 commits into
Closed
Conversation
Updated review branch
Reorganize repository by moving spatial-temporal-matching and calibration scripts into a centralized modules/ package (calibration.py, spatial_temporal_matching.py, test.py, .gitignore). Delete now-redundant README files and the old spatial-temporal-matching extractor. Add new modules/extract_insitu_data_to_csv.py which provides functions to extract in-situ mooring data from NetCDF into pandas DataFrames/CSV (extr_insitu_data and export_dataframe_to_file), using xarray, pandas, numpy and Module_all_functions utilities.
Add a new config/config.yaml with extraction and spatio-temporal matching settings. Rename modules/extract_insitu_data_to_csv.py -> modules/data_extraction_md.py and clean up the extraction function (remove unused imports, adjust return/print behavior and CSV write paths). Add a new stub modules/spatial_crossover_md.py with a spatial_cross signature. Update modules/.gitignore to ignore /test.py/ and simplify modules/test.py to list satellite CSV files and attempt a read (replace previous moorings NC-to-CSV conversion script).
Deleted datasets files. Updated the repo structure and made it more clean and readable. Added a dev folder where I test my functions. I added it to gitignore.
Reorganize codebase: move modules into a src/ package (src/config, src/io, src/processing, src/calibration), relocate legacy modules to legacy/, and add new processing and calibration modules. Added src/processing/{processing_df,spatial_matching,temporal_matching} to implement the spatio-temporal matching pipeline and alignment, and src/calibration/{calibration_methods,bc_techniques,metrics} for calibration workflows and metrics. Updated imports and scripts/main.py to use the new module paths and run matching -> calibration -> metrics. Renamed/export function (export_dataframe_to_file) and adjusted load_dataframe signatures/DateTime handling. Updated config files (.gitignore and config_local.yaml) to include local paths and crossMatching settings. Removed the old modules/spatial_matching.py and included some __pycache__ files.
Standardize package structure and naming: replace legacy 'modules' imports with package imports (config, processing, calibration, io_data), move io modules into src/io_data, and add corresponding __pycache__ files. Rename functions and identifiers to snake_case (e.g. linearCal -> linear_cal, BIAS/RMSE/CC/SI -> bias/rmse/cc/si, load_satData_csv -> load_sat_data_csv, load_moorData_nc -> load_moor_data_nc) and update all callers. Clean up formatting/whitespace, minor API adjustments (load_moor_data_nc signature/tolerance handling), and small logic/variable name fixes in spatial/temporal matching and metrics (haversine radius var, timing/tolerance computation, index naming). Overall aim: improve consistency, readability, and module organization while preserving existing behavior where possible.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR modularizes the spatio-temporal matching and calibration workflow for satellite vs. mooring datasets, introducing config loading, IO helpers, matching modules, and calibration/metrics utilities.
Changes:
- Added new spatio-temporal matching modules (
spatial_matching,temporal_matching) and an orchestration layer (processing_df). - Added IO utilities for loading mooring/satellite data and exporting extracted data to CSV.
- Added configuration loader + YAML configs, calibration utilities, and evaluation metrics, plus runnable scripts.
Reviewed changes
Copilot reviewed 16 out of 33 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| src/processing/spatial_matching.py | New spatial matching (haversine distance + min-distance co-location). |
| src/processing/temporal_matching.py | New temporal matching (time window + closest measurement). |
| src/processing/processing_df.py | Orchestrates spatio-temporal matching across mooring NetCDF files. |
| src/io_data/load_dataframe.py | Loads satellite CSV and mooring NetCDF into DataFrames. |
| src/io_data/export_data_to_csv.py | Exports extracted in-situ DataFrames to CSV. |
| src/config/load_configuration.py | YAML config loader utility. |
| src/calibration/calibration_methods.py | Calibration/validation dataset splitting helpers. |
| src/calibration/bc_techniques.py | Linear bias correction helper. |
| src/calibration/metrics.py | Bias/RMSE/SI/CC metric functions and metrics array output. |
| scripts/run_extraction.py | Script to extract mooring NetCDF data to CSV using config. |
| scripts/main.py | Main script orchestrating matching + calibration + metrics. |
| config/config.yaml | Default YAML configuration scaffold. |
| config/config_local.yaml | Local/developer configuration (currently committed). |
| config/.gitignore | Intended ignore for local config (pattern issue). |
| .gitignore | Updated ignore list (missing Python bytecode ignores). |
| legacy/spatial_temporal_matching.py | Legacy script updated (import path currently broken). |
| legacy/functions_cluster.py | Added large legacy/analysis script. |
| legacy/calibration.py | Added large legacy calibration/plotting script. |
| legacy/Module_all_functions.py | Minor fixes in legacy helper module. |
| src/processing/pycache/temporal_matching.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/processing/pycache/spatial_matching.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/processing/pycache/processing_df.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/io_data/pycache/load_dataframe.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/config/pycache/load_configuration.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/calibration/pycache/metrics.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/calibration/pycache/calibration_methods.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| src/calibration/pycache/bc_techniques.cpython-311.pyc | Committed bytecode artifact (should not be in VCS). |
| docs/methodology.md | Empty placeholder file. |
| docs/datasets.md | Empty placeholder file. |
| data/README.md | Empty placeholder file. |
| requirements.txt | Empty placeholder file. |
Comments suppressed due to low confidence (1)
legacy/spatial_temporal_matching.py:12
modulesdoes not exist in this repository, sofrom modules import Module_all_functions as fltwill fail at runtime. If the intention is to reference the legacy module in this repo, update the import to point at the correct path (e.g.,legacy.Module_all_functions) or remove this legacy script from the runnable code path.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+4
to
+15
| def load_config(config_path: str | Path) -> dict: | ||
|
|
||
| path = Path(config_path) | ||
|
|
||
| if not path.exists(): | ||
| raise FileNotFoundError(f"Config file not found: {path}") | ||
|
|
||
| try: | ||
| with path.open("r", encoding="utf-8") as f: | ||
| return yaml.safe_load(f) or {} | ||
| except yaml.YAMLError as e: | ||
| raise ValueError(f"Invalid YAML in config file: {path}") from e |
There was a problem hiding this comment.
The PR description mentions the configuration loader “read and validate” configs, but load_config currently only loads YAML without validating required keys/types. Either add minimal validation (e.g., required sections/fields) or adjust the PR description to match the implementation.
Comment on lines
+9
to
+12
| dir_input_mooring_nc: C:\Users\eduar\Desktop\materiale Tesi\moorings-nc | ||
| dir_output_mooring_csv: C:\Users\eduar\Desktop\materiale Tesi\moorings-csv | ||
| dir_input_sat_csv: C:\Users\eduar\Desktop\materiale Tesi\satellite-csv | ||
| path_sat_singleFile: C:\Users\eduar\Desktop\materiale Tesi\satellite-csv\ds_S6A_wave.csv |
There was a problem hiding this comment.
This file contains developer-specific absolute paths and appears to be intended as a local-only override, but it is currently committed. Remove it from version control and commit a config_local.yaml.example/template instead (and ensure it’s ignored by git).
Suggested change
| dir_input_mooring_nc: C:\Users\eduar\Desktop\materiale Tesi\moorings-nc | |
| dir_output_mooring_csv: C:\Users\eduar\Desktop\materiale Tesi\moorings-csv | |
| dir_input_sat_csv: C:\Users\eduar\Desktop\materiale Tesi\satellite-csv | |
| path_sat_singleFile: C:\Users\eduar\Desktop\materiale Tesi\satellite-csv\ds_S6A_wave.csv | |
| dir_input_mooring_nc: ./data/moorings-nc | |
| dir_output_mooring_csv: ./data/moorings-csv | |
| dir_input_sat_csv: ./data/satellite-csv | |
| path_sat_singleFile: ./data/satellite-csv/ds_S6A_wave.csv |
Owner
Author
There was a problem hiding this comment.
@copilot My idea is to give the possibility to modify config file with own directory path. These are just for demonstration. How should I operate
Comment on lines
+12
to
+13
| tol_time = np.timedelta64(cross_time_val, cross_time_unit) | ||
| rows, col = np.where(diff_time <= tol_time) |
There was a problem hiding this comment.
np.timedelta64(cross_time_val, cross_time_unit) does not accept floats for the value in many NumPy versions; passing a non-integer cross_time_val can raise TypeError. Use an integer number of units or switch to pandas.Timedelta (which supports floats) and compare with that.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
|
@EduardSala I've opened a new pull request, #6, to work on those changes. Once the pull request is ready, I'll request review from you. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…to development
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Regenerate compiled Python bytecode for the config, io_data, and processing modules (load_configuration, load_dataframe, processing_df). Only .pyc files in __pycache__ were updated—no source (.py) changes included; likely produced by interpreter/environment rebuild.
…to development
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several significant updates to the codebase, focusing on modularizing the pipeline for extracting, matching, calibrating, and evaluating satellite and mooring datasets. Major improvements include the addition of configuration management, new data processing and calibration modules, and scripts to orchestrate extraction and processing workflows. The changes also refactor and reorganize legacy code for clarity and maintainability.
Configuration Management:
config/config.yaml,config/config_local.yaml) for parameterizing extraction and matching processes, supporting both default and local/developer-specific settings. ([[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-38df760413887d0cc4ee5707919e6390456a863d326f6080644cc3d9c72e4bbaR1-R27),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-735ce84dbd3d89aceb46efd84399bcd9cb787f5d2fa976a0c90e6479b9b1ed01R1-R20))src/config/load_configuration.pyto read and validate YAML config files. ([src/config/load_configuration.pyR1-R16](https://github.com/EduardSala/Thesis/pull/5/files#diff-e5ab3bf32b66e0b23ec00d7fc4fb52bac0bdbcc3ac67404def76293b77b443d8R1-R16)).gitignoreto exclude local config files from version control. ([config/.gitignoreR1](https://github.com/EduardSala/Thesis/pull/5/files#diff-aa621cb41a08746a13e2864c0e6803f78ddcb754f06e30afd74a8295159890c9R1))Data Extraction and Processing Pipeline:
scripts/run_extraction.pyto automate extraction of in-situ data from NetCDF files to CSV, parameterized by config. ([scripts/run_extraction.pyR1-R30](https://github.com/EduardSala/Thesis/pull/5/files#diff-30b79f53e125c9e75984ec026172a34b24ccdcb0db1a1f049807d72042d6c70aR1-R30))src/io_data/export_data_to_csv.pyandsrc/io_data/load_dataframe.pyto modularize data loading and exporting for both satellite and mooring sources. ([[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-2889ad48ec616e1731e873f823ff6e0bb9b00f83251e6b5e5dd6085d31d86bceR1-R14),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-626adb3810f4df24bcc2abc23059f4faccbc42b25634981aa7fb864ef475658fR1-R56))src/processing/processing_df.pyfor spatio-temporal matching and alignment of satellite and mooring data, including handling of missing values. ([src/processing/processing_df.pyR1-R60](https://github.com/EduardSala/Thesis/pull/5/files#diff-5ad525d0e0f7bc831f3fc339ab0256fb62eac74cb6569c1824833ac8a0e37950R1-R60))Calibration and Evaluation:
src/calibration/calibration_methods.pyfor splitting datasets into calibration/validation sets, andsrc/calibration/bc_techniques.pyfor linear bias correction. ([[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-6cc944761c0c19bb95270dc0256f3fa56d1fd762cf52f9add2e6189c6b7ce26dR1-R23),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-3478a63642bc5c3a7f7ab80a6ea083d5acb173523cacabda25f7345f73f898b3R1-R18))src/calibration/metrics.pywith standard evaluation metrics (bias, RMSE, scatter index, correlation coefficient) and a metrics array function for summary statistics. ([src/calibration/metrics.pyR1-R43](https://github.com/EduardSala/Thesis/pull/5/files#diff-6deb5713c1dfa8ee86dfc6fb3f16e3304c265f3f61c225f50356f568c6ca5b16R1-R43))Workflow Orchestration:
scripts/main.pyas the main entry point to load configuration, perform spatio-temporal matching, apply calibration, and print evaluation metrics. ([scripts/main.pyR1-R28](https://github.com/EduardSala/Thesis/pull/5/files#diff-c4c324d443977ebacc39c903e2b7e3e4d30f95f9da67fb76469053bbada71549R1-R28))Refactoring and Legacy Code Updates:
[[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-950ebbbe875b88e21c7520cf420c252dfdefb6d20de1b3f5dfc60b74b4d2b640L35-R35),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-950ebbbe875b88e21c7520cf420c252dfdefb6d20de1b3f5dfc60b74b4d2b640L77-R77),[[3]](https://github.com/EduardSala/Thesis/pull/5/files#diff-caa120929261afbfbb9b738297e61ae4c4cfa1b1189c1a057104fcc147866982L6-R19))Configuration and Setup:
[[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-38df760413887d0cc4ee5707919e6390456a863d326f6080644cc3d9c72e4bbaR1-R27),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-735ce84dbd3d89aceb46efd84399bcd9cb787f5d2fa976a0c90e6479b9b1ed01R1-R20),[[3]](https://github.com/EduardSala/Thesis/pull/5/files#diff-e5ab3bf32b66e0b23ec00d7fc4fb52bac0bdbcc3ac67404def76293b77b443d8R1-R16),[[4]](https://github.com/EduardSala/Thesis/pull/5/files#diff-aa621cb41a08746a13e2864c0e6803f78ddcb754f06e30afd74a8295159890c9R1))Data Extraction and Matching:
[[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-30b79f53e125c9e75984ec026172a34b24ccdcb0db1a1f049807d72042d6c70aR1-R30),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-2889ad48ec616e1731e873f823ff6e0bb9b00f83251e6b5e5dd6085d31d86bceR1-R14),[[3]](https://github.com/EduardSala/Thesis/pull/5/files#diff-626adb3810f4df24bcc2abc23059f4faccbc42b25634981aa7fb864ef475658fR1-R56),[[4]](https://github.com/EduardSala/Thesis/pull/5/files#diff-5ad525d0e0f7bc831f3fc339ab0256fb62eac74cb6569c1824833ac8a0e37950R1-R60))Calibration and Evaluation:
[[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-6cc944761c0c19bb95270dc0256f3fa56d1fd762cf52f9add2e6189c6b7ce26dR1-R23),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-3478a63642bc5c3a7f7ab80a6ea083d5acb173523cacabda25f7345f73f898b3R1-R18),[[3]](https://github.com/EduardSala/Thesis/pull/5/files#diff-6deb5713c1dfa8ee86dfc6fb3f16e3304c265f3f61c225f50356f568c6ca5b16R1-R43))Workflow Orchestration:
[scripts/main.pyR1-R28](https://github.com/EduardSala/Thesis/pull/5/files#diff-c4c324d443977ebacc39c903e2b7e3e4d30f95f9da67fb76469053bbada71549R1-R28))Codebase Refactoring:
[[1]](https://github.com/EduardSala/Thesis/pull/5/files#diff-950ebbbe875b88e21c7520cf420c252dfdefb6d20de1b3f5dfc60b74b4d2b640L35-R35),[[2]](https://github.com/EduardSala/Thesis/pull/5/files#diff-950ebbbe875b88e21c7520cf420c252dfdefb6d20de1b3f5dfc60b74b4d2b640L77-R77),[[3]](https://github.com/EduardSala/Thesis/pull/5/files#diff-caa120929261afbfbb9b738297e61ae4c4cfa1b1189c1a057104fcc147866982L6-R19))