Develop

[EduardSala]

This pull request introduces several improvements to project structure, configuration, documentation, and code reliability. The main highlights include the addition of a Python CI workflow, standardization of data paths across configuration and documentation, enhanced logging for data export operations, and expanded documentation on dataset usage and sources.

Project Infrastructure and Automation:

• Added a GitHub Actions workflow for Python CI to automate linting and testing on pushes and pull requests targeting key branches. (.github/workflows/python-app.yml)
• Introduced a pyproject.toml for standardized Python project configuration and dependency management, specifying required packages and entry points.

Configuration and Documentation Consistency:

• Standardized all data directory paths in configuration files (config/config.yaml) and documentation (README.md) by removing leading ../, ensuring consistency and reducing path errors. [1] [2] [3] [4] [5]
• Updated repository structure documentation to use relative paths without ../ for clarity and correctness.
• Added a prominent note in the README about workflow compatibility with COPERNICUS MARINE SERVICE datasets.

Expanded Dataset Documentation:

• Added a comprehensive data/README.md detailing the selection, characteristics, and access instructions for both in-situ and satellite datasets, including platform lists, images, and links to data sources.

Code Quality and Reliability Enhancements:

• Improved logging in data export functions by replacing print statements with structured logger calls, providing clearer feedback and error reporting during data export operations. (src/io_data/export_data_to_csv.py) [1] [2] [3]
• Added logger and warning suppression for timezone messages when loading data, improving user experience and log clarity. (src/io_data/load_dataframe.py)

Calibration Method Adjustments:

• Updated calibration methods to use the first and last twenty days (instead of ten) for calibration and validation, and improved DataFrame selection logic for better reliability. (src/calibration/calibration_methods.py) [1] [2] [3]
• Changed polynomial fitting degree in quantile mapping correction from 0 to 2 for potentially improved bias correction. (src/calibration/bc_techniques.py)
• Fixed assignment bug in fdm correction by ensuring the corrected values are set in the final DataFrame. (src/calibration/bc_techniques.py)

These changes collectively improve the project's maintainability, reliability, and clarity for both users and contributors.

[Copilot]

Pull request overview

This PR improves project automation and packaging, standardizes configured data paths, expands dataset documentation, and replaces ad-hoc prints with structured logging across the extraction/matching/export workflow.

Changes:

• Added Python CI workflow and introduced pyproject.toml for packaging/installation.
• Standardized data path references in config/config.yaml and README.md, and added extensive dataset documentation under data/.
• Refactored processing/export code to use a shared logger and adjusted calibration/bias-correction behavior.

Reviewed changes

Copilot reviewed 16 out of 22 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
.github/workflows/python-app.yml	Adds GitHub Actions CI to install, lint, and run tests (if present).
.gitignore	Ignores build artifacts and logs/.
README.md	Updates docs (paths, installation section typo fix, workflow note).
config/config.yaml	Standardizes relative paths (removes leading ../).
data/README.md	Adds detailed dataset selection/access documentation.
logs/thesis.log	Adds a generated log file (should not be committed).
pyproject.toml	Adds packaging metadata, dependencies, and a console script entry point.
src/utils/logger_setup.py	Introduces a shared logger writing to logs/thesis.log.
src/load_configuration.py	Switches config loading to logging-based behavior (currently breaks API contract).
src/main.py	Updates config path usage and imports (currently broken due to renamed calibration API).
src/run_extraction.py	Updates config import (still not install-safe with new packaging).
src/io_data/load_dataframe.py	Adds logging + warning suppression; changes error handling (currently returns None on failure).
src/io_data/export_data_to_csv.py	Replaces prints with logger calls; adds basic export logging.
src/processing/processing_df.py	Removes tqdm usage for iteration.
src/processing/spatial_matching.py	Adds logging and empty-DataFrame checks (currently has incorrect return/logging behavior).
src/processing/temporal_matching.py	Adds logging and empty-DataFrame checks (currently contains runtime errors and missing returns).
src/calibration/calibration_methods.py	Renames first-ten-days calibration to first-twenty-days and updates masks (doc mismatch, call sites not updated).
src/calibration/bc_techniques.py	Fixes FDM assignment and increases QM polyfit degree (now risks crashing on small bins).

Comments suppressed due to low confidence (6)

src/calibration/calibration_methods.py:8

• The function was renamed to calib_df_first_twenty_days, but the docstring and inline comment still describe using the “first ten days”. This is misleading documentation and makes it easy for callers to use the wrong split logic; update the docstring/comment to match the 20-day behavior (or rename back if 10 days is intended).

def calib_df_first_twenty_days(df_mooring: pd.DataFrame, df_sat: pd.DataFrame) \
        -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:
    """
    Calibrates the satellite data using the first ten days of mooring data as the calibration set and the remaining
    days as the validation set. The function creates boolean masks based on the day of the month extracted from the

src/load_configuration.py:21

• If the config path does not exist, this logs a warning but continues into path.open(...), which will raise FileNotFoundError (not caught here). Either raise immediately (as before) or return a safe default to avoid an unexpected exception path.
  src/main.py:3
• This module uses top-level imports like import load_configuration / from processing import ..., which only work if src/ is on PYTHONPATH (e.g., running from within src). With the new pyproject.toml packaging, the recommended approach is to import from the installed package namespace (e.g., <package>.load_configuration, <package>.processing) so thesis-run works after pip install ..
  src/main.py:14
• calib_df_first_ten_days is still referenced later in this file, but calibration_methods.py now defines calib_df_first_twenty_days instead. This will break main() at runtime with an AttributeError; update the call site(s) or keep a compatibility alias in calibration_methods.py.
  src/load_configuration.py:25
• On yaml.YAMLError, the function logs but does not raise or return a dict, so it implicitly returns None despite the -> dict annotation. This can cause downstream TypeError/KeyError failures; raise a ValueError (or return {}) after logging to keep the API contract consistent.
  src/run_extraction.py:4
• import load_configuration relies on src/ being on PYTHONPATH and won’t be importable when the project is installed as a package via pyproject.toml (unless load_configuration.py is included as an installed module). Consider moving this into a real package namespace and importing via that namespace so it works both in CI and for end users.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.