Release/1.9.0#370
Open
jamesfwood wants to merge 36 commits into
Open
Conversation
Updated link to PODAAC forum
# Conflicts: # pyproject.toml
* Issue 346 and 288 * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Improved test_version.py --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* update aws provider to 5.0 * update changelog * removed lifecycle for iam policy * update aws provider to 6.0 * fix attachment of policys * poetry update * update so test runs later * update so sonar cloud after tests * ignore snyk for now as aiohttp conflicts with vcrpy * update required provider syntax
* Issue 308 - Add SOS Hydrology V3 products to SWOT_L2_HR_RiverSP_2.0 collection * Fix tests * - Removed sos_source_filename and sos_ingest_time from db write - Added README.md to sos_ingest * poetry update * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Updated to import os * Changed aws_profile to a required input arg with no default value * Need to set AWS REGION for Test and coverage * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Cleanup from code review * - Added limit arg - Added dry_run log - Fixed lakeflow date matching bug - Removed some unused imports --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prepares the 1.9.0 release by extending Hydrocron’s support for SWOT v2.0 (incl. SOS Hydrology V3-related fields), fixing version parsing/population (SWORD + collection_version), and introducing an SOS discharge ingest + verification tool with accompanying tests and CI updates.
Changes:
- Fix SWORD version parsing from shp.xml and populate
collection_versionfrom granule paths/filenames. - Add SOS discharge fields/alias handling and API fill-value behavior for missing DynamoDB attributes, plus validation preventing SOS fields on vD Reach requests.
- Introduce a new
SOS/ingest/scan CLI tool with extensive unit/integration tests and update CI to run those tests.
Reviewed changes
Copilot reviewed 39 out of 41 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_track_ingest.py | Adds tests for SWORD version parsing and collection_version extraction. |
| tests/test_io_swot_reach_node_shp.py | Updates expectations to require collection_version == "2.0". |
| tests/test_api.py | Adds tests for SOS field validation, fill values, alias behavior, and error message when collection_name defaults to vD. |
| tests/regression/test_version.py | Adds regression coverage for collection_version and sword_version in API responses. |
| tests/regression/README.md | Adds a regression-test guideline bullet. |
| SOS/tests/test_ui.py | Tests Rich UI output for the SOS ingest tool. |
| SOS/tests/test_time_matcher.py | Tests time matching behavior/tolerance/ambiguity detection. |
| SOS/tests/test_scanner.py | Tests post-ingest verification scan behavior and CSV output. |
| SOS/tests/test_reader.py | Tests NetCDF SOS reader extraction/filtering/time handling. |
| SOS/tests/test_processor.py | Integration tests for ingest orchestration (dry-run/live/resume/interrupt/etc.). |
| SOS/tests/test_error_logger.py | Tests error CSV writer and summary report output. |
| SOS/tests/test_db_client.py | Tests DynamoDB client query/update behaviors and retry config. |
| SOS/tests/test_cli.py | Tests CLI argument parsing into config. |
| SOS/tests/conftest.py | Adds fixtures (mini NetCDF generator + moto DynamoDB fixtures). |
| SOS/tests/init.py | Marks SOS tests as a package. |
| SOS/sos_ingest/ui.py | Implements Rich UI panels/progress bars for ingest/scan. |
| SOS/sos_ingest/time_matcher.py | Implements closest-time matching with ambiguity detection. |
| SOS/sos_ingest/scanner.py | Implements read-only scan mode comparing SOS file vs DynamoDB. |
| SOS/sos_ingest/README.md | Documents SOS ingest tool usage and behavior. |
| SOS/sos_ingest/reader.py | Implements SOS NetCDF reading and discharge record extraction. |
| SOS/sos_ingest/processor.py | Implements ingest orchestration, update loop, and summary logging. |
| SOS/sos_ingest/models.py | Adds dataclasses for algorithm defs, records, and summaries. |
| SOS/sos_ingest/helpers.py | Adds shared DB time parsing + row lookup helpers. |
| SOS/sos_ingest/error_logger.py | Writes per-timestep error CSV + summary report. |
| SOS/sos_ingest/db_client.py | DynamoDB query/update wrapper for ingest tool. |
| SOS/sos_ingest/config.py | Defines ingest runtime configuration dataclass. |
| SOS/sos_ingest/cli.py | Adds CLI entrypoint and optional dependency error messaging. |
| SOS/sos_ingest/init.py | Marks SOS ingest module as a package. |
| SOS/dev-utils/query_reach.py | Adds operator utility to inspect reach rows/columns. |
| SOS/dev-utils/list_reaches.py | Adds operator utility to list reach_ids in a range. |
| SOS/dev-utils/find_columns.py | Adds operator utility to find rows containing column prefixes. |
| SOS/dev-utils/count_rows.py | Adds operator utility to count table rows. |
| SOS/dev-utils/count_fill_rows.py | Adds operator utility to quantify fill/sentinel rows/anomalous RSTs. |
| README.md | Updates documentation and forum links. |
| pyproject.toml | Bumps version/deps, adds optional sos group deps and sos_ingest script. |
| hydrocron/utils/constants.py | Adds fill value constant, SOS field lists, aliases, and updates reach columns. |
| hydrocron/db/io/swot_shp.py | Fixes SWORD version parsing and populates collection_version from collection name. |
| hydrocron/api/controllers/timeseries.py | Adds vD SOS-field validation and fills missing/alias columns in outputs. |
| CHANGELOG.md | Adds 1.9.0 changelog entry. |
| .github/workflows/build.yml | Updates CI to install SOS deps and run SOS tests; alters security scanning steps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
66
to
+69
| [tool.poetry.scripts] | ||
| hydrocron_api = "hydrocron.api.__main__:__main__" | ||
| hydrocron_load = 'hydrocron.db.load_data:main' | ||
| sos_ingest = "SOS.sos_ingest.cli:main" |
Comment on lines
1
to
4
| [tool.poetry] | ||
| name = "hydrocron" | ||
| version = "1.8.0" | ||
| version = "1.9.0a8" | ||
| description = "OpenAPI access to Time Series data for SWOT features" |
Comment on lines
166
to
170
| - name: Run Snyk as a blocking step | ||
| if: false | ||
| uses: snyk/actions/python-3.12@master | ||
| env: | ||
| SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[1.9.0]
Added
Changed
Deprecated
Removed
Fixed
Security