Skip to content

adds cbrfc_wsv_seasonal data to local warehouse for crmms test locations - #68

Open
samland1116 wants to merge 2 commits into
mainfrom
65-add-cbrfc-wsv-seasonal-forecast-to-warehouse
Open

adds cbrfc_wsv_seasonal data to local warehouse for crmms test locations#68
samland1116 wants to merge 2 commits into
mainfrom
65-add-cbrfc-wsv-seasonal-forecast-to-warehouse

Conversation

@samland1116

Copy link
Copy Markdown
Collaborator
  • Restructures the warehouse/local dir to include a utils folder to store the setup_utils.py and any other ingest specific utils. These utils will move to a prefect workflow but are stored here temporarily to avoid blocking progress on frontend-development

  • CBRFC Accumulated WSV Seasonal Test Data:

    • adds notebook setup for CBRFC water supply volume seasonal forecasts for three CRMMS locations where we had a crosswalk.
    • the CBRFC WSV Seasonal data has observed volume accumulations in the data file -- added those as a primary_timeseries.
    • Included a QC plotting function in the notebook for direct comparison with the CBRFC seasonal plots, feel free to strip it out.

Will transition into prefect workflow as opposed to this scratch-work implementation once we have the full locations list

Closes #65

@samland1116 samland1116 self-assigned this Aug 20, 2026
@samland1116 samland1116 added the enhancement New feature or request label Aug 20, 2026
@samland1116 samland1116 linked an issue Aug 20, 2026 that may be closed by this pull request
@mgdenno

mgdenno commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Note: we should merge main into these branches as a lot has changed in a short time.

Some thoughts:

Notebook:

  • Given the scale of the data that we will have in the FVED I'm not sure we need to differentiate between local and remote for the warehouse notebooks at this time (might have jumped the gun on that).
  • Recent changes make the create_minio_spark_session() unnecessary. Use the TEEHR create_spark_session()
  • I don't think we are using ev.download. Can probably be removed.
  • Overall we want to separate the loading (adding domain data, locations, etc) from fetching data code. I'd think even at this stage we would want to be thinking about having that stuff broken out.
  • crosswalks and location ids - are the CBRFC IDs NWS IDs? wondering if cbrfc- is the right prefix of if it should be nws- or something else.
  • On the variables, in TEEHR we have to-date used a variable_interval_statistic pattern. I'm not super familiar with this data but we want to follow that as closely as we can unless there is a strong reason to deviate. It seems like it might be closer to e.g., volume_seasonal_p30 or something. we should have some discussions around this as I realize this is a new type of evaluation but i'd rather not break the norms unless we need to. also are we ingesting the ensemble members? Are wsv_esp and wsv_official more like configurations that variables? I think the big things is that when joining primary and secondary the variables must be the same. Is this the case?
  • If we want to exploratory notebooks (plots), I'd separate those out from loading, fetching code. Can be kind of annoying but make running the wrong code less likely.

Utils:

  • Schemas are a property of the TEEHR tables. redefining seems unnecessary.
  • I didn't get into the mechanics of fetching and parsing the data from CBRFC.
  • Agree this should be in Prefect but fine to test in a notebook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add CBRFC WSV seasonal forecast to warehouse

2 participants