Skip to content

Ensure cache url and etag values are updated if DB doesn't reflect Azure blob storage#105

Merged
simon-20 merged 4 commits into
developfrom
fix_cache_url_updates
Sep 15, 2025
Merged

Ensure cache url and etag values are updated if DB doesn't reflect Azure blob storage#105
simon-20 merged 4 commits into
developfrom
fix_cache_url_updates

Conversation

@simon-20
Copy link
Copy Markdown
Contributor

@simon-20 simon-20 commented Sep 15, 2025

This PR:

  • Fixes BDS doesn't update cached_dataset columns unless the values change #104, by checking that the Azure ETags for the cached versions of each dataset are the same as in the DB. (Currently, these fields are updated whenever a dataset is re-downloaded, but because these columns are new, and initialised to NULL, that means that they are not currently being set for datasets were successfully downloaded before these columns were added but which haven't been updated since).
    • I have not added a regression test for this issue because the inconsistent state only arose due to these columns being new and initialised to NULL. To properly test would require test code that forces the DB and Azurite into an inconsistent state, which seemed a lot of work for something that is a one-off issue due to the one-time addition of these new columns.
  • Alphabetises the lists of configuration secrets and variables, which makes it easier to ensure all the secrets and vars are set in GitHub Actions.
  • Adds some additional VS Code set ups.

This makes it easier to check that they exist in
the GitHub Actions interface.
This downloads the blob info from Azure and checks
that the ETag matches what is in the database, which
resolves #104.
Copy link
Copy Markdown
Contributor

@chrisarridge chrisarridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good.

@simon-20 simon-20 merged commit bd8d946 into develop Sep 15, 2025
1 check passed
@simon-20 simon-20 deleted the fix_cache_url_updates branch October 29, 2025 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BDS doesn't update cached_dataset columns unless the values change

2 participants