Add download mirror functionality#362
Draft
ahresse wants to merge 3 commits into
Draft
Conversation
Add -m/--mirror option to the download subcommand that rewrites source URLs to point to a local HTTP mirror, with automatic fallback to the original internet source on failure. The mirror URL is constructed by stripping the scheme from the original URL and prepending the mirror base URL, e.g.: - https://storage.googleapis.com/aom-test-data/file.ivf - http://mirror.local:8080/storage.googleapis.com/aom-test-data/file.ivf Changes: - Add rewrite_url() and MIRROR_NETWORK_ERRORS to utils.py - Add mirror parameter with fallback logic to utils.download() - Add -m/--mirror CLI argument to the download subcommand - Propagate mirror through fluster.py and test_suite.py call chain - Add unit and integration tests for mirror URL rewriting and download Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Add scripts/mirror_sync.py which scans test suite JSON files for source URLs and downloads them into a directory tree suitable for serving as a local mirror (e.g. via nginx or python -m http.server). Supports parallel downloads and configurable retries. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
Document the --mirror download option and mirror_sync.py script, including how the mirror URL rewriting works, usage examples, and instructions for setting up a mirror server. Signed-off-by: Alexandre Esse <alexandre.esse@canonical.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds support for downloading test suite resources from a local HTTP mirror, improving speed and efficiency in CI or multi-machine environments. It introduces a new
--mirroroption to the download command, updates the documentation, and adds a script to help set up and sync a local mirror directory. The most important changes are:Local Mirror Support:
Added a
--mirroroption to thefluster.py downloadcommand, allowing users to specify a base URL for a local mirror. Fluster will try downloading from the mirror first and fall back to the original source if the mirror fails. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]Implemented mirror URL rewriting and robust error handling in the download utility, ensuring proper fallback to the original source if the mirror is unavailable or returns an error. [1] [2]
Documentation Updates:
README.mdto document the new--mirroroption, provide usage examples, and explain how to set up and use a local mirror for test suite resources. [1] [2] [3] [4]Mirror Sync Script:
scripts/mirror_sync.py, which scans all test suite JSON files and downloads every referenced resource into a local directory tree, making it easy to set up a mirror server. The script supports parallel downloads and skipping already-downloaded files.These changes make it much easier and faster to run fluster in environments where repeated downloads from the internet are undesirable or slow.