Automatically generate inputdata lists for container contents#33
Automatically generate inputdata lists for container contents#33matthewhoffman wants to merge 13 commits into
Conversation
Create a first working generator entrypoint under tools/inputdata with deterministic output writing and strict-mode handling. Add helper modules for workflow parsing, CIME list extraction, and URL normalization so future iterations can refine behavior without changing CLI structure.
Clean up accidental py_compile outputs from the initial generator commit and prevent recurrence by adding standard Python cache ignore patterns.
Replace placeholder standalone manifest entries with static scans of EAMxx scripts/actions for check-input and inputdata-like file paths, then normalize to public URLs and attach source-file provenance.
Run test-all-eamxx in best-effort config mode, scan generated artifacts for inputdata references, and merge with static standalone discovery while preserving deterministic output and warning/report behavior.
Track case directories created per test invocation to reduce cross-test mixing, record start/finish timestamps, and include structured warning fields to make generator failures easier to triage.
Introduce a new GitHub Actions workflow that regenerates inputdata
manifests on a weekly schedule and via manual workflow_dispatch, then
opens a pull request only when tracked manifest artifacts change.
This commit adds:
- .github/workflows/update-inputdata-manifest.yml
- weekly cron + manual trigger
- regenerate step using tools.inputdata.generate_inputdata_manifest
- managed-file diff gate for conditional PR creation
- concurrency guard and job timeout hardening
- date-stamped branch naming for bot-created PRs
- tools/inputdata/build_pr_metadata.py
- reusable helper that builds PR body markdown from git diff stats
and e3sm-workflow-state metadata
- tools/inputdata/README.md
- related-modules entry for the new metadata helper
The existing inputdata image build workflow remains separate and unchanged;
this new workflow handles manifest regeneration and PR creation only.
|
@mahf708 , @rljacob , @bartgol , I tagged you all to review this, but I don't think all of you need to look at it closely. The basic idea is that it automates determining what inputdata needs to be included in the containers used for E3SM CI. It runs weekly and produces a list of all files needed for all tests in the workflows in the E3SM repo. If anything has changed, a PR in this repo is automatically opened to alert the team to the change. It also indicates if required files could not be found on either the e3sm or cesm servers. Once that is reviewed and merged, the updated file lists will trigger an existing workflow to regenerate the containers. This improves on the current state where the inputdata preloaded into the containers may end up missing files (leading to CI downloading potentially large files every night...) or having unnecessary files (which may lead to the container hitting the GH runner size limit). I've tested the file list generation piece in a container locally, and the results make sense to me. I have not tested the new workflow that would trigger a PR (because I didn't want to make a mess in this repo before you all looked at this), but I did lint the file locally. I also have not tested rebuilding the container with the resulting updated file lists to confirm that CI can run without needing to download any more data. I was hoping one of you, or someone else on the IG team, would be willing to do those steps (assuming you all think this is useful), because I'm not terribly familiar with how it all is supposed to work (esp. the standalone EAMxx testing workflow). For the context, this idea was spawned by the CI failure we are seeing in the NGEE Arctic project, and the realization that we were downloading a lot of unnecessary data. This felt like the right upstream fix to solve our problem, and I hope E3SM Project finds this contribution useful. If not, that's ok too, we can always manage this on our own. And, as is probably obvious, this was all written by copilot with guidance from me. |
mahf708
left a comment
mahf708
left a comment
There was a problem hiding this comment.
I reviewed this and I approve. I will wait a little before merging in case @rljacob or @bartgol have comments.
Two comments regarding the next steps (independent of this):
- we want to only generate the missing data for runners in the cloud (we also have runners that do stuff locally at SNL computers, for and for those we don't need to worry too much about data)
- one issue we may run into is... we will run out of space inside these containers :( some of the files needed/added for tests are way too huge and with gcam and other problems in E3SM repo, the repo itself is huge now... which means we are increasingly out of space... cloud-provided runners usually have little space in total
|
This is a lot of stuff. I trust Naser review, as he is more knowledgeable than me regarding cloud/container testing. |
There was a problem hiding this comment.
Pull request overview
This PR introduces an automated inputdata manifest generator (driven by E3SM
workflow definitions) and wires it into a scheduled GitHub Actions workflow so
inputdata/ manifests can be refreshed and proposed via bot-opened PRs without
manual curation.
Changes:
- Added a Python-based manifest generator that parses E3SM workflows, runs
setup-only CIMEcreate_test, and emits stable manifests plus provenance and
missing-file outputs. - Added URL normalization + remote existence probing against E3SM/CESM public
inputdata endpoints, with missing/unverified tracking. - Added a weekly
update-inputdata-manifestworkflow that regenerates outputs
and opens a PR when the manifest trigger files change.
Reviewed changes
Copilot reviewed 10 out of 14 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/inputdata/README.md | Documents generator purpose, inputs/outputs, and behavior. |
| tools/inputdata/generate_inputdata_manifest.py | Implements end-to-end discovery pipeline, manifest writing, provenance/missing reporting. |
| tools/inputdata/build_pr_metadata.py | Generates PR body content for bot-opened manifest update PRs. |
| tools/inputdata/lib/workflow_parser.py | Extracts CI-relevant tests/standalone refs from E3SM workflow YAML. |
| tools/inputdata/lib/cime_manifest.py | Runs setup-only create_test and parses Buildconf input list files. |
| tools/inputdata/lib/url_normalize.py | Filters/normalizes candidate paths and constructs public inputdata URLs. |
| tools/inputdata/lib/inputdata_validation.py | Probes candidate URLs (HEAD) and classifies resolution/missing/unverified. |
| tools/inputdata/init.py | Declares the tools.inputdata package. |
| tools/inputdata/lib/init.py | Declares the tools.inputdata.lib package. |
| .github/workflows/update-inputdata-manifest.yml | Adds scheduled/manual automation to regenerate manifests and open PRs. |
| inputdata/files-missing.txt | Adds managed output file for missing-inputdata tracking. |
| README.md | Adds repository-level documentation for the workflows and automation. |
| .gitignore | Ignores Python bytecode/cache artifacts. |
| readme | Removes legacy/duplicate top-level readme stub. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| `tools/inputdata` contains the generator that builds the list of inputdata files required for Github | ||
| Continuous Integration tests. The purpose is to produce the files that should be included in the CI | ||
| Docker container, avoiding the need for them to be downloaded during CI. |
| blocked_fragments = ( | ||
| "baseline", | ||
| "rest", | ||
| "restart", | ||
| "history", | ||
| "output", | ||
| "archive", | ||
| ) |
| value = path_value.strip().strip('"').strip("'") | ||
| value = value.replace("\\", "/") |
| method="HEAD", | ||
| headers={"User-Agent": "Mozilla/5.0", "Accept-Encoding": "identity"}, | ||
| ) | ||
| context = ssl._create_unverified_context() |
| return InputDataValidationResult( | ||
| relative_path=relative_path, | ||
| checked_urls=(e3sm_url, cesm_url), | ||
| resolved_url=e3sm_url, |
| parser.add_argument( | ||
| "--strict", | ||
| action="store_true", | ||
| help="Fail if workflow parsing or standalone discovery emits warnings.", |
| provenance_record = ProvenanceRecord( | ||
| workflow=test_record.workflow, | ||
| test=test_record.name, | ||
| case_dir=test_record.name, |
| for match in path_pattern.finditer(line): | ||
| raw = match.group("path") | ||
| candidates.add((file_path.as_posix(), raw)) |
| option_b_candidates = _scan_option_b_artifacts(e3sm_root, output_root) if option_b_ok else [] | ||
| all_candidates = static_candidates + option_b_candidates | ||
| if not all_candidates: | ||
| return [], {} | ||
|
|
||
| for source_path, raw_path in all_candidates: | ||
| validation = _normalize_and_validate(raw_path, din_loc_root) | ||
| if not validation: | ||
| continue | ||
| method = "standalone-static" | ||
| if source_path.startswith(output_root.as_posix()) or "/ctest-build/" in source_path: | ||
| method = "standalone-config" |
Summary
This PR adds an end-to-end inputdata manifest generator and wires it into
automation so CI inputdata images can be updated from E3SM workflow definitions
without manual file curation.
What Changed
references from E3SM workflow configuration.
references, including compatibility fixes and improved case attribution /
failure diagnostics.
artifact scanning.
endpoints (e3sm & cesm), with explicit handling of unresolved files.
diffs are visible when they appear.
workflow integration.
Resulting Behavior
full-model files, standalone files, missing files, and provenance/state.
metadata-only churn.
TLDR
CI will run weekly to:
files in .github/workflows
for committing the updated lists to main
regenerate updated containers.