Automatically generate inputdata lists for container contents

.github/workflows/update-inputdata-manifest.yml

@@ -0,0 +1,111 @@
+name: update-inputdata-manifest
+
+on:
+  schedule:
+    - cron: "0 12 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+concurrency:
+  group: update-inputdata-manifest
+  cancel-in-progress: false
+
+jobs:
+  regenerate:
+    runs-on: ubuntu-22.04
+    timeout-minutes: 240
+    container:
+      image: ghcr.io/e3sm-project/containers-ghci:main
+    steps:
+      - name: Checkout containers
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+          show-progress: false
+          persist-credentials: false
+
+      - name: Checkout E3SM
+        uses: actions/checkout@v4
+        with:
+          repository: E3SM-Project/E3SM
+          path: E3SM
+          ref: master
+          submodules: recursive
+          show-progress: false
+          persist-credentials: false
+
+      - name: Regenerate inputdata manifests
+        env:
+          NETCDF_C_ROOT: /usr
+          NETCDF_FORTRAN_ROOT: /usr
+          PARALLEL_NETCDF_ROOT: /usr
+        run: |
+          set -euo pipefail
+          mkdir -p /tmp/e3sm-inputdata-cases /tmp/e3sm-inputdata-output
+          python3 -m tools.inputdata.generate_inputdata_manifest \
+            --e3sm-root E3SM \
+            --test-root /tmp/e3sm-inputdata-cases \
+            --output-root /tmp/e3sm-inputdata-output \
+            --state-out inputdata/e3sm-workflow-state.json \
+            --files-out inputdata/files.txt \
+            --standalone-files-out inputdata/files-standalone.txt \
+            --missing-files-out inputdata/files-missing.txt \
+            --provenance-out inputdata/provenance.json \
+            --verbose
+
+      - name: Detect manifest changes
+        id: detect
+        run: |
+          set -euo pipefail
+          files=(
+            inputdata/files.txt
+            inputdata/files-standalone.txt
+            inputdata/files-missing.txt
+          )
+
+          if git diff --quiet -- "${files[@]}"; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Build PR metadata
+        if: steps.detect.outputs.changed == 'true'
+        id: metadata
+        run: |
+          set -euo pipefail
+
+          stamp="$(date -u +%Y%m%d)"
+          branch="bot/update-inputdata-manifest-${stamp}-${GITHUB_RUN_ID}"
+          echo "branch=${branch}" >> "$GITHUB_OUTPUT"
+
+          summary_path="/tmp/inputdata-pr-summary.md"
+
+          python3 -m tools.inputdata.build_pr_metadata \
+            --output-path "${summary_path}" \
+            --state-path inputdata/e3sm-workflow-state.json
+
+          echo "body_path=${summary_path}" >> "$GITHUB_OUTPUT"
+
+      - name: Create pull request
+        if: steps.detect.outputs.changed == 'true'
+        uses: peter-evans/create-pull-request@v6
+        with:
+          branch: ${{ steps.metadata.outputs.branch }}
+          delete-branch: true
+          base: main
+          title: "Update E3SM inputdata manifest"
+          commit-message: "Update inputdata manifest from E3SM workflow tests"
+          body-path: ${{ steps.metadata.outputs.body_path }}
+          add-paths: |
+            inputdata/files.txt
+            inputdata/files-standalone.txt
+            inputdata/files-missing.txt
+            inputdata/e3sm-workflow-state.json
+            inputdata/provenance.json
+          labels: |
+            automated
+            inputdata

.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

README.md

@@ -0,0 +1,126 @@
+# E3SM Containers Repository
+
+This repository contains Docker build contexts and automation for container
+images used by E3SM CI and related workflows.
+
+## Repository Layout
+
+- `ghci/`: Base GHCI CI container build context.
+- `standalone-ghci/`: Standalone GHCI container build context.
+- `inputdata/`: Inputdata container build context and manifest files.
+- `e3sm-diags-test-data/`: E3SM diags test-data image build context.
+- `tools/inputdata/`: Inputdata manifest generation and support utilities.
+- `.github/workflows/`: GitHub Actions workflows for build/publish and
+  manifest automation.
+
+## Workflow Summary
+
+All workflows live in `.github/workflows`.
+
+### `ghci.yaml`
+
+- Purpose: Build the `ghcr.io/<org>/<repo>-ghci` image.
+- Triggers:
+  - `pull_request` to `main` when `ghci/**` or workflow file changes.
+  - `push` to `main` on same paths.
+  - tags matching `ghci-*`.
+- Behavior:
+  - Always builds on PRs and pushes.
+  - Pushes image only when event is not `pull_request`.
+
+### `standalone-ghci.yaml`
+
+- Purpose: Build the `ghcr.io/<org>/<repo>-standalone-ghci` image.
+- Triggers:
+  - `pull_request` to `main` when `standalone-ghci/**` or workflow file
+    changes.
+  - `push` to `main` on same paths.
+  - tags matching `standalone-ghci-*`.
+- Behavior:
+  - Uses `ubuntu-22.04-arm` runner matrix.
+  - Pushes image only when event is not `pull_request`.
+
+### `e3sm-diags-test-data.yaml`
+
+- Purpose: Build the `ghcr.io/<org>/<repo>-e3sm-diags-test-data` image.
+- Triggers:
+  - `merge_group` to `main`.
+  - `pull_request` to `main` when `e3sm-diags-test-data/**` or workflow file
+    changes.
+  - `push` to `main` on same paths.
+  - tags matching `e3sm-diags-test-data-*`.
+- Behavior:
+  - Pushes image only when event is not `pull_request`.
+
+### `inputdata.yaml`
+
+- Purpose: Build inputdata images from manifest files in `inputdata/`.
+- Triggers:
+  - `pull_request` to `main` when `inputdata/**` or workflow file changes.
+  - `push` to `main` on same paths.
+  - tags matching `inputdata-*`.
+- Behavior:
+  - Matrix builds two variants:
+    - `files` from `inputdata/files.txt`.
+    - `files-standalone` from `inputdata/files-standalone.txt`.
+  - Pushes images only when event is not `pull_request`.
+
+### `update-inputdata-manifest.yml`
+
+- Purpose: Regenerate inputdata manifests from E3SM workflows and open a PR
+  when generated outputs change.
+- Triggers:
+  - Weekly schedule (`cron: 0 12 * * 1`).
+  - Manual `workflow_dispatch`.
+- Behavior:
+  - Checks out this repository and `E3SM-Project/E3SM` at `master`.
+  - Runs `python3 -m tools.inputdata.generate_inputdata_manifest`.
+  - Opens a PR only when these payload-defining files change:
+    - `inputdata/files.txt`
+    - `inputdata/files-standalone.txt`
+    - `inputdata/files-missing.txt`
+  - If a PR is opened, it also includes related metadata files:
+    - `inputdata/e3sm-workflow-state.json`
+    - `inputdata/provenance.json`
+  - If no changes: exits without opening a PR.
+  - If changes exist: uses `peter-evans/create-pull-request` to open a real
+    PR with generated metadata.
+
+## How Workflows Fit Together
+
+1. `update-inputdata-manifest.yml` is the source-of-truth updater for
+   inputdata manifest content.
+2. When it opens a PR that modifies files under `inputdata/`,
+   `inputdata.yaml` runs automatically on that PR and validates container builds
+   without pushing images.
+3. After the PR is reviewed and merged to `main`, `inputdata.yaml` runs again
+   on the `push` event and publishes updated inputdata images to GHCR.
+4. `ghci.yaml`, `standalone-ghci.yaml`, and `e3sm-diags-test-data.yaml` are
+   independent image pipelines and do not depend on inputdata-manifest updates.
+
+In short: one workflow updates inputdata manifests, a different workflow builds
+and publishes inputdata images from those manifests.
+
+## Common Trigger Pattern
+
+Most image workflows follow the same model:
+
+- Build on PRs for validation.
+- Publish on `push` to `main` or on release-style tags.
+- Scope execution with `paths` filters to avoid unrelated builds.
+
+## For New Contributors
+
+If you are updating manifest generation logic:
+
+1. Change code under `tools/inputdata/`.
+2. Run the generator locally if needed.
+3. Use `workflow_dispatch` on `update-inputdata-manifest.yml` to test the
+   automation path.
+
+If you are updating container images directly:
+
+1. Modify the corresponding directory (`ghci/`, `standalone-ghci/`,
+   `inputdata/`, or `e3sm-diags-test-data/`).
+2. Open a PR and confirm the matching workflow build succeeds.
+3. Merge to `main` to publish new images.

tools/inputdata/README.md

@@ -0,0 +1,56 @@
+# Inputdata Generator
+
+`tools/inputdata` contains the generator that builds the list of inputdata files required for Github
+Continuous Integration tests.  The purpose is to produce the files that should be included in the CI
+Docker container, avoiding the need for them to be downloaded during CI.
+
+## What it does
+
+The main entry point is `python3 -m tools.inputdata.generate_inputdata_manifest`. It scans an E3SM checkout, extracts test names from workflow files, runs CIME `create_test` for full-model cases, and writes manifest and provenance files that describe the input files referenced by those tests.
+
+It also performs a best-effort standalone EAMxx scan so the generated manifests can include input paths referenced outside of the full-model CIME flow.
+
+## How it works
+
+The generator uses a small pipeline:
+
+1. `tools.inputdata.lib.workflow_parser` reads `.github/workflows/*.yml` and `.yaml` files from the E3SM tree.
+2. It collects full-model test names from `create_test` lines and matrix definitions, and records standalone EAMxx references when present.
+3. `tools.inputdata.lib.cime_manifest.run_create_test` invokes `cime/scripts/create_test` for each full-model test in a setup-only mode (`--no-build`, `--no-run`, `--namelists-only`). That creates the case metadata and Buildconf files needed to discover input lists, but it does not compile or execute the test case, and it avoids actually pulling model input data during the run.
+4. `tools.inputdata.lib.url_normalize` converts discovered local paths into public inputdata URLs when they point at supported data.
+5. `tools.inputdata.lib.inputdata_validation.validate_inputdata_reference` checks the candidate URL with a lightweight remote existence probe without downloading the file. It tries the E3SM public inputdata server first, then the CESM mirror. When both endpoints definitively return not-found responses, the entry is recorded as missing; when the servers cannot be verified cleanly, the entry is kept but marked unverified in provenance.
+6. The generator writes the final outputs in a stable order so repeated runs produce the same files.
+
+Some `create_test` invocations can still print `FAIL` during setup. In this workflow, that usually means CIME could not finish case setup for that particular test, so the generator records the failure as a warning and moves on to the next test instead of treating it as a fatal error for the whole manifest run.  Usually, the needed information is discovered despite this.
+
+## Inputs
+
+The command requires:
+
+- `--e3sm-root`: path to the E3SM checkout
+- `--test-root`: directory where CIME test cases are created
+- `--output-root`: directory where generated artifacts and temporary outputs are written
+- `--state-out`, `--files-out`, `--standalone-files-out`, `--missing-files-out`, `--provenance-out`: output files to write
+
+## Outputs
+
+The generator writes five files to the paths passed on the command line:
+
+- `state-out`: workflow and extraction metadata, including the extracted test inventory. This is the snapshot you can inspect to see which workflows and tests were discovered from the E3SM checkout.
+- `files-out`: full-model inputdata URLs confirmed present on the E3SM or CESM server. This is the primary manifest consumed by the inputdata automation and other tooling that needs the full-model file list.
+- `standalone-files-out`: standalone EAMxx inputdata URLs. This is used for the separate standalone manifest so those paths can be tracked independently from the full-model list.
+- `missing-files-out`: relative paths of input files that could not be confirmed on either the E3SM or CESM server. A `WARNING: missing inputdata` line is also printed to stderr for each entry. Pass `--strict` to fail the run when any missing entries are found.
+- `provenance-out`: per-URL provenance showing where each input was discovered. This file is useful for tracing a URL back to the workflow, test, and case directory or source scan that produced it. It is supplemental traceability data, not a required input to the generator. For standalone discoveries, `case_dir` is intentionally empty because there is no CIME case directory.
+
+## Environment notes
+
+The generator needs an environment with an E3SM checkout and a working CIME `create_test` script available at `cime/scripts/create_test`. It also expects writable paths for `--test-root` and `--output-root`.
+
+## Related modules
+
+- [generate_inputdata_manifest.py](generate_inputdata_manifest.py)
+- [build_pr_metadata.py](build_pr_metadata.py)
+- [lib/workflow_parser.py](lib/workflow_parser.py)
+- [lib/cime_manifest.py](lib/cime_manifest.py)
+- [lib/url_normalize.py](lib/url_normalize.py)
+- [lib/inputdata_validation.py](lib/inputdata_validation.py)