Skip to content

ci(tofu-plan): revive a workflow that has been rejected before starting since 2026-06-23 - #1097

Open
mdheller wants to merge 1 commit into
mainfrom
fix/tofu-plan-setup-opentofu-blocked
Open

ci(tofu-plan): revive a workflow that has been rejected before starting since 2026-06-23#1097
mdheller wants to merge 1 commit into
mainfrom
fix/tofu-plan-setup-opentofu-blocked

Conversation

@mdheller

@mdheller mdheller commented Jul 30, 2026

Copy link
Copy Markdown
Member

What was wrong

tofu-plan.yml has been 100% inert since 2026-06-23. Last success 2026-06-10; every run from 2026-06-23 onward is startup_failure.

{"failure":6,"startup_failure":25,"success":2}

The workflow's logic is fine. The run is rejected at creation, before any job starts, so there are no logs and no annotation pointing at the cause — which is exactly why this sat quiet for five weeks.

This repo runs allowed_actions: selected:

{"github_owned_allowed":true,"verified_allowed":true,
 "patterns_allowed":["dtolnay/*","Swatinem/*","oven-sh/*","tauri-apps/*"]}

opentofu/setup-opentofu (lines 26 and 95) is not github-owned, matches no pattern, and is not on the Marketplace at allhttps://github.com/marketplace/actions/setup-opentofu returns 404, so verified_allowed cannot cover it. It is on none of the three permitted grounds.

The fix

Both opentofu/setup-opentofu@v1 steps become run: scripts/ci/install-opentofu.sh — the pinned, checksum-verifying installer from #1090. It honours TOFU_VERSION, which this workflow already sets to 1.8.3 (line 17), and already carries that version's sha256 for linux amd64/arm64.

Downloading a release archive and checking it against a committed sha256 is strictly stronger than the action was: the checksum is reviewed in-tree, and no third-party code gets the workflow token.

After this change every uses: left in tofu-plan.yml is actions/* (github-owned) plus google-github-actions/auth@v2, which is empirically fine — provision-secrets.yml uses it and succeeded as recently as 2026-07-29.

Why not just allowlist opentofu/*

An admin could add opentofu/* to patterns_allowed. That was considered and rejected: it is a settings change rather than a code change, it cannot be reviewed in a diff, and the installer removes the third-party dependency entirely. This is the same call the drift lane made in #1090; the two lanes should not diverge on it.

Cherry-pick, not a merge — and the divergence risk

#1090 has not merged. At the time of writing it is OPEN / BLOCKED with every check QUEUED (~400 jobs deep, no self-hosted runners), so waiting was not practical.

scripts/ci/install-opentofu.sh here is byte-identical to #1090's copy:

sha256  2d59604f1e3baadef6a5bb075da714840b9275c86720ce592bf8db26881daaf9   (both)

This means the file will exist on two branches until one lands. Whichever merges second will conflict on an identical add, or silently drift if either copy is edited in the meantime. Please do not edit it in only one place. Merging #1090 first and rebasing this PR is the cleanest order, but either order works as long as the file stays identical.

Also: the workflow could not test itself

The paths: filter was infra/tofu/** only. No PR that touched tofu-plan.yml could ever run tofu-plan.yml — which is a large part of why five weeks of startup_failure had no surface on which to become visible.

This adds .github/workflows/tofu-plan.yml and scripts/ci/install-opentofu.sh to the filter, mirroring what #1090 did for infra-drift-detect.yml. It also makes this PR self-verifying: the presence of a tofu-plan run on this PR that is not startup_failure is the proof the fix works.

tools/check_workflow_path_filters.py reports 0 failures and the same 130 pre-existing advisory gaps before and after — this adds no new path-filter debt, and the new installer step is covered by the added glob.

First execution anywhere

Worth a reviewer's eye: #1090 never actually executes this script. Its cloud lanes are all if: github.event_name != 'pull_request', and its self-test covers exit-code mapping, not installation. The fmt-and-validate job here is therefore the first real run of install-opentofu.sh anywhere.

Specifically unproven until this PR's run goes green: the script does install -m 0755 … /usr/local/bin/tofu without sudo, and there is no other precedent in this repo for writing to /usr/local/bin in CI. If that turns out to need sudo on ubuntu-latest, it affects both lanes and should be fixed once, in the shared script.

This is not the only workflow this happened to

Auditing all 109 registered workflows against their run history, four are blocked by this same policy, not one:

workflow runs startup_failure last success blocked on
tofu-plan.yml 33 25 2026-06-10 opentofu/setup-opentofu
infra-drift-detect.yml 27 26 never opentofu/setup-opentofu (fixed by #1090)
helm-release.yml 17 17 never helm/chart-releaser-action@v1.6.0
search-orchestrator-image-pin.yml 11 9 2026-04-26 peter-evans/create-pull-request@v6

helm-release.yml has never had a successful run — it was born broken on 2026-06-23, the same day tofu-plan.yml died, which is when this policy appears to have been applied.

Both remaining blockers are isolated by elimination: every other action in those two workflows is proven to run today in a sibling workflow that succeeds (azure/setup-helm@v4 in deploy-tests.yml, google-github-actions/auth@v2 in provision-secrets.yml, actions/* throughout).

Out of scope for this PR — filed as findings, not fixed here, to keep this change to the one workflow.

Does anything depend on tofu-plan?

No. The only required status check on main is diagnostics-gate (ruleset main-required-checks); the other active ruleset is Copilot review. tofu-plan gates no merge and blocks no deploy — it is plan-only, and the file's own header notes no apply gate exists yet.

So this was untidy rather than a hole. infra-drift-detect.yml is the one that matters: it is the scheduled lane that is supposed to notice infrastructure drift, and it has never once run.


⚠️ Expect this PR to go RED — and that is the fix working

Reviving the workflow immediately catches real breakage that the blackout was hiding. Reproduced locally with OpenTofu against origin/main, touching no .tf file in this PR:

$ tofu fmt -check -recursive infra/tofu/
Error: Invalid single-argument block definition
  on infra/tofu/envs/gcp-landing/main.tf line 215, in resource "google_billing_budget" "platform":
 215:   threshold_rules { threshold_percent = 0.8  spend_basis = "FORECASTED_SPEND" }

A single-line block definition must end with a closing brace immediately
after its single argument definition.

exit 2

Plus two files that are merely unformatted:

infra/tofu/modules/github-oidc-aws/main.tf
infra/tofu/modules/ibm-trusted-profile/variables.tf

All three arrived in the same commit — c5bc4cfe ("feat(model-zoo+iac): …", #674), merged 2026-07-03. That is ten days after this workflow went dark on 2026-06-23. It merged through a guard that was already silently rejected at startup, and nothing has looked at infra/tofu/ since — ten commits landed there in that window.

infra/tofu/envs/gcp-landing/main.tf line 215 is not a style nit: it is invalid HCL that does not parse. It has been on main for four weeks.

Deliberately not fixed here

This PR is scoped to tofu-plan.yml plus the cherry-picked installer, and infra/tofu/ is an active lane. The .tf fixes are filed as a separate follow-up. The one-line fix is:

  threshold_rules {
    threshold_percent = 0.8
    spend_basis       = "FORECASTED_SPEND"
  }

and the other two are tofu fmt -w.

Sequencing: the .tf fix should land first, or be merged together with this PR. Either order is fine — but until it lands, fmt-and-validate is red for a genuine reason and this PR should not be force-merged past it.

Depth of rot is still unknown

fmt check is the first step in fmt-and-validate. Because it exits non-zero, the four tofu init -backend=false && tofu validate steps (local, gcp-landing, shared, prod) never execute. Whatever those would have caught across seven weeks is still unmeasured — this PR proves the gate runs again, not that what is behind the gate is clean.

tofu-plan.yml has produced nothing but `startup_failure` since 2026-06-23.
It last succeeded on 2026-06-10; 25 of its last 33 runs never started.

The cause is not in the workflow's logic. This repo sets
`allowed_actions: selected` with github_owned + verified-Marketplace + the
patterns dtolnay/*, Swatinem/*, oven-sh/*, tauri-apps/*.
`opentofu/setup-opentofu` is none of those — it is not listed on the
Marketplace at all, so `verified_allowed` cannot reach it. GitHub rejects the
run at creation, which is why the conclusion is `startup_failure` and why
nothing in the logs points anywhere.

Replaces both setup-opentofu steps with scripts/ci/install-opentofu.sh, the
pinned checksum-verifying installer. It honours TOFU_VERSION, which this
workflow already sets to 1.8.3, and carries that version's sha256.

Also adds this workflow and the installer to its own `paths:` filter. The
filter was `infra/tofu/**` only, so no PR that touched the workflow could ever
run it — five weeks of silence had no surface on which to show up.

Refs #1090.
Copilot AI review requested due to automatic review settings July 30, 2026 04:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Revives the tofu-plan GitHub Actions workflow that has been failing at creation time due to the repo’s restricted allowed_actions policy, by removing the disallowed third-party action and ensuring workflow changes can trigger the workflow itself.

Changes:

  • Replace opentofu/setup-opentofu@v1 with an in-repo checksum-verifying installer script.
  • Expand paths filters so changes to the workflow and installer script trigger tofu-plan.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
scripts/ci/install-opentofu.sh Adds a pinned, checksum-verified OpenTofu installer to avoid disallowed third-party actions.
.github/workflows/tofu-plan.yml Replaces the disallowed setup action with the script and updates paths filters to make the workflow self-testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +44 to +55
DEST="${TOFU_INSTALL_DIR:-/usr/local/bin}"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

echo "Downloading ${URL}"
curl -fsSL --retry 3 --retry-delay 2 -o "${TMP}/${ZIP}" "$URL"

echo "${SHA256} ${TMP}/${ZIP}" | sha256sum -c -

unzip -q -o "${TMP}/${ZIP}" tofu -d "$TMP"
install -m 0755 "${TMP}/tofu" "${DEST}/tofu"

Comment on lines +40 to +41
- name: Install OpenTofu
run: scripts/ci/install-opentofu.sh
Comment on lines +23 to +32
TOFU_VERSION="${TOFU_VERSION:-1.8.3}"
ARCH="$(uname -m)"
case "$ARCH" in
x86_64|amd64) GOARCH=amd64 ;;
aarch64|arm64) GOARCH=arm64 ;;
*) echo "::error::unsupported architecture ${ARCH}" >&2; exit 1 ;;
esac

# sha256 of tofu_<version>_linux_<arch>.zip, from the upstream SHA256SUMS.
case "${TOFU_VERSION}_${GOARCH}" in
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants