fix(iac): repair invalid HCL on main since 2026-07-03 + fix what the tofu-plan blackout hid by mdheller · Pull Request #1103 · SocioProphet/prophet-platform

mdheller · 2026-07-30T04:38:56Z

Why

.github/workflows/tofu-plan.yml has returned startup_failure since 2026-06-23 and has never actually run. Its first step is tofu fmt -check, so when that step began failing, the four tofu init -backend=false && tofu validate steps behind it never executed either. This PR fixes the HCL and then fixes what the silence was hiding.

All work verified with OpenTofu 1.8.3 — the pinned TOFU_VERSION in tofu-plan.yml, not whatever is on a laptop.

Blocking relationship with #1097

⚠️ #1097 revives tofu-plan.yml and is red until this lands — this breakage is exactly what it catches. Either this merges first, or the two merge together. This PR deliberately does not touch tofu-plan.yml; that file is #1097's lane.

Same 2026-06-23 root cause, also in flight: #1090 (drift lanes), helm-release.yml, search-orchestrator-image-pin.yml — four workflows killed by the allowed_actions: selected policy.

1. The parse error (formatting)

envs/gcp-landing/main.tf had two threshold_rules single-line blocks each carrying two arguments, which HCL forbids. Arrived in c5bc4cfe (#674, 2026-07-03).

There were two bad lines, not one. The parser only ever reports the first (215); 216 was identical and would have failed the very next run. Both are fixed.

tofu fmt also realigned three more files. environments/aws-eks/main.tf was not in the original report — the parse error aborted the recursive walk before fmt ever reached it.

2. What the blackout hid — `tofu validate`

env	before	after
`envs/local`	clean	✅ `Success!`
`envs/gcp-landing`	❌ `Unsupported block type: notifications_rule`	✅ `Success!`
`envs/shared`	clean	✅ `Success!`
`envs/prod`	clean	✅ `Success!`

local, shared and prod were genuinely clean — a useful negative, and they were actually run, not assumed.

⚠️ Semantic change to gcp-landing — flagged, not folded in

Renamed block notifications_rule → all_updates_rule on google_billing_budget.platform.

notifications_rule is not a valid block on this resource in any provider version. Verified against the real provider schema (tofu providers schema -json) for both google 6.50.0 and 7.42.0 — in both, the notification block is all_updates_rule, and all four attributes already inside the block (monitoring_notification_channels, pubsub_topic, schema_version, enable_project_level_recipients) are valid members of it. So this is a pure rename with no change to intended behaviour.

Consequence worth noting: gcp-landing could never have been applied in this state, so the $5k/month budget cap has never been able to deliver a notification.

3. Beyond the four envs (second commit, droppable)

I validated the ten other config roots CI never looks at. Nine were clean. environments/ibm-iks could not even init: modules/ibm-trusted-profile uses ibm_* resources but declared no provider source, so OpenTofu inferred hashicorp/ibm, which does not exist. A root-level declaration does not propagate to child modules.

fab2d02b adds the source declaration. It is isolated in its own commit so it can be dropped without affecting the four envs CI validates.

This does not make ibm-iks valid. It unblocks init, which then surfaces two further pre-existing errors that were previously unreachable, left for follow-up as they need IBM domain decisions:

ibm_is_vpc.this — tag repo:SocioProphet/prophet-platform violates IBM's tag regex ^[A-Za-z0-9:_ .-]+$ (the / is rejected).
ibm_iam_trusted_profile_claim_rule.github — type = "Profile-OIDC" is not permitted; the provider accepts only Profile-SAML or Profile-CR.

4. Findings reported, deliberately NOT fixed here

Providers are completely unpinned in all four CI envs. None of envs/{local,gcp-landing,shared,prod} declares required_providers. infra/tofu/shared/versions.tf pins google to ~> 6.0 and its comment claims it is "used only in envs/gcp-*", but shared/ is a standalone directory that no env ever loads — envs only reference ../../modules/*. gcp-landing therefore resolved google 7.42.0, a full major version above the stated intent. Combined with .terraform.lock.hcl being gitignored, every CI run re-resolves to whatever is newest that day. "Validates clean today" is not reproducible tomorrow. Pinning would change the resolved provider and is a real semantic decision, so it belongs in its own PR.

CI validates 4 of 14 config roots. bootstrap/{aws,azure,gcp}, environments/* (6 dirs) and shared/ are never validated — which is why the ibm-iks break survived indefinitely. fmt -check -recursive covers all of them; validate does not. Flagging for #1097's owner rather than editing that workflow.

Proof

$ tofu version
OpenTofu v1.8.3

$ tofu fmt -check -recursive infra/tofu/
FMT_CHECK_EXIT=0

envs/local        init exit=0  validate exit=0  Success! The configuration is valid.
envs/gcp-landing  init exit=0  validate exit=0  Success! The configuration is valid.
envs/shared       init exit=0  validate exit=0  Success! The configuration is valid.
envs/prod         init exit=0  validate exit=0  Success! The configuration is valid.

Clean-room run: all .terraform/ and lock files deleted first. gcp-landing used TF_VAR_billing_account="000000-000000-000000", matching CI.

Every change is inside infra/tofu/**.

`tofu fmt -check` has been failing with a hard parse error (exit 2) on envs/gcp-landing/main.tf since c5bc4cf (#674). It went unnoticed because tofu-plan.yml has returned startup_failure since 2026-06-23 and never ran. FORMATTING ONLY: - envs/gcp-landing/main.tf: two `threshold_rules` single-line blocks each carried two arguments, which HCL does not allow. Expanded to multi-line. Note the parser only ever reported the first (line 215); the second was identical and would have failed the next run. - environments/aws-eks/main.tf, modules/github-oidc-aws/main.tf, modules/ibm-trusted-profile/variables.tf: `tofu fmt` alignment. aws-eks was not previously flagged because the parse error aborted the recursive walk before reaching it. SEMANTIC (called out deliberately, not folded in): - envs/gcp-landing/main.tf: renamed block `notifications_rule` -> `all_updates_rule` on resource google_billing_budget.platform. `notifications_rule` is not a valid block on this resource in ANY provider version; verified against the real provider schema for both google 6.50.0 and 7.42.0, where the notification block is named `all_updates_rule`. All four attributes already inside the block (monitoring_notification_channels, pubsub_topic, schema_version, enable_project_level_recipients) are valid members of it, so this is a pure rename with no change to intended behaviour. Without it the whole gcp-landing env fails `tofu validate` and could never have been applied, meaning the $5k/month budget alert has never been able to notify. Verified with OpenTofu 1.8.3 (the pinned TOFU_VERSION in tofu-plan.yml): tofu fmt -check -recursive infra/tofu/ -> exit 0 envs/{local,gcp-landing,shared,prod} tofu validate -> all "Success!"

Out of scope for the CI-green fix; isolated in its own commit so it can be dropped without affecting the four envs tofu-plan.yml validates. modules/ibm-trusted-profile uses ibm_* resources but declared no provider source. OpenTofu therefore inferred `hashicorp/ibm`, which does not exist, so `tofu init` failed outright for environments/ibm-iks: provider registry registry.opentofu.org does not have a provider named registry.opentofu.org/hashicorp/ibm The root env declares `IBM-Cloud/ibm` correctly, but that does not propagate to child modules — each module must name a non-hashicorp/ source itself. This is the only module in the repo using a non-hashicorp/ namespace provider, which is why no other module needs the block. Source only; the version stays pinned once, in the root. This does NOT make environments/ibm-iks valid. It unblocks `init`, which then surfaces two further pre-existing errors that were previously unreachable and are left for a follow-up (they need IBM domain decisions): - ibm_is_vpc.this: tag "repo:SocioProphet/prophet-platform" violates IBM's tag regex ^[A-Za-z0-9:_ .-]+$ (the "/" is rejected). - ibm_iam_trusted_profile_claim_rule.github: type "Profile-OIDC" is not a permitted value; the provider accepts only Profile-SAML or Profile-CR. environments/ibm-iks is not one of the four envs CI validates, so neither the old failure nor the remaining two were ever going to be caught.

Copilot

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Repairs OpenTofu/Terraform configuration validity by fixing HCL parse errors, correcting an invalid Google Billing Budget block name, and unblocking IBM module initialization by explicitly declaring the IBM provider source.

Changes:

Fix invalid single-line threshold_rules blocks and rename notifications_rule → all_updates_rule in gcp-landing budget config.
Add required_providers source for the IBM provider inside modules/ibm-trusted-profile to prevent OpenTofu inferring hashicorp/ibm.
Apply tofu fmt alignment changes across a few additional .tf files.

Reviewed changes

Copilot reviewed 2 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
infra/tofu/modules/ibm-trusted-profile/variables.tf	Formatting alignment consistent with `tofu fmt`.
infra/tofu/modules/ibm-trusted-profile/main.tf	Adds IBM provider source declaration to fix `tofu init` provider resolution.
infra/tofu/modules/github-oidc-aws/main.tf	Formatting alignment inside IAM policy JSON.
infra/tofu/envs/gcp-landing/main.tf	Fixes HCL invalid blocks and corrects the Google budget notification block name.
infra/tofu/environments/aws-eks/main.tf	Formatting alignment consistent with `tofu fmt`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+# The IBM provider is the only non-hashicorp/ namespace provider in this repo,
+# so the source address must be declared here as well as in the root module —
+# otherwise OpenTofu infers "hashicorp/ibm", which does not exist, and init
+# fails before it can resolve the real provider. Version stays pinned in the
+# root (envs/.../versions.tf) to keep a single constraint.


… it cannot do infra/tofu/environments/ibm-iks has never passed `tofu validate`. It failed at `tofu init` until #1103 declared the IBM provider source; that unblocked init and exposed two errors that had been unreachable the whole time. Error 1 — IBM user tags must match ^[A-Za-z0-9:_ .-]+$, which rejects the "/" in "repo:SocioProphet/prophet-platform". Split into "org:socioprophet" + "repo:prophet-platform". This is not a new convention: aws-eks, azure-aks and shared/labels.tf already carry a separate "org" = "socioprophet" key and never embed the owner in the repo name. local.prophet_tags is defined per-root (and is a list here, a map elsewhere), so this is scoped to ibm-iks and applies to all four resources that reference it. Error 2 — type = "Profile-OIDC" on ibm_iam_trusted_profile_claim_rule. No such type exists, and this is not a string swap. IBM IAM accepts exactly two: Profile-SAML — federated human users from a registered SAML realm Profile-CR — IBM Cloud compute only; cr_type ∈ VSI, PVS, BMS, IKS_SA, ROKS_SA, CE A GitHub-hosted runner is neither, and cannot mint the compute resource token the cr-token grant requires. IBM's IAM identity-provider API has no OIDC type either — ibm_iam_idp allows only saml, appid, ldap. So there is no value of `type` that federates GitHub Actions; the module's "IBM's equivalent of AWS IRSA / Azure WIF / GCP WIF" header was never achievable. Rather than pick the value that makes the validator happy and federate nothing, the invalid claim rule is removed and the gap is documented in place: module header, calling comment, output description, and README all now say the profile is NOT assumable by CI, with the three supported options and a recommendation. Also flagged, not fixed (out of lane): infra-drift-detect.yml feeds a GitHub OIDC token to grant_type=urn:ibm:params:oauth:grant-type:cr-token, which cannot succeed for the same reason. Verified locally with OpenTofu 1.8.3 (TOFU_VERSION in tofu-plan.yml); Actions is at a spend cap so CI could not run.

… it cannot do (#1107) infra/tofu/environments/ibm-iks has never passed `tofu validate`. It failed at `tofu init` until #1103 declared the IBM provider source; that unblocked init and exposed two errors that had been unreachable the whole time. Error 1 — IBM user tags must match ^[A-Za-z0-9:_ .-]+$, which rejects the "/" in "repo:SocioProphet/prophet-platform". Split into "org:socioprophet" + "repo:prophet-platform". This is not a new convention: aws-eks, azure-aks and shared/labels.tf already carry a separate "org" = "socioprophet" key and never embed the owner in the repo name. local.prophet_tags is defined per-root (and is a list here, a map elsewhere), so this is scoped to ibm-iks and applies to all four resources that reference it. Error 2 — type = "Profile-OIDC" on ibm_iam_trusted_profile_claim_rule. No such type exists, and this is not a string swap. IBM IAM accepts exactly two: Profile-SAML — federated human users from a registered SAML realm Profile-CR — IBM Cloud compute only; cr_type ∈ VSI, PVS, BMS, IKS_SA, ROKS_SA, CE A GitHub-hosted runner is neither, and cannot mint the compute resource token the cr-token grant requires. IBM's IAM identity-provider API has no OIDC type either — ibm_iam_idp allows only saml, appid, ldap. So there is no value of `type` that federates GitHub Actions; the module's "IBM's equivalent of AWS IRSA / Azure WIF / GCP WIF" header was never achievable. Rather than pick the value that makes the validator happy and federate nothing, the invalid claim rule is removed and the gap is documented in place: module header, calling comment, output description, and README all now say the profile is NOT assumable by CI, with the three supported options and a recommendation. Also flagged, not fixed (out of lane): infra-drift-detect.yml feeds a GitHub OIDC token to grant_type=urn:ibm:params:oauth:grant-type:cr-token, which cannot succeed for the same reason. Verified locally with OpenTofu 1.8.3 (TOFU_VERSION in tofu-plan.yml); Actions is at a spend cap so CI could not run.

…alidated bar as envs/* #1108 pinned providers and committed locks for the four `envs/*` roots CI validates, and deliberately left the other nine unlocked. This closes that gap for the six `environments/*` and three `bootstrap/*` roots, so every cloud root in the tree is offline-validated on the same terms: `required_version` present, `required_providers` pinned, lock committed, `tofu init -backend=false && tofu validate` clean. All measurement with OpenTofu 1.8.3 — the TOFU_VERSION in tofu-plan.yml — from a clean .terraform with no plugin cache. Four asymmetries closed: * bootstrap/* declared no `required_version`. It was the only tier without one. The terraform block moves to a versions.tf in each root, so all 14 roots now declare versions in the same filename, and `>= 1.8.0` matches every peer. * Three google majors in one estate, none of it recorded. bootstrap/gcp moves ~> 5.0 -> ~> 6.0, joining environments/gcp-* and (via #1108) envs/*. One google major, on purpose. Validated at BOTH 5.45.2 and 6.50.0: clean, no warnings either side. Same call for azurerm, which was split 3.x/4.x for the same non-reason: bootstrap/azure moves ~> 3.0 -> ~> 4.0 to match environments/azure-aks. Also validated at both majors. * environments/aws-eks pulled tls, time, null and cloudinit transitively from terraform-aws-modules/{vpc,eks} under floor-only constraints — `tls >= 3.0.0` was resolving 4.3.0 and would have crossed into 5.x unannounced. A floor without a ceiling is not a pin. All four are now declared at the root with ceilings; tls and null match the constraints already written in shared/versions.tf. No resolved version changed. * bootstrap/aws was the only root validating with a warning: aws_s3_bucket_lifecycle_configuration with neither `filter` nor `prefix`, which the provider says "will be an error in a future version". `filter {}` is the documented way to say "all objects", which is what the rule already meant. Warning gone. Also fixed: environments/gcp-{ephemeral,persistent} carry their own .gitignore that ignores .terraform.lock.hcl. A nested .gitignore overrides its parent, so #1108's un-ignore does not reach them and both locks would have been dropped silently by `git add`. Those two lines are removed; state stays ignored. Locks use the same platform set as #1108 (linux_amd64, darwin_arm64, darwin_amd64) — verified 3 h1 hashes per provider across all 21 entries — and are stable: a second clean init leaves all eight byte-identical. environments/ibm-iks is deliberately untouched. It is the one root that cannot init, #1103 fixes the cause and #1107 the validate errors, and both are open against it. Its lock belongs to whichever of those lands; the generated content is in the PR body. Evidence is local only. GitHub Actions is at a spend cap with hundreds of jobs queued and none running, so no CI run backs any of this.

…alidated bar as envs/* (#1115) #1108 pinned providers and committed locks for the four `envs/*` roots CI validates, and deliberately left the other nine unlocked. This closes that gap for the six `environments/*` and three `bootstrap/*` roots, so every cloud root in the tree is offline-validated on the same terms: `required_version` present, `required_providers` pinned, lock committed, `tofu init -backend=false && tofu validate` clean. All measurement with OpenTofu 1.8.3 — the TOFU_VERSION in tofu-plan.yml — from a clean .terraform with no plugin cache. Four asymmetries closed: * bootstrap/* declared no `required_version`. It was the only tier without one. The terraform block moves to a versions.tf in each root, so all 14 roots now declare versions in the same filename, and `>= 1.8.0` matches every peer. * Three google majors in one estate, none of it recorded. bootstrap/gcp moves ~> 5.0 -> ~> 6.0, joining environments/gcp-* and (via #1108) envs/*. One google major, on purpose. Validated at BOTH 5.45.2 and 6.50.0: clean, no warnings either side. Same call for azurerm, which was split 3.x/4.x for the same non-reason: bootstrap/azure moves ~> 3.0 -> ~> 4.0 to match environments/azure-aks. Also validated at both majors. * environments/aws-eks pulled tls, time, null and cloudinit transitively from terraform-aws-modules/{vpc,eks} under floor-only constraints — `tls >= 3.0.0` was resolving 4.3.0 and would have crossed into 5.x unannounced. A floor without a ceiling is not a pin. All four are now declared at the root with ceilings; tls and null match the constraints already written in shared/versions.tf. No resolved version changed. * bootstrap/aws was the only root validating with a warning: aws_s3_bucket_lifecycle_configuration with neither `filter` nor `prefix`, which the provider says "will be an error in a future version". `filter {}` is the documented way to say "all objects", which is what the rule already meant. Warning gone. Also fixed: environments/gcp-{ephemeral,persistent} carry their own .gitignore that ignores .terraform.lock.hcl. A nested .gitignore overrides its parent, so #1108's un-ignore does not reach them and both locks would have been dropped silently by `git add`. Those two lines are removed; state stays ignored. Locks use the same platform set as #1108 (linux_amd64, darwin_arm64, darwin_amd64) — verified 3 h1 hashes per provider across all 21 entries — and are stable: a second clean init leaves all eight byte-identical. environments/ibm-iks is deliberately untouched. It is the one root that cannot init, #1103 fixes the cause and #1107 the validate errors, and both are open against it. Its lock belongs to whichever of those lands; the generated content is in the PR body. Evidence is local only. GitHub Actions is at a spend cap with hundreds of jobs queued and none running, so no CI run backs any of this.

mdheller added 2 commits July 30, 2026 00:36

Copilot AI review requested due to automatic review settings July 30, 2026 04:38

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Copilot started reviewing on behalf of mdheller July 30, 2026 12:33 View session

mdheller merged commit 3b698d4 into main Jul 31, 2026
68 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(iac): repair invalid HCL on main since 2026-07-03 + fix what the tofu-plan blackout hid - #1103

fix(iac): repair invalid HCL on main since 2026-07-03 + fix what the tofu-plan blackout hid#1103
mdheller merged 2 commits into
mainfrom
fix/tofu-fmt-validate-blackout

mdheller commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mdheller commented Jul 30, 2026

Why

Blocking relationship with #1097

1. The parse error (formatting)

2. What the blackout hid — tofu validate

⚠️ Semantic change to gcp-landing — flagged, not folded in

3. Beyond the four envs (second commit, droppable)

4. Findings reported, deliberately NOT fixed here

Proof

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

2. What the blackout hid — `tofu validate`