DEVOP-618/619: scheduled govulncheck + pinned Go env in CI

[srt0422]

Linear: DEVOP-618, DEVOP-619

Follow-on to #951 (go-install-hardened adoption) — completes the Shai-Hulud Go-side defense for this repo.

What this PR does

DEVOP-618 — scheduled govulncheck

Adds .github/workflows/go-vuln-scan.yml, a thin scheduled caller of the reusable allora-network/ci-workflows-private/.github/workflows/go-vuln-scan.yml workflow.

• Triggers: weekly cron: '17 5 * * 1' and workflow_dispatch
• fail_on_findings: true so newly-disclosed vulns in our deps surface loudly between PR runs
• go_version: '1.23.5' matches go.mod's toolchain and the pin already used by go-hardened.yml
• Slack alerts via secrets.SLACK_SECURITY_WEBHOOK

The PR-time govulncheck check is already covered transitively by #951 (go-install-hardened.yml runs govulncheck in PR CI). This PR adds the scheduled leg so we keep catching vulns even when no PRs are landing.

DEVOP-619 — pinned Go env

Adds a top-level env: block to every workflow that invokes go ... at the runner level:

Workflow	env added	Why
format_and_test.yml	yes	Runs setup-go + go test (+ integration tests via test/local_testnet_l1.sh)
golangci-lint.yml	yes	Runs setup-go + custom go run linters
goreleaser.yml	yes	Runs setup-go + goreleaser (which shells out to go build)
go-hardened.yml	skipped	Reusable workflow already exports these env vars internally
buf-ci.yaml	skipped	No go at runner level
build_push_docker_hub.yml	skipped	go runs inside the container, not on the runner
build_push_upgrader_docker_hub.yml	skipped	Same

The env block:

env:
  GOPROXY: "https://proxy.golang.org,direct"
  GOSUMDB: "sum.golang.org"
  GOFLAGS: "-mod=readonly"
  CGO_ENABLED: "1"

CGO_ENABLED=1 carve-out (intentional)

DEVOP-619 defaults to CGO_ENABLED=0. This PR mirrors the convention already in go-hardened.yml, which passes cgo_enabled: '1' to the reusable hardened-install workflow, so the Shai-Hulud env block stays uniform across allora-chain's CI.

Notes on the actual CGO picture in this repo (verified against go.mod/go.sum in the diff):

• cosmwasm/wasmvm is NOT in this repo's dependency graph. An earlier draft of this PR (and the comment in go-hardened.yml from feat(ci): adopt go-install-hardened workflow (Shai-Hulud defense) #951) cited wasmvm as the CGO driver; that was inaccurate and the inline comments + plan doc in this PR have been corrected (see commit 3141d489).
• Real CGO-linking deps in the graph: github.com/cockroachdb/pebble (pebbledb build tag) and the cosmos ledger-cosmos-go / ledger-go transitives (ledger build tag).
• Release binaries are built with CGO_ENABLED=0 per-build in .goreleaser.yaml (env: [CGO_ENABLED=0]), which overrides the workflow-level env in goreleaser.yml. The workflow env is effectively a no-op for the release build itself; it's kept at "1" so a single Shai-Hulud env block stays reusable. If the audit shows the test/lint workflows tolerate CGO_ENABLED=0 cleanly, flipping is straightforward — left for a follow-up rather than scoped into this PR.

Forbidden-env audit

rg 'GONOSUMCHECK|GOFLAGS:.*-insecure|GOSUMDB:.*off' .github/workflows/

Clean — no hits, nothing to roll back.

Reusable-workflow pin — reviewer action required

The new go-vuln-scan.yml is pinned to commit 185ec34dc707e6779bcc06fe2a592e07f29de8ed, which is the current HEAD of ci-workflows-private PR #17 (DEVOP-616). PR #17 has not yet merged to ci-workflows-private's default branch.

Why this is safe to merge as-is:

• The SHA is reachable today via refs/pull/17/head, so uses: resolves immediately and the workflow file is valid.
• The same SHA stays reachable after PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17 merges (it becomes part of main history), so the scheduled run keeps working the moment PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17 lands.
• The only GC scenario is PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17 being closed without merging: GitHub keeps PR-head commits for ~90 days after the source branch is deleted. If that happens, the next cron tick (Monday 05:17 UTC) surfaces it as a workflow failure rather than silently regressing.

What the reviewer should do:

• Option A (preferred): merge PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17 first, then re-target this PR's pin to PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17's merge SHA, then merge this.
• Option B: merge this PR first, then open a follow-up to re-pin go-vuln-scan.yml to the merge SHA once PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17 lands.

Either way, the long-term pin should be the merge commit on main for auditability.

Plan

docs/plans/2026-05-25-devop-618-619-govulncheck-go-env.md

Files modified

• .github/workflows/format_and_test.yml — env block
• .github/workflows/golangci-lint.yml — env block
• .github/workflows/goreleaser.yml — env block
• .github/workflows/go-vuln-scan.yml — new (scheduled govulncheck caller)
• docs/plans/2026-05-25-devop-618-619-govulncheck-go-env.md — new

How I tested

• actionlint .github/workflows/{format_and_test,golangci-lint,goreleaser,go-vuln-scan}.yml — only pre-existing warnings (goreleaser tag glob), no new issues from this PR.
• Audit rg for forbidden Go env: clean.
• Workflow syntax matches existing patterns in this repo (permissions: + env: at top level, jobs unchanged).
• Two cycles of ce-code-review (headless, plan-aware) run against this PR. Cycle 1 surfaced two P2 advisories (wasmvm citation accuracy + PR [ORA-673] [ORA-674] Payment for Inferences: Add keeper data structures #17 GC risk inline doc); both were addressed in commit 3141d489. Cycle 2 verdict: Ready to merge.

Risk

• Low. New env block matches what go-install-hardened.yml already exports for PR-time hardened builds; behavior of existing jobs is unchanged except for the integrity-pinning guarantees. CGO build path is preserved (and overridden to 0 per-build by .goreleaser.yaml for releases, as before).
• The scheduled workflow first runs after merge; if pin resolution fails we'll see it in the next Monday tick and can re-pin.

Out of scope / known pre-existing

• goreleaser.yml still pins go-version: 1.22.2 against go.mod's toolchain go1.23.5. Pre-existing; left for a separate change to keep this PR focused on Shai-Hulud env hardening.

[cubic-dev-ai]

cubic analysis

2 issues found across 5 files

Linked issue analysis

Linked issue: DEVOP-618: Add govulncheck to PR CI in all 11 active Go repos

Status	Acceptance criteria	Notes
✅	Add a scheduled go-vuln-scan.yml invocation (weekly)	A new scheduled workflow file was added that invokes the reusable go-vuln-scan workflow on a weekly cron and on workflow_dispatch, with fail_on_findings=true.
⚠️	Ensure PR CI runs govulncheck (call go-install-hardened.yml OR add standalone govulncheck step)	The PR states PR-time govulncheck is covered transitively by prior PR #951 (go-install-hardened adoption), but this PR's diff does not add or modify a PR-time govulncheck step itself.
✅	Add pinned Go env to workflows that invoke go at the runner level (GOPROXY, GOSUMDB, GOFLAGS, CGO_ENABLED)	Top-level env blocks were added to the workflows that run go on the runner as described.
❌	CI green; govulncheck must pass with 0 known vulnerabilities, or document an unfixable-and-unreachable exception	No govulncheck result or exception documentation is present in this PR. The scheduled run will occur after merge; PR-time results are claimed to be covered by another PR but not evidenced here.
✅	One PR per repo (or roll into adoption umbrella)	This is a per-repo PR implementing the scheduled scan and env pin for this repository.

Architecture diagram

sequenceDiagram
    participant Sched as Cron Schedule
    participant Dispatch as workflow_dispatch
    participant VulnScan as go-vuln-scan.yml
    participant Reusable as reusable go-vuln-scan.yml
    participant GitHub as GitHub API
    participant Slack as Slack Webhook

    participant FormatTest as format_and_test.yml
    participant Lint as golangci-lint.yml
    participant Goreleaser as goreleaser.yml
    participant Go as Go Toolchain

    Note over Sched,Slack: DEVOP-618: Scheduled Govulncheck

    alt Weekly cron '17 5 * * 1'
        Sched->>VulnScan: Trigger weekly
    else Manual dispatch
        Dispatch->>VulnScan: Trigger on demand
    end

    VulnScan->>Reusable: uses: allora-network/ci-workflows-private/.github/workflows/go-vuln-scan.yml@185ec34dc
    Reusable->>GitHub: Fetch module graph from default branch
    GitHub-->>Reusable: Module graph
    Reusable->>Reusable: govulncheck with go_version: '1.23.5', fail_on_findings: true

    alt Vulnerabilities found
        Reusable->>Slack: POST alert via secrets.SLACK_SECURITY_WEBHOOK
        Slack-->>Reusable: 200 OK
    else No vulnerabilities
        Reusable->>Slack: (optional) Success notification
    end

    Note over FormatTest,Go: DEVOP-619: Pinned Go Env in CI

    FormatTest->>Go: go test (with env block)
    Lint->>Go: go run (custom linters, with env block)
    Goreleaser->>Go: Shells out to go build (with env block)

    Note over FormatTest,Goreleaser: Env block applied at workflow level
    Note over FormatTest,Goreleaser: GOPROXY, GOSUMDB, GOFLAGS, CGO_ENABLED=1

    Go->>Go: CGO_ENABLED=1 required for cosmwasm/wasmvm
    Note over Go: CGO carve-out matches go-hardened.yml behavior
    Note over Go: Removal tracked in ENGN-8441

Loading

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[xmariachi]

a comment on CGO_ENABLED on linter file.
They must be aligned. Maybe it'd be better to consolidate all findings in the same branch?

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[srt0422]

@xmariachi — addressing your review comment:

a comment on CGO_ENABLED on linter file. They must be aligned. Maybe it'd be better to consolidate all findings in the same branch?

Addressed in 9aa6225 (consolidated into this PR): all four CGO_ENABLED surfaces are now aligned on '0' in the same branch:

• .github/workflows/format_and_test.yml — top-level env: CGO_ENABLED: "0"
• .github/workflows/golangci-lint.yml — top-level env: CGO_ENABLED: "0"
• .github/workflows/goreleaser.yml — top-level env: CGO_ENABLED: "0"
• .github/workflows/go-hardened.yml — cgo_enabled: '0' input to the reusable hardened-install workflow

Investigation while resolving this PR confirmed the original CGO=1 justification on go-hardened.yml was incorrect:

• rg -i 'wasmvm|libwasmvm|cosmwasm' go.mod go.sum → 0 matches
• rg -l 'import\s+"C"' --type go → 0 matches

No follow-up PR is needed — the matrix is consistent now. The plan doc (docs/plans/2026-05-25-devop-618-619-govulncheck-go-env.md) has been updated to reflect the consolidated state, and each workflow's CGO_ENABLED comment block references ENGN-8441 for the broader wasmvm question being tracked there.

The cubic inline thread on goreleaser.yml (P2 about the same alignment) has been resolved with a verification confirmation.

[srt0422]

Needs-human follow-up #1 addressed (autonomous Option A): added a caller-side notify-on-workflow-failure job to .github/workflows/go-vuln-scan.yml in `c17262cf`. It runs on `needs.govulncheck.result == 'failure'` and posts the workflow-run URL to `SLACK_SECURITY_WEBHOOK`. This covers the silent-failure modes the in-reusable-workflow alert can't catch from inside itself:

• reusable-workflow ref no longer resolves (SHA pin GC'd or repo perms changed) → `govulncheck` job never starts
• runner/checkout/setup-go failure before the scanner runs → in-job alert step is never reached
• govulncheck exits 1 or 2 (scanner error, distinct from exit-3 = findings) → in-job alert's `finding_count != 0` gate is false

Trade-off: on findings (with `fail_on_findings: true`), both the reusable workflow's findings alert AND this workflow-failure alert will post. The duplicate is the acceptable cost of also covering the silent-failure modes.

Re: Option B (re-pin to ci-workflows-private#17's merge SHA): PR #17 merged 2026-05-25 at `e3ddce05...`. The current pin (`185ec34d...` = PR #17 HEAD) is still reachable indefinitely via `refs/pull/17/head` so the GC risk in the original comment is no longer active. Repinning to the merge SHA is a cosmetic follow-up — happy to do it in a follow-up commit on request, but skipping it here to keep this change minimal.

`actionlint` and `python -c 'import yaml; yaml.safe_load(...)'` pass on the edited file. The new job depends only on `bash + jq + curl` (Ubuntu defaults) and gracefully no-ops with a workflow-level `::warning::` when `SLACK_SECURITY_WEBHOOK` is not yet provisioned.

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[srt0422]

Patch ready — needs a workflow-scoped push to land (token gap, not a code issue).

Addressing cubic's open P2 on this PR (#952 (comment)) about the notify-on-workflow-failure job in .github/workflows/go-vuln-scan.yml.

Finding (cubic, valid)

cubic correctly identifies that the comment block above notify-on-workflow-failure overstates the notifier's coverage. Specifically, the claim that this job covers the "reusable-workflow ref no longer resolves (e.g. SHA pin garbage-collected or repo permissions changed) → the govulncheck job never starts" case is incorrect:

• When a uses: reusable-workflow ref fails to resolve, GitHub Actions fails the workflow run at YAML/validation time. The run shows as startup_failure with an empty job graph — no jobs are instantiated at all.
• A job gated by needs: govulncheck therefore never runs in that scenario, so the Slack alert step inside notify-on-workflow-failure cannot fire.
• The same is true for malformed YAML and missing workflow_call secrets.

The notifier does correctly cover post-start failures (runner provisioning / checkout / setup-go errors, govulncheck exit-1/exit-2 scanner errors) — those are real and previously unalerted gaps. The doc just needed to stop claiming more than that.

Fix

Comment-only update to .github/workflows/go-vuln-scan.yml:

• Narrow the documented "covered cases" list to post-start failure modes.
• Add an explicit NOT covered subsection for pre-start workflow-validation failures, noting they fall back to GitHub's native workflow_run failure signal (email + Actions UI red badge), and that an owner-side workflow_run-triggered alerter would be the right fix if/when Slack coverage of that gap is needed (already tracked as DEVOP-618 caller-side follow-up).
• Keep the duplicate-alert-on-findings note intact (behaviour unchanged).

No behavioural change to the workflow itself; the comment now accurately reflects what the job catches.

Diff

diff --git a/.github/workflows/go-vuln-scan.yml b/.github/workflows/go-vuln-scan.yml
@@ -37,20 +37,29 @@ jobs:
     secrets:
       slack_webhook_url: ${{ secrets.SLACK_SECURITY_WEBHOOK }}
 
-  # Defense-in-depth notifier for workflow-level failure modes that the
-  # reusable workflow cannot catch from inside itself:
-  #   - reusable-workflow ref no longer resolves (e.g. SHA pin garbage-collected
-  #     or repo permissions changed) → the `govulncheck` job never starts;
+  # Defense-in-depth notifier for post-start failure modes that the reusable
+  # workflow's in-job Slack alert cannot catch from inside itself. The
+  # `govulncheck` job must actually be instantiated for `needs: govulncheck`
+  # to fire; covered cases are therefore limited to:
   #   - runner provisioning, checkout, or setup-go failure before the scanner
   #     runs → the in-job Slack alert step is never reached;
   #   - govulncheck exits with status 1 or 2 (scanner execution error, distinct
   #     from exit-3 = findings) → the in-job alert step's gating condition
   #     (`finding_count != 0`) is false and no alert is posted.
+  # Explicitly NOT covered: pre-start workflow-validation failures (reusable-
+  # workflow ref no longer resolves because the SHA pin was garbage-collected
+  # or repo permissions changed, malformed YAML, missing required secrets at
+  # the workflow_call layer, etc.). Those surface as `startup_failure` runs
+  # with an empty job graph, so a job gated by `needs: govulncheck` never
+  # runs. They fall back to GitHub's native workflow_run failure signal
+  # (email + Actions UI red badge); an owner-side `workflow_run`-triggered
+  # alerter would be the right tool if/when that gap needs Slack coverage —
+  # tracked as DEVOP-618 caller-side follow-up.
   # On a clean run with findings, the reusable workflow already posts a
   # detailed alert with the finding count; this job will also fire (because
   # fail_on_findings: true causes `govulncheck` to fail) — a benign duplicate
   # that is the acceptable cost of also covering the silent-failure modes
-  # above. Tracked as DEVOP-618 caller-side follow-up.
+  # above.
   notify-on-workflow-failure:
     needs: govulncheck
     if: ${{ always() && needs.govulncheck.result == 'failure' }}

YAML re-validated with python3 -c "import yaml; yaml.safe_load(...)" after the edit.

Why this isn't pushed

The OAuth token used by the review-fix-loop subagent that produced this patch only has gist, read:org, repo scopes — no workflow scope. GitHub rejects the push with:

! [remote rejected]   scott/devop-618-619-govulncheck-go-env -> scott/devop-618-619-govulncheck-go-env
  (refusing to allow an OAuth App to create or update workflow `.github/workflows/go-vuln-scan.yml` without `workflow` scope)

The patch is committed locally in the review worker's worktree as 5964d7f1 against scott/devop-618-619-govulncheck-go-env. A maintainer with a workflow-scoped token can either:

1. Cherry-pick the diff above onto the PR branch and push, or
2. Re-run the patch locally:

   git checkout scott/devop-618-619-govulncheck-go-env
   # apply the diff block shown above
   git commit -am "ci(govulncheck): scope notifier doc to post-start failure modes"
   git push origin scott/devop-618-619-govulncheck-go-env

Same workflow-scope constraint that landed dotgithub#9's last patch via maintainer push; flagging the same way here.

Other open threads

Re-reviewed the rest of this PR — all earlier inline threads (cubic 3296702290, 3296702292, 3300164909 + xmariachi's CGO-alignment comment) were already resolved in earlier commits (c5df96a, 9aa62254) and cubic acknowledged. CI is green on the current HEAD c17262cf (GitGuardian + cubic AI code reviewer both pass).

This PR is otherwise clean; the comment fix above is the last actionable finding.

_{Posted by review-fix-loop subagent. final_state=blocked (gh_token_missing_workflow_scope) until a maintainer lands the patch; once landed, the PR should converge clean on the next loop pass.}

[srt0422]

Final cross-provider review (agent-code-review, 3-pass)
verdict: Ready to merge
counts: P0=0 P1=0 P2=0 P3=1
coverage: Pass1: kimi-k3 correctness+security (sub for Claude Opus 4.7 per user mandate), z-ai/glm-5.2 style (sub for Claude Sonnet 5); Pass2: google/gemini-2.5-pro correctness, deepseek/deepseek-v4-pro security, meta-llama/llama-4-maverick style; Pass3: meta-llama/llama-4-maverick correctness, z-ai/glm-5.2 security (sub for Claude Sonnet 5), meta-llama/llama-4-scout style. Anthropic tier replaced per user directive; claude-code CLI not invoked. All Pass 2/3 findings source-verified before inclusion.

severity	file:line	category	reviewer	summary	suggested_fix
P3	.github/workflows/go-vuln-scan.yml:86	maintainability	persona	curl ... >/dev/null discards the response body that --fail-with-body prints on 4xx/5xx, so a failed Slack delivery surfaces as a red job (intended) but without the diagnostic body in the log. Cross-pass agreement: gemini-2.5-pro (P2) + llama-4-maverick (P3); severity set to P3 because the red-job surfacing still works.	Drop the >/dev/null redirect (Slack's success body is the literal ok), or capture the body into the log only on curl failure.

Refuted (source-verified false positives, not findings):

• llama-4-maverick P1 "pinned SHA may be garbage-collected" — deliberate pin; GC risk and the re-pin follow-up are documented in the workflow's own comment block (lines 5-14).
• gemini-2.5-pro P3 "doc date 2026 is a typo for 2024" — current date is 2026-07; the plan date 2026-05-25 is correct.
• llama-4-maverick P2 "file naming inconsistent" — go-vuln-scan.yml matches the kebab-case convention of go-hardened.yml.
• llama-4-scout "documented workflow paths do not exist" — all 8 referenced workflow files verified present in .github/workflows/.
• llama-4-maverick P3 "$SLACK_URL not validated" — empty check exists at line 74; format validation is unnecessary because a curl failure intentionally surfaces as a red job.
• llama-4-maverick P3 "check the webhook secret in the job-level if:" — secrets.* is not available in if: expressions; the env-materialization pattern used here is the documented workaround.
• llama-4-scout P2 "env var naming inconsistent" (x4 files) — GOPROXY/GOSUMDB/GOFLAGS/CGO_ENABLED are standard Go toolchain names, identical across files.
• llama-4-scout P3 "workflow name not descriptive" — Go vulnerability scan (scheduled) is descriptive.
• llama-4-maverick P2 "env block duplicated across 3 workflows" — suppressed: GitHub Actions has no cross-file YAML anchors; the structural quality pass adjudicated the duplication as acceptable for this platform.