Skip to content

ci(aw): upgrade gh-aw toolchain to v0.88.2 to fix squid startup - #236

Closed
yigitdot wants to merge 3 commits into
mainfrom
claude/github-issue-234-plan-7b4741
Closed

yigitdot wants to merge 3 commits into
mainfrom
claude/github-issue-234-plan-7b4741

Conversation

@yigitdot

@yigitdot yigitdot commented Sep 4, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #234

Problem

Agentic Triage has failed on every run since 2026-09-01 (runs 33537656702, 33738200080, 33825940584), each one filing a fresh failure issue. Last green run was 2026-07-26.

From run 33825940584's logs and agent artifact:

  • awf-squid starts, then reports Error ~0.5 s later with exit code 1, so docker compose up -d --pull never aborts with dependency failed to start: container awf-squid exited (1).
  • firewall/logs/ in the artifact holds only api-proxy-logs/ — squid never wrote access.log or cache.log, i.e. it died during config parse, before opening log files.
  • The audited firewall/audit/squid.conf carries the dual-stack listener:
    http_port 3128
    http_port [::]:3128
    

That matches github/gh-aw-firewall#2139 (FATAL: http_port: IPv6 is not available → ~15 ms container lifetime → exited (1) → no logs), which fires when the Docker daemon has IPv6 disabled in the container netns. The upstream mitigation is a sed in the squid image entrypoint, so it rides on the firewall image version rather than the generated config. We pinned ghcr.io/github/gh-aw-firewall/*:0.25.41 (released 2026-05-06) — a four-month-old toolchain — and the first failure lands one day after runner ImageVersion: 20260831.293.1 shipped, the only thing that changed on our side.

Ruled out: Dependabot's August bumps of uses: lines inside the generated .lock.yml files (#213, #215) are not the cause — the run log confirms awf, the compose images and the config schema all resolved to 0.25.41 consistently. They did leave the embedded gh-aw-manifest header and .github/aw/actions-lock.json stale at v0.72.1 while the uses: lines said v0.84.0.

Changes

Recompiled with a current compiler rather than hand-editing generated lock files.

before after
gh-aw compiler v0.72.1 v0.88.2
gh-aw-firewall (awf + images) 0.25.41 0.28.12
setup action pin v0.84.0, manifest said v0.72.1 v0.88.2, manifest + actions-lock in sync
awf invocation sudo -E awf … rootless (--rootless, no sudo)
  • .github/workflows/issue-triage.md — added features.awf-diagnostic-logs: true, so a repeat failure ships docker logs awf-squid, exit codes and the sanitized compose config under sandbox/firewall/ in the agent artifact instead of leaving us blind.
  • .github/dependabot.yml — ignores github/gh-aw-actions/setup; it has to move in lockstep with the awf binary version, the firewall imageTag and the manifest header, which only gh aw compile can do together.
  • Both .lock.yml files regenerated, then prettier --write (repo convention — lint-staged formats *.yml and CI runs format:check).

Deliberately not run: gh aw update, which would re-pull the upstream githubnext/agentics markdown at a newer ref and clobber the local customisation in issue-triage.md. Only the compiler version moves here; the prompt is untouched.

Two things the new compiler forced

  1. Strict mode requires an explicit tools.bash when github.min-integrity: none. Compilation failed until bash: ["gh label list"] was added — the prompt's only shell need (step 3). The agent previously ran under Copilot's --allow-all-tools, so this narrows the grant rather than widening it.
  2. .gitattributes lost merge=ours on *.lock.yml — the compiler rewrote the file. Kept as-is, since every future compile would strip it again.

Also new: an optional GH_AW_DEFAULT_OTLP_HEADERS telemetry secret, guarded by GH_AW_OTLP_IF_MISSING: ignore. Nothing to configure.

Verification

  • gh aw compile exits clean; pnpm format:check and pnpm lint pass.
  • Version pins verified to move together: GH_AW_VERSION: v0.88.2, install_awf_binary.sh v0.28.12 --rootless, all three gh-aw-firewall/*:0.28.12 images, the awf-config.schema.json URL, container.imageTag, and the manifest header.
  • --diagnostic-logs confirmed present on both awf invocations; triggers, permissions: read-all, the 10-minute agent timeout and the prompt body are unchanged.

This cannot be exercised from the PR branch — on: issues workflows always run from the default branch. After merge, reopen #234 (or open a throwaway issue) and confirm Execute GitHub Copilot CLI gets past Container awf-squid Healthy and the agent job concludes success.

If squid still exits 1, the diagnostics/ directory under sandbox/firewall/ in the run's agent artifact will now carry the squid stderr to file upstream at github/gh-aw-firewall.

🤖 Generated with Claude Code

Closes #234

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 4, 2026 01:57
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Deploying website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 19e9748
Status: ✅  Deploy successful!
Preview URL: https://36effbb1.website-70y.pages.dev
Branch Preview URL: https://claude-github-issue-234-plan.website-70y.pages.dev

View logs

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regenerated *.lock.yml workflows contain invalid YAML (run: followed by | on the next line), which is likely to break workflow parsing/validation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Upgrades the repository’s gh-aw/AWF toolchain and regenerates the compiled *.lock.yml workflows to address the recurring awf-squid startup failures (and to improve diagnostics when the sandbox fails to start).

Changes:

  • Bump gh-aw compiler / setup action pins and regenerate lock workflows (including updated AWF/firewall image versions and rootless AWF invocation).
  • Enable features.awf-diagnostic-logs and explicitly allow the gh label list bash command for strict-mode tool allowlisting.
  • Prevent Dependabot from independently bumping github/gh-aw-actions/setup, keeping it in lockstep with gh aw compile output.
File summaries
File Description
.github/workflows/issue-triage.md Enables AWF diagnostic logs and explicitly allowlists the required bash command for strict mode.
.github/workflows/cost-tracker.lock.yml Regenerated compiled workflow with updated action/image pins and runtime behavior changes from newer gh-aw.
.github/workflows/issue-triage.lock.yml Regenerated compiled workflow with updated action/image pins and runtime behavior changes from newer gh-aw.
.github/dependabot.yml Ignores github/gh-aw-actions/setup so pins move only via gh aw compile.
.github/aw/actions-lock.json Syncs the setup action pin/sha to v0.88.2.
.gitattributes Removes merge=ours from generated lock workflow attributes.
Review details
  • Files reviewed: 4/6 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/cost-tracker.lock.yml Outdated
Comment on lines 1858 to 1862
run:
| # zizmor: ignore[github-env] - GITHUB_SERVER_URL is set by GitHub Actions, not user input.
# Derive GH_HOST from GITHUB_SERVER_URL so the gh CLI targets the correct
# GitHub instance (GHES/GHEC). On github.com this is a harmless no-op.
GH_HOST="${GITHUB_SERVER_URL#https://}"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The YAML is valid — a block scalar indicator on the line after the key is legal YAML, and GitHub Actions parses it fine. Confirmed with a real parser:

$ ruby -ryaml -e ... 
issue-triage.lock.yml OK jobs=6
  run first line: "# Derive GH_HOST from GITHUB_SERVER_URL so the gh CLI targets the correct\n"

The run: scalar resolves to the full script body, so nothing is broken.

There is a smaller real problem underneath it, though. gh aw compile emits that line as:

run: | # zizmor: ignore[github-env] - GITHUB_SERVER_URL is set by GitHub Actions, not user input.

and Prettier was splitting it, detaching the zizmor: ignore directive from the step it annotates. Fixed in b923506 by adding .github/workflows/*.lock.yml to .prettierignore — these are compiler output and Prettier was also rewriting their quote style for no benefit. The lock files now match gh aw compile byte-for-byte, pnpm format:check and pnpm lint still pass, and both files parse.

Comment thread .github/workflows/issue-triage.lock.yml Outdated
Comment on lines 1869 to 1873
run:
| # zizmor: ignore[github-env] - GITHUB_SERVER_URL is set by GitHub Actions, not user input.
# Derive GH_HOST from GITHUB_SERVER_URL so the gh CLI targets the correct
# GitHub instance (GHES/GHEC). On github.com this is a harmless no-op.
GH_HOST="${GITHUB_SERVER_URL#https://}"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The YAML is valid — a block scalar indicator on the line after the key is legal YAML, and GitHub Actions parses it fine. Confirmed with a real parser:

$ ruby -ryaml -e ... 
issue-triage.lock.yml OK jobs=6
  run first line: "# Derive GH_HOST from GITHUB_SERVER_URL so the gh CLI targets the correct\n"

The run: scalar resolves to the full script body, so nothing is broken.

There is a smaller real problem underneath it, though. gh aw compile emits that line as:

run: | # zizmor: ignore[github-env] - GITHUB_SERVER_URL is set by GitHub Actions, not user input.

and Prettier was splitting it, detaching the zizmor: ignore directive from the step it annotates. Fixed in b923506 by adding .github/workflows/*.lock.yml to .prettierignore — these are compiler output and Prettier was also rewriting their quote style for no benefit. The lock files now match gh aw compile byte-for-byte, pnpm format:check and pnpm lint still pass, and both files parse.

@yigitdot
yigitdot requested a review from thiras September 4, 2026 02:01
yigitdot and others added 2 commits September 4, 2026 05:03
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@yigitdot

yigitdot commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up review turned up two things I had wrong, both now fixed in 19e9748.

1. The firewall-audit-logs artifact does not exist. That name came from the upstream feature-flags docs, which are stale relative to what the v0.88.2 compiler emits. grep -rn firewall-audit .github/ matched only my own comment. The workflow uploads activation, agent, agent-output-fallback, detection, usage and safe-outputs-items; the firewall log and audit directories go into agent (issue-triage.lock.yml:1170,1190). Corrected in the workflow comment, in the PR description, and in my two replies on the Copilot threads above.

2. gh label list probably stops working under the new toolchain. The regenerated sandbox config sets isolation:true with topologyAttach:[awmg-mcpg], and its network.allowDomains no longer contains any GitHub domain — api.github.com, github.com and raw.githubusercontent.com were all present on main and are gone (issue-triage.lock.yml:947). GitHub access is meant to flow through the MCP gateway now, but the upstream githubnext/agentics prompt still tells the agent to shell out to gh label list, and there is no GH_TOKEN in the sandbox (GITHUB_MCP_SERVER_TOKEN is --exclude-env'd).

That command demonstrably worked before: the last green run (30217561902, 2026-07-26) shows gh label list 2>/dev/null || echo in agent-stdio.log and applied high-priority, a custom label it could not have guessed.

Fix is two-layered, because add-labels.allowed turns out to be enforcement-only — it is not rendered into the prompt, the agent only ever sees add_labels(max:5):

  • safe-outputs.add-labels.allowed — the guardrail. Rejects anything off the list even under prompt injection.
  • Step 3 of the prompt now names the same labels as a fallback for when the shell call comes back empty.

The two lists have to stay in sync; both carry a comment saying so. Sourced from gh label list, minus dependencies / javascript / github_actions, which are Dependabot's.

Also in this commit: copilot-setup-steps.yml moves setup-cli from v0.84.0 to v0.88.2 so the repo runs one gh-aw generation (it is hand-written, and setup-cli is a distinct dependency name that the new Dependabot ignore does not cover), and AGENTS.md gains a Gotchas bullet — the "generated, don't hand-edit" rule was spread across four files and nothing in CI checks that a lock matches its .md source.

Two review findings I looked at and did not act on: unknown labels are rejected, not silently created (create-if-missing defaults to false), and concurrency.queue: max is a real Actions field gh-aw emits by default when cancel-in-progress is off.

gh aw compile, pnpm format:check and pnpm lint all pass; both lock files parse.

@yigitdot yigitdot closed this Sep 4, 2026
@yigitdot
yigitdot deleted the claude/github-issue-234-plan-7b4741 branch September 4, 2026 16:40
@yigitdot

yigitdot commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this out for the record: we changed our minds on the approach.

Rather than upgrade the toolchain to v0.88.2 and keep maintaining it, we're removing GitHub Agentic Workflows from the repository entirely. Agentic Triage had failed on every run since 2026-09-01 and its only visible output for the last month was auto-filed failure issues; the cost of tracking a fast-moving upstream (a four-month-lagging pin was already enough to break us) isn't worth what the workflows were returning.

Superseded by #237, which reverts the union of #94, #95 and #96 — both agentic workflows and their generated lock files, copilot-setup-steps.yml, .github/agents/, .github/aw/, .vscode/, and .gitattributes (its only line was the *.lock.yml rule). 10 files, 3,331 deletions, no additions. website.yml and docs.yml are now the only workflows in the repo.

The diagnosis in this PR still stands and may be useful if we ever revisit: squid exits 1 on FATAL: http_port: IPv6 is not available (github/gh-aw-firewall#2139), and the fix rides on the firewall image version rather than the generated config.

#234 and #226 have been closed as obsolete and the agentic-workflows label deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw] Agentic Triage failed

2 participants