From 6ee4a2a3c40a999720cafd45962beb5864c0e36a Mon Sep 17 00:00:00 2001 From: "Behzad.Mirkhanzadeh" Date: Wed, 2 Sep 2026 23:34:51 +0000 Subject: [PATCH] ci: prevent the coding agent from working in a discarded worktree The Copilot coding agent opened #4756 and #4833 with detailed descriptions and zero changed files. Both reported success and CI was green, so the failure was silent. Log forensics on the working directory of every tool call: run 33692935610 (#4833) worktree -> 0 files run 32806937007 (#4756) worktree -> 0 files run 32814359584 (fork #49) workspace -> 46 files The agent did real work -- edits, make dockerfiles, go build, go vet, git add . -- inside a git worktree under session-state. Its commit/push step only sees $GITHUB_WORKSPACE, so all of it was discarded. agents.md section 6 mandates worktrees and declares the repo root read-only. That is correct for local CLI sessions, where one clone is shared across concurrent sessions, and wrong for the cloud agent, which gets an ephemeral single-session container. An earlier theory blamed the firewall. It was wrong: the successful fork run had 8 blocked domains, the failing run had 1 (telemetry only). Instructions alone are not a control here. The same agents.md was present for all three runs and produced worktree behavior twice and workspace behavior once, so the fork validation proved nothing. copilot-setup-steps.yml therefore installs a git shim at /usr/bin/git and /usr/local/bin/git, with the real binary preserved at /usr/local/lib/git-guard/git.real. Installing at both entry points means every PATH-resolved git for the rest of the job, including the agent phase. It rejects 'worktree add' with a message redirecting to $GITHUB_WORKSPACE and execs real git for everything else. Shadowing by PATH position means it does not depend on $GITHUB_PATH or $GITHUB_ENV propagating into that phase. Argument parsing skips global options that consume a value (-C, -c, --git-dir, ...). Without that, 'git -C worktree add' treats the path as the subcommand and bypasses the guard -- a real hole found while testing the first version. The step self-tests and fails setup if the shim does not resolve first, does not block all five probes, or leaks a directory, so it cannot end up silently unguarded. Verified on ubuntu-latest in run 33695179506: blocked worktree add / add -b / -C worktree add / --no-pager worktree add / -c a=b -C worktree add / --git-dir worktree add passes --version, status, worktree list, -C worktree list, log, -c core.pager=cat diff, commit -m 'worktree add stuff' (not a false positive) The documentation changes scope the worktree rule to local CLI sessions. copilot-instructions.md carries the carve-out above the line that pulls agents.md in, since that line is how the rule reached the cloud agent. SKILL.md adds a working-directory check and forbids reporting completion on an empty diff. None of these are load-bearing; correctness rests on the shim. Verification after merge: re-run #4832 and confirm the agent log has zero tool.execution_start lines under a worktrees path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b043d9fe-4797-42bf-9482-c337444c5a6f Hardening: the shim is installed at /usr/bin/git as well as /usr/local/bin/git. A PATH-position-only shim was bypassable with an explicit '/usr/bin/git worktree add', demonstrated locally. The real binary is preserved at /usr/local/lib/git-guard/git.real, copied once under a guard so a re-run cannot overwrite it with the shim. /usr/lib/git-core/git is deliberately left untouched so git's internal helper dispatch is unaffected. Self-test now covers 8 spellings including both absolute paths. --- .github/copilot-instructions.md | 11 ++ .github/skills/acn-go-version-bump/SKILL.md | 32 +++++ .github/workflows/copilot-setup-steps.yml | 139 ++++++++++++++++++++ agents.md | 13 +- 4 files changed, 194 insertions(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 4bcf90b3fa..628d6d0987 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,3 +1,14 @@ +> **If you are the GitHub Copilot coding agent running on a GitHub Actions runner +> (`$GITHUB_ACTIONS` = `true`): do ALL work in `$GITHUB_WORKSPACE`. Do NOT create +> or use a `git worktree`.** Your commit/push step only sees `$GITHUB_WORKSPACE`. +> Edits made anywhere else are silently discarded and produce a PR with zero +> files. The worktree guidance in `agents.md` §6 applies to local Copilot CLI +> sessions only, where one clone is shared between concurrent sessions — that is +> not your situation. Before your first edit, confirm: +> `cd "$GITHUB_WORKSPACE" && pwd && git rev-parse --show-toplevel`. +> Before reporting completion, confirm `git diff --stat origin/master...HEAD` is +> non-empty; if it is empty, say so instead of describing uncommitted work. + Use all `agents.md` files found from the repository root to the current directory as instructions and context, applying them in root-to-leaf order; if instructions conflict, the `agents.md` closest to the current directory takes precedence. Use relevant repo skills from `.github/skills/` when applicable. Public authorship is human-owned. Copilot and all other AI agents must never diff --git a/.github/skills/acn-go-version-bump/SKILL.md b/.github/skills/acn-go-version-bump/SKILL.md index 2c95e72ecc..d247496d51 100644 --- a/.github/skills/acn-go-version-bump/SKILL.md +++ b/.github/skills/acn-go-version-bump/SKILL.md @@ -588,6 +588,20 @@ When upgrading Go, verify compatibility with AKS supported Kubernetes versions: - **Windows builds**: CNG backend typically works without CGO or GOEXPERIMENT — verify per version - **Do NOT assume "no GOEXPERIMENT" is safe** — always verify the default backend's CGO requirements +### Working Directory (MANDATORY — read before you edit anything) + +**If you are the GitHub Copilot coding agent running on a GitHub Actions runner, do ALL work in `$GITHUB_WORKSPACE`. Do NOT create a git worktree.** + +`agents.md` instructs local CLI sessions to work in a dedicated worktree because the local repo root is shared between concurrent sessions. That does **not** apply to you: the cloud runner is a single-session ephemeral container, and your commit/push step only sees `$GITHUB_WORKSPACE`. Work done in a worktree elsewhere on disk is **silently discarded**, producing a PR with a detailed description and zero files. + +This has already caused two empty PRs (#4756, #4833). Verify before you start: + +```bash +cd "$GITHUB_WORKSPACE" && pwd && git rev-parse --show-toplevel +``` + +Both must print the same path, and every later command must run from there. + ### Pre-Submit Cleanup (MANDATORY) **Before committing or updating a PR, you MUST audit your working tree for build artifacts and binaries.** Go builds can produce binaries in module directories that must NOT be committed. @@ -612,3 +626,21 @@ done ``` **Known binary locations to NEVER commit:** `azure-ip-masq-merger/azure-ip-masq-merger`, `azure-iptables-monitor/azure-iptables-monitor`, `cilium-log-collector/cilium-log-collector`, `tools/azure-npm-to-cilium-validator/azure-npm-to-cilium-validator`, and any other ELF binary produced by `go build`. + +### Final Verification Before Reporting Done (MANDATORY) + +**Never write a PR description describing changes you have not committed.** Before your final `report_progress` / PR update, prove the branch is non-empty: + +```bash +cd "$GITHUB_WORKSPACE" +git status --short +git diff --stat origin/master...HEAD + +if git diff --quiet origin/master...HEAD; then + echo "FATAL: branch has no changes — do not claim the upgrade is done." + echo "Re-check that you edited files under \$GITHUB_WORKSPACE and committed them." + exit 1 +fi +``` + +A Go minor upgrade touches roughly 40-50 files. If `git diff --stat` shows zero (or only a handful), something went wrong — **say so explicitly in the PR description instead of describing work that was not committed.** If a command failed due to a network/DNS block, report the exact blocked command rather than silently giving up. diff --git a/.github/workflows/copilot-setup-steps.yml b/.github/workflows/copilot-setup-steps.yml index def8115959..25c6a71b63 100644 --- a/.github/workflows/copilot-setup-steps.yml +++ b/.github/workflows/copilot-setup-steps.yml @@ -32,6 +32,145 @@ jobs: echo "${GOPATH}/bin" >> "$GITHUB_PATH" echo "GOPATH=${GOPATH}" >> "$GITHUB_ENV" + - name: Enforce work-in-workspace (block git worktree) + run: | + set -euo pipefail + + # The coding agent has twice done all its work inside a git worktree + # outside $GITHUB_WORKSPACE (#4756, #4833). Only $GITHUB_WORKSPACE is + # committed and pushed, so that work was silently discarded and the PR + # landed with zero files. + # + # agents.md tells local CLI sessions to use a worktree, which is + # correct there (one clone shared by concurrent sessions) and wrong + # here. Instructions alone did not reliably stop it, so this removes + # the capability rather than asking. + # + # /usr/local/bin precedes /usr/bin in PATH, so this shim intercepts + # every PATH-resolved `git` for the rest of the job, including the + # agent phase. It does not depend on $GITHUB_PATH or $GITHUB_ENV + # propagating into that phase. + + REAL_GIT=$(command -v git) + if [ "$REAL_GIT" != "/usr/bin/git" ]; then + echo "::error::Expected git at /usr/bin/git, found ${REAL_GIT}. Refusing to install shim." + exit 1 + fi + + # Preserve the real binary once. Guarded so a re-run cannot copy an + # already-installed shim over it and destroy real git. + sudo mkdir -p /usr/local/lib/git-guard + if [ ! -x /usr/local/lib/git-guard/git.real ]; then + sudo cp -p /usr/bin/git /usr/local/lib/git-guard/git.real + fi + /usr/local/lib/git-guard/git.real --version > /dev/null + + sudo tee /usr/local/lib/git-guard/git-shim > /dev/null <<'SHIM' + #!/usr/bin/env bash + # Guard: block worktree creation in the Copilot cloud agent environment. + # All other git subcommands and options pass through untouched. + + args=("$@") + n=${#args[@]} + i=0 + sub="" + + # Walk past git's global options to find the subcommand. Options that + # consume a following value (-C , -c =, ...) must skip two + # slots, otherwise their value is mistaken for the subcommand and + # `git -C worktree add` slips through. + while [ "$i" -lt "$n" ]; do + case "${args[$i]}" in + -C|-c|--git-dir|--work-tree|--namespace|--exec-path|--super-prefix|--config-env) + i=$((i + 2)) ;; + -*) + i=$((i + 1)) ;; + *) + sub="${args[$i]}"; break ;; + esac + done + + if [ "$sub" = "worktree" ]; then + # Find the worktree subcommand (add / list / remove / prune / ...). + j=$((i + 1)) + while [ "$j" -lt "$n" ]; do + case "${args[$j]}" in + -*) j=$((j + 1)) ;; + *) break ;; + esac + done + + if [ "${args[$j]:-}" = "add" ]; then + cat >&2 <<'MSG' + git worktree add is disabled in this environment. + + You are the GitHub Copilot coding agent on a GitHub Actions runner. + Only $GITHUB_WORKSPACE is committed and pushed. Work done in a worktree + elsewhere on disk is discarded and produces a pull request with zero + files. + + Do all work directly in $GITHUB_WORKSPACE: + + cd "$GITHUB_WORKSPACE" + + The worktree guidance in agents.md section 6 applies to local Copilot + CLI sessions only, where one clone is shared between concurrent + sessions. That is not your situation. + MSG + exit 1 + fi + fi + + exec /usr/local/lib/git-guard/git.real "$@" + SHIM + + # Install at every entry point the agent could plausibly invoke, not + # just the PATH-resolved one. /usr/lib/git-core/git is deliberately + # left as the real binary so git's own helper dispatch is untouched. + sudo chmod 755 /usr/local/lib/git-guard/git-shim + sudo cp /usr/local/lib/git-guard/git-shim /usr/local/bin/git + sudo cp /usr/local/lib/git-guard/git-shim /usr/bin/git + sudo chmod 755 /usr/local/bin/git /usr/bin/git + + # Prove the shim resolves first, blocks every spelling of + # `worktree add`, and leaves everything else working. Any failure here + # fails setup rather than silently leaving the agent unguarded. + hash -r + test "$(command -v git)" = "/usr/local/bin/git" + + git --version > /dev/null + git -C "$GITHUB_WORKSPACE" rev-parse --show-toplevel > /dev/null + git -C "$GITHUB_WORKSPACE" status --porcelain > /dev/null + git -C "$GITHUB_WORKSPACE" worktree list > /dev/null + git -c user.name=x -C "$GITHUB_WORKSPACE" log --oneline -1 > /dev/null + # Fetch/clone still work, so nothing legitimate is broken. + git -C "$GITHUB_WORKSPACE" fetch --dry-run origin > /dev/null 2>&1 || true + + blocked() { + if "$@" 2>/dev/null; then + echo "::error::Shim failed to block: $*" + exit 1 + fi + } + blocked git worktree add /tmp/guard-probe-1 + blocked git worktree add -b probe /tmp/guard-probe-2 + blocked git -C "$GITHUB_WORKSPACE" worktree add /tmp/guard-probe-3 + blocked git --no-pager worktree add /tmp/guard-probe-4 + blocked git -c core.pager=cat -C "$GITHUB_WORKSPACE" worktree add /tmp/guard-probe-5 + blocked git --git-dir "$GITHUB_WORKSPACE/.git" worktree add /tmp/guard-probe-6 + # Absolute-path invocation must not be an escape hatch. + blocked /usr/bin/git worktree add /tmp/guard-probe-7 + blocked /usr/local/bin/git worktree add /tmp/guard-probe-8 + + for p in /tmp/guard-probe-*; do + if [ -e "$p" ]; then + echo "::error::Worktree leaked past the shim: $p" + exit 1 + fi + done + + echo "✓ git worktree add blocked (8 variants, incl. absolute paths); all other git subcommands pass through" + - name: Pre-download module dependencies (before firewall activates) run: | set -euo pipefail diff --git a/agents.md b/agents.md index 212b55c4f6..69a5f86593 100644 --- a/agents.md +++ b/agents.md @@ -110,6 +110,15 @@ code. ## 6. PR Workflow +> **Exception — GitHub Copilot coding agent (cloud):** if you are running as the +> Copilot coding agent on a GitHub Actions runner (i.e. `$GITHUB_ACTIONS` is +> `true` and the repo is checked out at `$GITHUB_WORKSPACE`), **do NOT create a +> worktree.** Do all work directly in `$GITHUB_WORKSPACE`. That environment is a +> single-session ephemeral container, so there is nothing to collide with, and +> the agent's commit/push step only ever sees `$GITHUB_WORKSPACE` — work done in +> a worktree elsewhere on disk is silently discarded and produces an empty PR. +> The rest of this section applies to local Copilot CLI sessions only. + **All local agent work MUST happen in a dedicated git worktree, never in the shared repo root.** The repo root is shared across concurrent agent sessions; mutating it causes parallel branches, working trees, and build artifacts (`output/`, `bin/`) to collide. From the repo root, create a worktree under the active session folder before touching files: ```bash @@ -130,7 +139,9 @@ publishing is requested. After the maintainer confirms that the work was merged or abandoned, prune with `git worktree remove "$WT"` and `git branch -D "$BRANCH"`. The shared root is read-only — only use it for inspection (e.g. `git worktree list`, `git fetch`), -never to edit, build, or commit. +never to edit, build, or commit. (Again: this read-only rule is for local CLI +sessions. The cloud coding agent must edit, build, and commit directly in +`$GITHUB_WORKSPACE`.) **Go-specific notes (no per-worktree install):** - The Go module cache (`$(go env GOMODCACHE)`, default `$GOPATH/pkg/mod`) and build cache (`$(go env GOCACHE)`) are **content-addressed and concurrent-safe**; Go uses file locking. Do not try to isolate them per worktree.