fix(ci): stop semgrep-general timing out on every PR

[gantoine]

Problem

semgrep-general trips its own timeout often enough to be a real tax on the merge queue. When it does, GitHub reports the job as cancelled rather than failed, so it doesn't read like a normal CI failure. Example run: the job ran 20m15s against a 20 min limit and was killed mid-scan, while every other job in that workflow passed.

I sampled every Security run from Jul 28 to Aug 4 (692 completed semgrep-general jobs):

day	n	timeouts	rate	p50	p90	max
07-28	97	1	1.0%	11.3	12.3	20.2
07-29	100	4	4.0%	11.4	13.6	20.3
07-30	99	0	0.0%	12.1	13.2	15.4
07-31	100	1	1.0%	12.1	13.3	20.2
08-01	98	1	1.0%	12.3	13.2	20.3
08-02	96	2	2.1%	12.3	14.0	20.3
08-03	101	1	1.0%	12.2	13.2	20.2

1.4% overall, and in one 2.5 hour window on Aug 4 it was 7 of 258 (2.7%). Semgrep Checks Pass is a required check, so each one blocks a PR or kicks it out of the queue.

Root cause: the job got 4x slower on Jun 24

This job used to take 3 minutes. Sampling its duration back through the year:

date	p50
2026-02-04	2.1 min
2026-04-04	2.5 min
2026-06-04	2.8 min
2026-06-23	3.0 min
2026-06-24	12.0 min
2026-07-20	11.6 min
2026-08-03	12.2 min

Not gradual growth from the repo getting bigger. A step change overnight. Pulling all 598 semgrep-general runs from Jun 24 puts the cutover inside a two minute window:

20:52:31    2.9m
20:54:43   12.3m   <-- 
20:56:03   11.4m

#65921 (fix(ci): upgrade semgrep 1.163.0 → 1.167.0 to fix intermittent hang) merged at 20:54:35 UTC, eight seconds before the first slow run started. It is the only change to ci-security.yaml or .semgrep/ that day. Before it, 455 runs at p50 3.1 min. After it, 143 runs at p50 12 min.

Warning

The 1.167.0 pin is the actual cause of the slowness, and it is still in place. This PR does not fix that, it just stops it from failing PRs. I tried bumping the version and it did not help, see below.

That regression left a 12 min p50 under a 20 min timeout, about 1.6x headroom, so ordinary runner variance tips it over. #70907 added --jobs 4 three weeks ago to claw the time back and the tail did not move.

Second, independent problem: semgrep-general was the only semgrep job without a needs: changes gate. It scans 1,605 files in packages/, tools/, .github/, bin/ and friends, since everything else is --excluded. Of the last 400 first-parent merges to master, 370 (92.5%) touched nothing it scans and still paid for a 12 min job.

Changes

Gate the job on the complement of its own --exclude list, and raise the timeout to 30 to match semgrep-python.

 semgrep-general:
+    needs: changes
+    if: needs.changes.outputs.general == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 20
+    timeout-minutes: 30

general:
  - '**'
  - '!cli/**'
  - '!common/**'
  - '!ee/**'
  - '!frontend/**'
  - '!livestream/**'
  - '!nodejs/**'
  - '!posthog/**'
  - '!products/**'
  - '!rust/**'
  - '!.semgrep/**'
  - '!docs/**'
  - '!services/**'

Note

Coverage is unchanged. Pushes to master skip the filter entirely (if: github.event_name != 'push'), so master still runs the full scan, and oversized PRs still force every scan via force_all. The aggregator already tolerated skipped for this job, so no change was needed there.

flowchart TD
    A{{PR}} --> B[changes path filter]
    B --> C[semgrep-python]
    B --> D[semgrep-js]
    B --> E[semgrep-devex]
    A --> F[semgrep-general<br/>always runs, 12 min]
    C --> G[Semgrep Checks Pass]
    D --> G
    E --> G
    F --> G
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    class A,G phYellow;
    class B phBlue;
    class F phRed;

Loading

flowchart TD
    A{{PR}} --> B[changes path filter]
    B --> C[semgrep-python]
    B --> D[semgrep-js]
    B --> E[semgrep-devex]
    B --> F[semgrep-general<br/>skipped on ~92% of PRs]
    C --> G[Semgrep Checks Pass]
    D --> G
    E --> G
    F --> G
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    class A,G phYellow;
    class B,F phBlue;

Loading

What I tried and rejected: bumping the version

Pushed 1.172.0 to this PR to see if a newer build cleared the amd64 regression. It did not:

job	p50 on 1.167.0	this run on 1.172.0
semgrep-general	12.3 min	18.7 min
semgrep-python	15.2 min	14.1 min
semgrep-js	5.7 min	5.7 min
semgrep-devex	2.4 min	2.5 min

Nowhere near the 3.1 min it ran at before #65921, and if anything worse. Reverted, so this PR leaves the pin at 1.167.0. One sample against a p50 on shared runners, so I would not claim 1.172.0 is worse, only that it plainly does not fix this.

The regression is isolated to one job

Splitting all 597 Jun 24 semgrep jobs on the 20:54:35 boundary:

job	p50 before	p50 after	change
semgrep-general	3.1 min	12.1 min	3.9x
semgrep-python	8.9 min	10.5 min	1.2x
semgrep-js	4.4 min	4.9 min	1.1x
semgrep-devex	1.7 min	1.8 min	1.0x
semgrep-go	0.7 min	0.7 min	1.0x
semgrep-rust	1.0 min	1.0 min	1.0x
semgrep-products-frontend	0.8 min	0.8 min	1.0x
semgrep-test-rules	0.8 min	0.7 min	1.0x

So it is not a general engine slowdown. Whatever 1.167.0 changed, it only bites semgrep-general's configuration. Two things are unique to that job: it is the only one using --config p/github-actions, and it is the only one running the full rule set without --severity=ERROR (which is what keeps semgrep-desktop at 1 min despite scanning 4,827 files with the same expensive packs, since it cuts 812 rules to 228 and the generic <multilang> rules from 24 to 3).

Which pack: measured, not guessed

I ran two rounds of temporary diagnostic jobs on this PR that split the rule packs, with the 4-pack semgrep-general as a same-commit control. My first guess was p/github-actions, since it is unique to this job and its curl-eval / gha-curl-pipe-shell rules throw parse errors 24 times per scan. That was wrong.

Round 1 cleared it:

job	packs	time
diag-only-github-actions	github-actions	0.7 min
diag-without-github-actions	owasp + security-audit + trailofbits	12.2 min
semgrep-general (control)	all 4	11.6 min

Round 2 pinned it:

job	packs	time
diag-trailofbits	trailofbits	11.0 min
diag-owasp	owasp-top-ten	1.5 min
diag-security-audit	security-audit	1.0 min
diag-severity-error	all 3 + --severity=ERROR	1.4 min
semgrep-general (control)	all 4	12.5 min

So p/trailofbits is essentially the entire cost, 11.0 of 12.5 min, and --severity=ERROR across all three packs collapses it to 1.4 min.

This reconciles with everything else. Local profiling put trailofbits at 69s of 81s, the same shape. If trailofbits is ~11 of today's 12.5 min, it was ~2 of the pre-regression 3.1 min, so Jun 24 was a ~5.5x slowdown localized to that one pack rather than a broad engine change, which is why the other seven jobs stayed flat.

The diagnostic jobs have been removed. They were never wired into semgrep_checks, so they gated nothing while they existed.

Follow-ups (not in this PR)

Decide what to do about p/trailofbits. Three options, all of them someone else's call since they trade off security coverage:

1. Add --severity=ERROR to semgrep-general, as semgrep-desktop already does. Takes it to ~1.4 min. Costs real coverage: 812 rules drop to 228, and the generic <multilang> rules go from 24 to 3. Not a free win.
2. Drop or replace p/trailofbits in this job. Other jobs already load it, so the coverage loss is narrower than it looks.
3. Report it upstream. A pack getting 5.5x slower on linux/amd64 between 1.163.0 and 1.167.0, with no change on arm64, looks like a semgrep bug worth filing.

curl-eval and gha-curl-pipe-shell are broken. Unrelated to performance, now that p/github-actions is cleared. They throw metavariable-pattern failed when parsing 24 times per scan against .github/workflows/**, so those two rules are not doing their job today.

Priority note. With the path gate, semgrep-general runs on roughly 8% of PRs, so the 12 min is much less costly than it was. Worth fixing, no longer urgent.

How did you test this code?

I could not reproduce the slowness locally, and chasing that is what found the root cause. On an arm64 Mac the exact CI command runs in 81 seconds against CI's ~12 minutes, so I A/B'd the pinned versions on identical hardware and flags:

semgrep	local wall time	rules run
1.163.0	82.4s	641 on 1597 files
1.167.0	81.7s	637 on 1597 files
1.172.0	80.9s	-

Identical. The regression does not reproduce on arm64, which is why it points at the linux/amd64 build rather than at rule content. I confirmed both image digests are proper multi-arch OCI indexes with a real linux/amd64 entry, so it is not QEMU emulation. Both versions also emit the same 24 internal matching errors and 217 syntax errors, so they are doing the same work.

Beyond the local A/B, the pack isolation above was measured in CI on this PR across two rounds of throwaway diagnostic jobs, each with the 4-pack scan as a same-commit control. Those jobs carried continue-on-error: true, declared timeout-minutes, and were deliberately kept out of semgrep_checks.needs so they could not gate the required check. All of them have been removed, so the net diff is just the path gate and the timeout.

Automated checks I actually ran:

• bin/hogli lint:workflows - 7/7 checks pass across 123 workflows, including WF003-dorny-negation (which exists to catch exactly the negation footgun this filter relies on) and WF007-required-check-gates.
• hogli ci:preflight - 0 failures.
• The 13 pattern filter run through the vendored paths-filter's real Filter class over 20 representative paths. Temporary test, not committed.

Filter match cases (all pass)

Matches (job runs): .github/workflows/ci-security.yaml, packages/quill/packages/charts/src/core/canvas-renderer.ts, tools/hogli-commands/hogli_commands/doctor.py, bin/sandbox, terraform/main.tf, uv.lock, .agents/skills/x/SKILL.md, playwright/foo.ts

No match (job skips): posthog/api/team.py, products/foo/backend/api.py, frontend/src/App.tsx, services/mcp/src/index.ts, .semgrep/rules/security/a.yaml, rust/capture/src/main.rs, docs/internal/x.md, ee/api/x.py, common/x/y.py, nodejs/x.ts, livestream/main.go, cli/src/main.rs

The dotfile cases matter: the action sets dot: true, so ** does cover .github/** and .agents/**.

The 92.5% figure comes from replaying git log --first-parent -400 master and checking each merge's changed files against the exclude list. Since this PR edits ci-security.yaml, the new filter matches and semgrep-general runs on this PR, so CI here exercises the gate rather than skipping it.

No new tests. The behavior here is a workflow path filter, and bin/hogli lint:workflows already guards the conventions this change touches.

Automatic notifications

• Publish to changelog?

Docs update

None needed.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

I pointed Claude (Claude Code, Opus 5) at a failing semgrep-general job and asked how often it breaks and whether it could be fixed. No repo skills were invoked.

The first useful finding was that the job was not failing on a semgrep rule at all. GitHub surfaces a timeout-minutes kill as cancelled, which is why this reads as noise rather than a bug. Confirming it meant checking that sibling jobs in the same run had completed successfully, ruling out a whole-run cancellation, then separating the ~20.2 min kills from genuine cancellations by duration across a week of runs.

The first pass stopped there and shipped the gate plus the timeout, with a guess that the p/trailofbits pack was simply expensive. That guess was wrong, and the 81s local vs 12 min CI gap was the thread worth pulling: a 14x difference is too large for hardware. Ruling things out one at a time (semgrep version on arm64, rule count, file set, engine errors, image architecture) left "something environmental changed at a point in time", which turned into plotting the job's duration back through the year and bisecting to a two minute window on Jun 24.

Worth flagging for review: the code change here is unchanged from that first pass and is deliberately defensive rather than curative. It stops the bleeding without touching the security scanner pin. Whether to chase the version regression is a separate call, and reverting is specifically not safe given why #65921 happened.

[trunk-io]

🚫 This stack was removed from the merge queue because it was canceled by Georges-Antoine Assi (a GitHub user). See more details here.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(ci): stop semgrep-general timing out..." | Re-trigger Greptile}

[Copilot]

Pull request overview

This PR reduces CI tax and merge-queue churn by preventing the semgrep-general shard in the Security workflow from running on PRs that only touch directories it explicitly excludes from scanning, while also increasing its timeout to avoid runner-variance timeouts.

Changes:

• Add a new general paths-filter output and gate semgrep-general on it (so it skips on PRs that only change excluded paths).
• Increase semgrep-general’s timeout-minutes from 20 to 30 to add runtime headroom.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rnegron]

the needs: changes makes sense to me... similar to #74230

[gantoine]

/trunk merge

[trunk-io]

This PR is already queued as a stacked merge. Cancel it first to re-submit.

[greptile-apps]

_{Reviews (2): Last reviewed commit: "chore(ci): remove temporary semgrep diag..." | Re-trigger Greptile}

[greptile-apps]

_{Reviews (3): Last reviewed commit: "Merge branch 'master' into worktree-keen..." | Re-trigger Greptile}

[gantoine]

Recreated as #78225 on a fresh branch, since this one kept getting stuck in the merge queue. Same diff, same description.