Skip to content

perf(ci): make per-PR cargo-test fast (<10 min) — unit tests for affected crates; integration → nightly/tags #5411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
proggeramlug merged 1 commit into from
Jun 18, 2026
+380 −128
Merged

perf(ci): make per-PR cargo-test fast (<10 min) — unit tests for affected crates; integration → nightly/tags #5411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
139 changes: 88 additions & 51 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,13 @@ on:
- '*.md'
- '!CLAUDE.md'
- '!CHANGELOG.md'
schedule:
# Nightly full-workspace cargo-test safety net (04:00 UTC). The per-PR
# cargo-test gate only exercises crates affected by the diff
# (scripts/ci_test_scope.py); a cross-crate regression that slips a scoped
# PR is caught here within a day. `schedule` is not `pull_request`, so the
# cargo-test job runs the full workspace.
- cron: '0 4 * * *'
# Manual escape hatch for the opt-in jobs. Maintainers (write access)
# can dispatch the workflow against any ref with `run_extended_tests=true`
# to run parity / compile-smoke / package smokes / doc-tests on demand
Expand Down Expand Up @@ -288,6 +295,8 @@ jobs:
# test builds accumulated in target/debug.
CARGO_PROFILE_TEST_DEBUG: "0"
CARGO_PROFILE_DEV_DEBUG: "0"
# For `gh pr view` (PR changed-file list → affected-crate scope).
GH_TOKEN: ${{ github.token }}
run: |
(
while sleep 60; do
Expand All @@ -297,65 +306,93 @@ jobs:
cargo_test_heartbeat_pid=$!
trap 'kill "$cargo_test_heartbeat_pid" 2>/dev/null || true' EXIT

# Test scope: a per-PR run only exercises the crates the diff can
# affect (changed crates + their reverse-dependency closure, plus a
# `perry` edge for runtime-linked stdlib/ext archives) instead of the
# whole workspace (~90 min). Release tags, the nightly cron, and
# workflow_dispatch run the FULL workspace as the safety net. See
# scripts/ci_test_scope.py for the rules.
if [ "${{ github.event_name }}" = "pull_request" ]; then
changed_files="$(gh pr view "${{ github.event.pull_request.number }}" \
--json files --jq '.files[].path')"
echo "Changed files in PR:"; printf '%s\n' "$changed_files"
scope="$(printf '%s\n' "$changed_files" | python3 scripts/ci_test_scope.py)"
else
scope="$(python3 scripts/ci_test_scope.py --full </dev/null)"
fi
echo "Packages in test scope:"; printf '%s\n' "$scope"
if [ -z "$scope" ]; then
echo "No crates affected by this diff — nothing to test."
exit 0
fi
Comment on lines +315 to +327

@coderabbitai coderabbitai Bot Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Read-only verification: compare gh PR-view file count vs paginated REST count.
# Usage:
#   REPO=owner/name PR_NUMBER=123 bash verify_pr_file_count.sh
set -euo pipefail
: "${REPO:?set REPO=owner/name}"
: "${PR_NUMBER:?set PR_NUMBER=<number>}"

echo "gh pr view files count:"
gh pr view "$PR_NUMBER" --repo "$REPO" --json files --jq '.files | length'

echo "Paginated REST files count:"
gh api "repos/$REPO/pulls/$PR_NUMBER/files?per_page=100" --paginate --jq '.[].filename' | wc -l

Repository: PerryTS/perry

Length of output: 102

🏁 Script executed:

# First, locate and read the workflow file
git ls-files ".github/workflows/*.yml" | head -20

Repository: PerryTS/perry

Length of output: 514

🏁 Script executed:

# Read the specific section being reviewed
cat -n .github/workflows/test.yml | sed -n '310,330p'

Repository: PerryTS/perry

Length of output: 1385

🏁 Script executed:

# Check if there are any other references to this script or error handling patterns
rg "gh pr view|changed_files|ci_test_scope" --type yaml -A 2 -B 2

Repository: PerryTS/perry

Length of output: 39

🌐 Web query:

GitHub CLI gh pr view pagination limit file listing behavior

💡 Result:

The gh pr view command, when used with the --json files flag, historically suffered from a silent truncation of results at 100 items [1][2]. This behavior occurred because the GitHub GraphQL API, which the command relies upon, has a 100-node limit per request, and the command previously did not implement pagination to fetch subsequent pages [1][3]. As of May 2026, this behavior has been updated [1]. A pull request (cli/cli#13340) was merged to introduce pagination for files and commits in gh pr view --json [1]. This update adds PageInfo to the Files and Commits structs and implements concurrent preloading to fetch results beyond the initial 100-node limit [1]. Before this change, users requiring a complete list of files for pull requests with more than 100 changed files were advised to use the gh api command with the --paginate flag to correctly handle GraphQL pagination [3][4]. For simple listings, gh pr diff --name-only is also a common alternative that avoids this specific GraphQL limitation [5].

Citations:

🏁 Script executed:

# Check if GitHub CLI version is pinned in the workflow
cat -n .github/workflows/test.yml | sed -n '1,100p' | grep -i "actions/github-script\|github-cli\|gh"

Repository: PerryTS/perry

Length of output: 311

🏁 Script executed:

# Look for the full context of where this code runs to understand the shell environment
cat -n .github/workflows/test.yml | sed -n '280,340p'

Repository: PerryTS/perry

Length of output: 3976

🏁 Script executed:

# Check if set -euo pipefail is used anywhere in this workflow script
cat .github/workflows/test.yml | sed -n '280,340p'

Repository: PerryTS/perry

Length of output: 3549

🌐 Web query:

GitHub Actions runner GitHub CLI version June 2026 update pagination files

💡 Result:

As of June 2026, the GitHub CLI (gh) is at version 2.94.0 [1], which is the version included in current GitHub Actions runner images [2][3]. Regarding pagination and updates related to GitHub CLI files and commands in June 2026: 1. GitHub CLI 2.94.0 Release: This release, published on June 10, 2026, introduced several new commands, such as the gh discussion set and gh skill list [1]. It does not contain major breaking changes to general pagination logic but maintains standard behavior for existing commands [1]. 2. Pagination in gh pr view: Recent development has addressed silent truncation issues for specific pull request sub-resources [4]. Specifically, pull requests now support improved pagination for files and commits when using gh pr view --json, ensuring that results exceeding the standard 100-item limit are correctly fetched rather than truncated [4]. 3. General Pagination Behavior: - The --paginate flag remains the primary method for handling automatic pagination in gh api calls [5][6]. - For commands like gh issue list or gh pr list, users should continue to use the -L (--limit) flag to define result caps (defaulting to 30) [5]. - Users are advised that --paginate can only handle pagination for a single connection at a time; nested pagination (e.g., paginating comments within an issue) requires custom logic [6]. - Documentation notes that combining --paginate with --jq or --slurp can sometimes lead to complex or unexpected output behavior [7][5]. GitHub Actions runner images have been updated to include GitHub CLI 2.94.0 across all major OS platforms (Ubuntu 26.04, macOS-26, etc.) as of the mid-June 2026 image updates [2][8][9][10][11].

Citations:

🌐 Web query:

cli/cli#13340 GitHub Actions deployment release date

💡 Result:

The GitHub issue referenced as cli/cli#13340 is titled Paginate files and commits in gh pr view --json [1]. This pull request was created on May 4, 2026, and is not related to a GitHub Actions deployment release date [1]. The repository cli/cli does maintain a deployment workflow located at.github/workflows/deployment.yml [2][3], but this file is part of the repository's internal infrastructure and is not associated with a public GitHub Actions release date [3]. There is no specific public release date for a GitHub Actions deployment feature under this issue number [1].

Citations:

Add error handling for gh pr view failures in PR test scope logic.

Line 316 calls gh pr view without guarding against command failures. If the CLI call fails (authentication issue, transient error, etc.), changed_files becomes empty or contains error text, and the script silently proceeds to line 324 where an empty scope exits with success, skipping all tests without alerting.

While GitHub CLI 2.94.0 (deployed to Actions runners mid-June 2026) fixed the 100-item truncation issue for PR files, the lack of error handling remains a availability risk. Tests can be silently skipped due to transient CLI failures.

Suggested hardening 
+          set -euo pipefail
           if [ "${{ github.event_name }}" = "pull_request" ]; then
-            changed_files="$(gh pr view "${{ github.event.pull_request.number }}" \
-              --json files --jq '.files[].path')"
-            echo "Changed files in PR:"; printf '%s\n' "$changed_files"
-            scope="$(printf '%s\n' "$changed_files" | python3 scripts/ci_test_scope.py)"
+            if ! changed_files="$(gh api \
+              "repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files?per_page=100" \
+              --paginate --jq '.[].filename')"; then
+              echo "::warning::Failed to fetch PR files; falling back to full workspace scope."
+              scope="$(python3 scripts/ci_test_scope.py --full </dev/null)"
+            elif [ -z "$changed_files" ]; then
+              echo "::warning::Empty PR file list; falling back to full workspace scope."
+              scope="$(python3 scripts/ci_test_scope.py --full </dev/null)"
+            else
+              echo "Changed files in PR:"; printf '%s\n' "$changed_files"
+              scope="$(printf '%s\n' "$changed_files" | python3 scripts/ci_test_scope.py)"
+            fi
           else
             scope="$(python3 scripts/ci_test_scope.py --full </dev/null)"
           fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/test.yml around lines 315 - 327, The gh pr view command
execution on line 316 lacks error handling, which means if the command fails due
to authentication or transient errors, the changed_files variable becomes empty
or contains error text and the script silently proceeds. Add error checking
immediately after the gh pr view command invocation to detect and fail fast when
the command fails, ensuring that transient CLI failures do not result in
silently skipped tests. Use set -e or an explicit error check like if [ $? -ne 0
] to guard against the gh pr view command failure and prevent the script from
continuing with invalid data.


# #1444: perry-runtime's tests share process-global state — the
# per-thread arena/GC, the timer queues, and the `NOTIFIED` flag are
# process singletons. Running them across the default test-harness
# thread pool lets one test's `js_notify_main_thread` / timer
# scheduling perturb another's wait budget (the event_pump timing
# flakes) and races the GC/threading tests into intermittent SIGSEGV.
# Run perry-runtime single-threaded so the tests can't interfere.
RUST_TEST_THREADS=1 cargo test -p perry-runtime
# `cargo test` only builds lib/bin/test targets — NOT the `staticlib`
# crate-type — so libperry_runtime.a / libperry_stdlib.a are never
# produced by the steps above; they only exist if restored from the
# cache. A PR that touches perry-runtime/perry-stdlib invalidates the
# cached staticlib and cargo never rebuilds it, so integration tests
# that compile with PERRY_NO_AUTO_OPTIMIZE=1 (e.g.
# functional_batch2_regressions, which link the prebuilt archive
# directly) fail with "Could not find libperry_runtime.a". Build the
# staticlibs explicitly so those tests are deterministic regardless of
# cache state.
cargo build -p perry-runtime -p perry-stdlib
find target/debug/deps -maxdepth 1 -type f -perm -111 ! -name '*.so' -delete
# The remaining workspace includes large `perry` / `perry-stdlib`
# test binaries. Keep Cargo build jobs serialized so the runner
# does not link several of those large test binaries at once, then
# run packages one at a time and prune linked test executables so
# target/debug/deps does not exhaust the runner disk mid-job.
export CARGO_BUILD_JOBS=1
workspace_packages="$(
cargo metadata --no-deps --format-version 1 | python3 -c '
import json
import sys

excluded = {
"perry-runtime",
"perry-ui-macos",
"perry-ui-ios",
"perry-ui-visionos",
"perry-ui-tvos",
"perry-ui-watchos",
"perry-ui-gtk4",
"perry-ui-android",
"perry-ui-windows",
"perry-ui-windows-winui",
"perry-doc-fixture-my-bindings",
}
metadata = json.load(sys.stdin)
workspace_members = set(metadata["workspace_members"])
for package in metadata["packages"]:
if package["id"] in workspace_members and package["name"] not in excluded:
print(package["name"])
'
)"

for package in $workspace_packages; do
echo "::group::cargo test -p $package"
cargo test -p "$package"
echo "::endgroup::"
cargo clean -p "$package" || true

if [ "${{ github.event_name }}" != "pull_request" ]; then
# ---- FULL run: release tags / nightly cron / workflow_dispatch ----
# Every target, including the slow auto-optimize integration tests.
if printf '%s\n' "$scope" | grep -qx 'perry-runtime'; then
RUST_TEST_THREADS=1 cargo test -p perry-runtime
fi
# `cargo test` only builds lib/bin/test targets — NOT the `staticlib`
# crate-type — so libperry_runtime.a / libperry_stdlib.a are never
# produced by the steps above; they only exist if restored from the
# cache. Integration tests that compile with PERRY_NO_AUTO_OPTIMIZE=1
# (e.g. functional_batch2_regressions) link the prebuilt archive
# directly and fail with "Could not find libperry_runtime.a" if the
# cached staticlib was invalidated. Build them explicitly.
if printf '%s\n' "$scope" | grep -qE '^(perry|perry-stdlib)$'; then
cargo build -p perry-runtime -p perry-stdlib
fi
find target/debug/deps -maxdepth 1 -type f -perm -111 ! -name '*.so' -delete
done
# Large perry / perry-stdlib integration-test binaries: serialize
# builds and prune linked executables between packages so the runner
# disk doesn't exhaust mid-job.
export CARGO_BUILD_JOBS=1
for package in $(printf '%s\n' "$scope" | grep -vx 'perry-runtime'); do
echo "::group::cargo test -p $package"
cargo test -p "$package"
echo "::endgroup::"
cargo clean -p "$package" || true
find target/debug/deps -maxdepth 1 -type f -perm -111 ! -name '*.so' -delete
done
else
# ---- FAST per-PR run (<10 min target) ----
# Unit / lib / bin tests for the affected crates only. The slow
# auto-optimize *integration* tests (tests/*.rs — each shells out to
# `perry compile`, ~4–6 min apiece, ~30 of them) are NOT run per-PR;
# they run in the nightly full job, on release tags, and on demand
# via the `run-extended-tests` label. Unit-test binaries are small,
# so builds parallelize safely (no serialization / clean churn) and
# there is no staticlib dependency to build.
#
# perry-runtime first, single-threaded (process-global state); it is
# a lib-only crate, so filter to --lib.
if printf '%s\n' "$scope" | grep -qx 'perry-runtime'; then
RUST_TEST_THREADS=1 cargo test --lib -p perry-runtime
fi
rest="$(printf '%s\n' "$scope" | grep -vx 'perry-runtime' || true)"
if [ -n "$rest" ]; then
# One parallel invocation across the rest of the scope. `cargo test
# --lib` errors if NO selected package has a library (a perry-only
# diff → just the bin-only `perry` crate), so include --lib only
# when some selected crate has one; --bins is always safe.
pkg_args="$(printf '%s\n' "$rest" | sed 's/^/-p /' | tr '\n' ' ')"
if printf '%s\n' "$rest" | python3 scripts/ci_test_scope.py --has-lib; then
target_filter="--lib --bins"
else
target_filter="--bins"
fi
echo "Running: cargo test $target_filter $pkg_args"
cargo test $target_filter $pkg_args
fi
fi

# ---------------------------------------------------------------------------
# GC write-barrier stress (optional / non-blocking)
Expand Down
35 changes: 34 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,37 @@
## v0.5.1184 — fix: unblock main CI — dead-stripped symbol + cargo-test staticlib + oversized link/mod.rs
## v0.5.1185 — perf(ci): make per-PR cargo-test fast (<10 min) — unit tests for affected crates; integration tests move to nightly/tags

The `cargo-test` gate took ~90 min: it built the **entire workspace** in debug,
serially (`CARGO_BUILD_JOBS=1` + `cargo clean` between packages, a 14 GB-disk
workaround), and 30/40 perry **integration** test files (`tests/*.rs`) each shell
out to `perry compile` on the **auto-optimize** path — a whole-program optimized
rebuild, ~4–6 min apiece, and concurrent auto-opt builds thrash a shared target
so they can't be parallelized. That made the gate roughly worthless on every PR.

Per-PR `cargo-test` is now two things at once — **scoped** and **unit-only**:

1. **Scoped to the diff** (`scripts/ci_test_scope.py`): test only each changed
`crates/<dir>` plus its **reverse-dependency closure** (a foundational-crate
change still fans out). Runtime-linked crates (`perry-stdlib`, `perry-ffi`,
`perry-ext-*`) add a `perry` edge (the driver links those archives at runtime,
not via cargo). Infra changes (`.github/`, `scripts/`, `rust-toolchain*`) or
any unrecognized path → full; metadata-only changes (`CHANGELOG.md`,
`CLAUDE.md`, `*.md`, `docs/`, root `Cargo.toml`/`Cargo.lock`) → nothing (a
version-bump PR is instantly green).
2. **Unit / lib / bin tests only** (`cargo test --lib --bins`): the slow
auto-optimize integration tests are **not** run per-PR. Unit-test binaries are
small, so builds parallelize safely (no serialization / clean churn) and there
is no staticlib to build. This is the part that bounds the per-PR wall-clock.

The **full** suite — including every integration test — runs on **release tags**,
a new **nightly `schedule`** (04:00 UTC), `workflow_dispatch`, and any PR labeled
`run-extended-tests`. Release tags gate publishing, so nothing ships untested;
the nightly run is the cross-crate / integration regression backstop (main pushes
don't trigger Tests today). `test.yml` branches the cargo-test step on
`github.event_name`: `pull_request` → fast path; everything else → full.

Trade-off (chosen deliberately, prioritizing a usable per-PR gate): a regression
only an integration test would catch lands on a PR and is caught by the nightly /
release-tag full run rather than at PR time.

Three pre-existing CI fragilities on `main` were turning required checks red for
PRs. Fixed together since all are "make main's CI green again."
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

Perry is a native TypeScript compiler written in Rust that compiles TypeScript source code directly to native executables. It uses SWC for TypeScript parsing and LLVM for code generation.

**Current Version:** 0.5.1184
**Current Version:** 0.5.1185


## TypeScript Parity Status
Expand Down
Loading
Loading