Skip to content

CI: check the generated docs, run e2e in the Playwright container, add a nightly job for the slow tests - #2

Merged
minchenlee merged 1 commit into
mainfrom
ci/gha-improvements
Nov 17, 2026
Merged

minchenlee merged 1 commit into
mainfrom
ci/gha-improvements

Conversation

@minchenlee

Copy link
Copy Markdown
Owner

Why

The full unit tests and the e2e tests move from the developer's machine to GitHub Actions.

  • CI did not run scripts/accuracy.ts --check, scripts/ai-guide.ts --check, or 8 of the 10 files of npm run test:slow.
  • 3 of the last 9 CI runs failed in the Install Chromium step (apt, exit code 100). None failed in a test.

What changes

Only .github/workflows/. No source, test or config file changes.

ci.yml, check job (blocks a pull request)

  • New step: node scripts/accuracy.ts --check (docs/accuracy.md is what the script generates).
  • New step: node scripts/ai-guide.ts --check.

ci.yml, e2e job (blocks a pull request)

  • Runs in mcr.microsoft.com/playwright:v1.61.0-noble. The image has Chromium and its system libraries, so the apt step is gone.
  • A new step fails the job when @playwright/test in package-lock.json is not the same version as the container. Change both together.
  • npx playwright test --retries=2 --trace=on-first-retry --reporter=list,github. A test that passes only on a retry is reported as flaky, with a trace. playwright.config.ts is unchanged, so a local run behaves as before.
  • The results are uploaded on every run that is not cancelled (before: only on a failure).

nightly.yml (new; does not block a pull request)

  • Runs every day at 19:00 UTC (03:00 in Taipei), and by hand for any branch (Run workflow).
  • Three groups, each on its own runner, so tests with a time budget do not compete for CPU:
    • format: tests/format.test.ts
    • timing: tests/timescale.test.ts, tests/import/limit.test.ts
    • examples: tests/learn.test.ts, tests/constraints.test.ts, tests/components.test.ts, tests/oncall.test.ts, tests/queueing.test.ts
  • npx vitest run --retry=1. A failed scheduled run sends GitHub's default mail.

How it was checked

  • Both files parse as YAML. Every test file and script they name exists.
  • The version check step passes with 1.61.0 and fails with 1.60.0 (run locally).
  • The container tag v1.61.0-noble exists in the registry.
  • The --retries, --trace, --reporter (playwright) and --retry (vitest) flags exist in the installed versions.
  • Not checked: actionlint (not installed). The workflows have not run on GitHub yet; this pull request is their first run.

What to look at in the first runs

  • e2e: the job starts in the container and finishes in about the same time as before (6.5 to 8.5 minutes).
  • nightly does not run on a pull request. After the merge, start it once by hand and note the time of each group.
  • Tests that may fail on a slower runner, all in the nightly job: the CPU budget in tests/timescale.test.ts (15 s), the 120 s wall-clock limit in tests/import/limit.test.ts, the 30 s per-example timeout in tests/format.test.ts. If one fails, raise that limit in its own commit with the measured time.

🤖 Generated with Claude Code

https://claude.ai/code/session_015uxaDTazJv7ozbPd3u5Cub

…d add a nightly job for the slow tests

Why: the full unit tests and the e2e tests move from the developer's machine to GitHub Actions.
CI did not run the accuracy and AI guide checks or 8 of the 10 slow test files, and 3 of the last 9
runs failed in "Install Chromium" (apt, exit code 100), never in a test.

ci.yml, check job (blocks a pull request):
- `node scripts/accuracy.ts --check`: docs/accuracy.md is what the script generates.
- `node scripts/ai-guide.ts --check`: the AI guide is up to date.

ci.yml, e2e job (blocks a pull request):
- Runs in mcr.microsoft.com/playwright:v1.61.0-noble, which has Chromium and its system libraries,
  so there is no apt step. A step fails the job when @playwright/test in package-lock.json is not
  the same version as the container.
- `--retries=2 --trace=on-first-retry --reporter=list,github`: a test that passes only on a retry
  is reported as flaky, with a trace. playwright.config.ts is unchanged, so a local run is too.
- The results are uploaded on every run that is not cancelled, not only on a failure.

nightly.yml (new; does not block a pull request):
- Every day at 19:00 UTC (03:00 in Taipei), and by hand for any branch.
- Three groups, each on its own runner, so tests with a time budget do not compete for CPU:
  format; timescale + import/limit; learn + constraints + components + oncall + queueing.
  With golden and briefs, which the check job runs, this is every file of `npm run test:slow`.
- `--retry=1`; a failure sends GitHub's default mail for a scheduled workflow.

Checked locally: both files parse as YAML; every test file and script they name exists; the version
check step passes with 1.61.0 and fails with 1.60.0; the container tag exists in the registry; the
playwright and vitest flags exist. actionlint was not available. Not run on GitHub yet: the
workflows run only after a push.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uxaDVERbfy9cVbzQ8qB9F
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T06:52:06.613251Z a756e5e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Failed ❌

View logs ↗
a756e5e 2026-11-17T06:51:32.506Z View logs ↗

@minchenlee
minchenlee merged commit fbdd46d into main Nov 17, 2026
2 of 3 checks passed
minchenlee added a commit that referenced this pull request Nov 18, 2026
* chore: add codemod for English source text

* refactor(i18n): make English the source text

Output of scripts/codemod-english-source.ts plus the hand-made parts:
t()/tx() ids, KINDS, examples, scenarios and the 17 practice questions
are English; src/i18n/zh-TW.ts (English id -> Traditional Chinese)
replaces src/i18n/en.ts. Thirteen English texts with two Chinese texts
carry a context mark (Off@@not-used) that t() strips. Old Chinese ids
in saved designs still translate in English mode.

One English text changes: the saas-instagram PostgreSQL label was 61
characters, over the 60-character label limit, and is now
"PostgreSQL (12 shards, replicas in another AZ)" (lead decision).
Golden hashes do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(i18n): warn on missing Chinese, document English source text

i18n:check lists entries without Traditional Chinese and passes; it
still fails on placeholder mismatches and an out-of-date zh-TW.ts.
i18n:strict (nightly on main) fails on missing Chinese. AGENTS.md,
CONTRIBUTING.md and docs/design-system.md say English is the source
text. schema/design.schema.json descriptions are now English (191 lines).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: follow the English source text

Assertions on message ids and raw example data use the English ids;
assertions on zh-TW output stay and go through t(). The text dump used
for the acceptance check runs only with DUMP_OUT set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(i18n): allow English-only contributions with empty Chinese

An example added in English only leaves its zh-TW entry empty, so zh-TW
shows English. Only require Chinese when the catalog has it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the zh-complete check in nightly.yml

PR #2 added nightly.yml with its own schedule and concurrency group, so
ci.yml goes back to main's version and the nightly translation check
becomes a job next to the slow tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(i18n): show a user's own text with @@ as written

t() stripped the @@context from any text, so a scenario named
"壓測@@臺北" showed as "壓測" in both languages. Only a catalog id has a
context mark now; other text stays as written. Catalog lookups use
Object.hasOwn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(oncall): check the English title for words that name the fault

The check only looked at the Traditional Chinese title, so an English
title with no translation ("Database failure") passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(i18n): clearer English for three labels

"Replica reads" -> "Read replica count", "Replica applies write cost" ->
"Replica write-apply cost", "Not enough result" -> "Insufficient data"
(with the three import messages that start with it). The Traditional
Chinese moves with each id and does not change. Regenerated schema/ and
the AI guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(app): the scenario menu finds a preset by its old Chinese name

Built-in scenario names are English now. An old share link or a saved
version still has the Traditional Chinese name, so the menu did not mark
the current preset. presetOf() (src/app/presetName.ts) also matches the
Chinese name in zh-TW.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: remove the one-time codemod and the text-dump test

The English source text conversion is done and checked; copies stay
with the validation data outside the repo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant