Skip to content

fix: serialize Playwright CI runs against shared external test sites - #1228

Merged
tripodsan merged 4 commits into
mainfrom
pwserial
Aug 11, 2026
Merged

tripodsan merged 4 commits into
mainfrom
pwserial

Conversation

@tripodsan

@tripodsan tripodsan commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Both playwright.yml (Da-Admin) and playwright-hlx.yml (Helix) run e2e tests against fixed, shared external sites (da-sites/da-status and da-testautomation/da-e2e-tests), with no isolation between concurrent CI runs and no concurrency: guard in either workflow.
  • Investigated a failing/"hanging" run (run 31389050902, PR fix: upload pasted base64 images via source.uploadMedia #1227): 27 tests failed, all timing out waiting for the editor (div.ProseMirror) to load after "Create document" — across chromium/firefox/webkit and unrelated spec files, which points to a shared-backend problem rather than isolated test flakiness. gh run list confirmed two other PRs' Helix Playwright jobs were running concurrently against the same shared site at the same time, and the log shows cleanup steps hitting 404s consistent with racing create/delete traffic.
  • Adds a concurrency group (keyed on github.workflow, cancel-in-progress: false) to each workflow so runs targeting the same shared site queue instead of racing each other. Different PRs' runs of the same workflow will now wait their turn rather than hammering the shared backend simultaneously.
  • Tradeoff: CI can take longer end-to-end when multiple PRs are active concurrently, since Playwright runs for the same workflow now serialize repo-wide. This is a deliberate exchange of throughput for reliability against a resource that all runs share.
  • Follow-up fix found while validating: after the concurrency fix, a subsequent run on this PR still failed one test — preview_publish.spec.js "Clicking Preview opens a confirmation dialog" — with locator.click: Test timeout of 30000ms exceeded, blocked by #nx-toast-host's "Notifications" region. Root cause: dismissAlertBanner() (test/e2e/utils/utils.js) only checked once, with a bounded 3s wait, for a dismissible alert near the start of each test. The "public sandbox" org warning banner (shown for da-sites/da-status, the default test org) depends on an async check that can resolve later than that window, so the one-shot check missed it, and it appeared mid-test, overlapping and blocking the later preview-button click. Fixed by arming a background watcher for the rest of the test when the banner isn't visible in the initial window, so a late appearance still gets dismissed.
  • Not included (potential follow-up, not done here): giving each CI run its own namespaced test folder (e.g. embedding the run ID) for further isolation; also noticed but not fixed — a few browser.newPage() calls in edit.spec.js/delete.spec.js/authenticated/collab.spec.js are never closed, and reporter: 'html' has no explicit open: 'never' (harmless under GitHub Actions' automatic CI=true, but worth hardening).

Test plan

  • Verified both workflow YAML files still parse correctly (python3 -c "import yaml; yaml.safe_load(open(f))" for both).
  • Watched a full run on this PR to completion post-concurrency-fix: 149/152 passed, only the one alert-banner-related failure described above (down from 27 failures pre-fix).
  • After merge, confirm via gh run list --workflow=playwright-hlx.yml that overlapping pushes/PRs queue rather than run in parallel.
  • Monitor Playwright CI failure rate over the following days to confirm it drops and no run approaches the 60-minute job timeout.

Both playwright.yml and playwright-hlx.yml run tests against fixed,
shared external sites (da-sites/da-status and
da-testautomation/da-e2e-tests respectively) with no isolation between
concurrent runs. Overlapping CI jobs from different PRs collide on the
same backend, causing document create/cleanup races and timeouts that
look like hangs (e.g. run 31389050902 on #1227: 27 failures, all
timing out waiting for the editor to load, while two other PRs' Helix
jobs were running against the same site concurrently).

Add a concurrency group per workflow so runs queue instead of racing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@aem-code-sync

aem-code-sync Bot commented Aug 10, 2026

Copy link
Copy Markdown

Hello, I'm the AEM Code Sync Bot and I will run some actions to deploy your branch.
In case there are problems, just click the checkbox below to rerun the respective action.

  • Re-sync branch
Commits

dismissAlertBanner only checked once, with a bounded 3s wait, near the
start of each test. Some banners (e.g. the "public sandbox" org
warning on da-sites/da-status) depend on an async check that can
resolve later than that window, so the one-shot check missed them —
the banner then appeared mid-test and sat on top of later controls,
blocking clicks (e.g. "Clicking Preview opens a confirmation dialog"
timing out after 30s trying to click button.preview-button, blocked by
#nx-toast-host's "Notifications" region).

Now, if the banner isn't visible in the initial window, arm a
background watcher for the rest of the test so a late appearance still
gets dismissed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

@kptdobe kptdobe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah... thanks. This explains why it was so hard to reproduce!

@tripodsan
tripodsan merged commit f39d060 into main Aug 11, 2026
4 of 5 checks passed
@tripodsan
tripodsan deleted the pwserial branch August 11, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants