Repository navigation
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the weekly CI digest changes.
No open issues 🎉
What changed in this PR
Adds a reusable weekly CI digest for the Forms repositories, comparing recent workflow failures and run times with the previous 12 weeks. Digests initially appear only in job summaries, with Slack posting disabled.
Changes:
- Collects workflow history, calculates failure rates and timing trends, and summarises failed jobs and RSpec examples.
- Adds scheduled, manual and pull-request execution through a shared workflow.
- Adds fixture-based tests to Shell script CI and documents local use and Slack rollout.
| File | Description |
|---|---|
support/ci-digest/testdata/runs.json |
Supplies workflow histories and edge cases for tests. |
support/ci-digest/testdata/expected.txt |
Defines expected failure-rate and timing output. |
support/ci-digest/testdata/expected-with-reasons.txt |
Defines expected failure explanations and run links. |
support/ci-digest/README.md |
Documents interpretation, local use and rollout. |
support/ci-digest/ci-digest.test.sh |
Checks report output, API failures and query splitting. |
support/ci-digest/ci-digest.sh |
Fetches workflow history and collects failure details. |
support/ci-digest/ci-digest.jq |
Calculates comparisons and formats the digest. |
.github/workflows/shell-ci.yml |
Runs digest tests in existing shell CI. |
.github/workflows/reusable-ci-digest.yml |
Generates job summaries and gates optional Slack posting. |
.github/workflows/ci-digest.yml |
Adds weekly, manual and digest-related PR triggers. |
Review effort: Balanced
|
Warning You are changing the Terraform version. Before you merge this PR, you must apply the |
There was a problem hiding this comment.
🟡 Changes recommended
Active issues can omit retries from weekly failure counts and reasons when they cross the reporting cutoff, while failure links can open a later successful attempt instead of the failed attempt.
2 open issues
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
Earlier re-run attempts with a null workflow name bypass the fallback and cause gsub to fail, aborting the entire digest; the fallback and fixture coverage need to include re-runs.
1 open issue
1 resolved since last review
Review effort: Balanced
We have no view of how often CI fails or slows down over time, and a failed run on main does not stop AWS deploying it. A weekly summary compared with the previous 12 weeks shows the trend without an alert for every failed run. It runs as a dry run for now: the digest goes to the job summary and nothing is posted to Slack until we have seen a few weeks of output. Pull requests that change the digest run it too, with their own version of the script, so changes are tried out before they are merged. Those runs never post to Slack, even once it is live. GitHub lists a workflow run once, as its last attempt, so a run that failed and then passed when re-run would look like a success. The script reads the earlier attempts of re-run runs too and counts each as a run of its own, so flaky runs show up as failures. The logic lives here so each forms repo can call it from a thin scheduled workflow.
What problem does this pull request solve?
Trello card: https://trello.com/c/jPeqHHPu/772-improve-the-observability-of-our-github-workflows
There is no view of how often CI fails or slows down over time. This adds a weekly "CI digest" for each forms repo: how often its workflows failed in the last week compared with the previous 12 weeks, which jobs and specs failed, and how long runs take. The logic is in
support/ci-digestand is called throughreusable-ci-digest.yml. Each repo has a small scheduled workflow that calls it, and the other forms repos have matching PRs.It is a dry run for now. Each run writes the digest to its job summary, and nothing is posted to Slack until we have seen a few weeks of output. To go live, change the default of
dry_runtofalseinreusable-ci-digest.yml; every repo then posts to#govuk-forms-deployment-notificationson its next run. Only scheduled and manually started runs can post, so a pull request run is always a dry run.This is the second layer of a stack: #2332 (timeouts and the lint for them) is below it and #2378 (tests) above it. Review and merge in that order.
A failed run on
mainor in the merge queue does not stop AWS deploying it, so failures are easy to miss. A weekly digest shows the trend without an alert for every failed run.The script only reads GitHub's API, and
support/ci-digest/README.mdexplains how to read a digest and run one locally. A pull request that changes the digest runs it on the pull request, using its own version of the script. The tests are in #2378, stacked on this PR.GitHub lists a workflow run once, as its last attempt, so a run that failed and then passed when re-run would look like a success. The script reads the earlier attempts of re-run runs too and counts each as a run of its own, so flaky runs show up as failures. That costs one extra API call per earlier attempt, and the script stops without a digest if one cannot be read.
Before merging
main)@main, so merge this one before them.Before going live
SLACK_WEBHOOK_GOVUK_FORMS_DEPLOYMENTSsecret exists and is available to the forms-* repos.Things to consider when reviewing
GH_REPO=govuk-forms/forms-deploy support/ci-digest/ci-digest.shto see one.