Skip to content

Improve observability of GitHub workflows - #2436

Draft
whi-tw wants to merge 2 commits into
mainfrom
observe-workflow-failures
Draft

whi-tw wants to merge 2 commits into
mainfrom
observe-workflow-failures

Conversation

@whi-tw

@whi-tw whi-tw commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this pull request solve?

Trello card: https://trello.com/c/jPeqHHPu/772-improve-the-observability-of-our-github-workflows

There is no view of how often CI fails or slows down over time. This adds a weekly "CI digest" workflow: how often this repo's workflows failed in the last week compared with the previous 12 weeks, which jobs and specs failed, and how long runs take. It calls the shared workflow added in govuk-forms/forms-deploy#2365, where the digest is described in full.

It is a dry run for now. Each run writes the digest to its job summary and nothing is posted to Slack until we have seen a few weeks of output. Going live is a one-line change in forms-deploy, with nothing to change here.

It also sets timeout-minutes on every job that runs on a runner, so a hung job fails fast instead of running for up to six hours. The limits are generous and well above the observed run times, so they only catch hangs. Jobs that only call a reusable workflow cannot set one; their limits come from forms-deploy. Once govuk-forms/forms-deploy#2332 merges, the shared workflow lint fails any job that has none.

Earlier commits on this branch called the shared workflows from a forms-deploy branch, and those runs passed; they can be found in this branch's Actions history. The digest only runs from the default branch, so its first run in Actions will be after merge.

Before merging

Posting to Slack needs the org secret SLACK_WEBHOOK_GOVUK_FORMS_DEPLOYMENTS to be available to this repo. That is only needed once the digest goes live, not to merge this.

Things to consider when reviewing

  • Ensure that you consider the wider context.
  • Is it clear what the code is doing?
  • Do the commit messages explain why the changes were made?
  • Has all relevant documentation been updated?

@whi-tw
whi-tw force-pushed the observe-workflow-failures branch 3 times, most recently from 93f7d8a to 95d9960 Compare October 20, 2026 13:16
@whi-tw
whi-tw requested a review from theseanything October 20, 2026 13:21
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from 95d9960 to e9e6402 Compare October 20, 2026 13:29
There is no view of how often CI fails or slows down over time, so this
runs a weekly summary by calling the shared workflow in forms-deploy. It
is a dry run for now: the digest goes to the job summary, not Slack.
Without a timeout a hung job runs for six hours. The limits are
well above normal run times and only exist to catch hangs.
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from e9e6402 to 4533486 Compare October 29, 2026 13:38
@github-actions

Copy link
Copy Markdown
Contributor

🎉 A review copy of this PR has been deployed! It is made of up two components

  1. A review copy of forms-runner
  2. A production copy of forms-admin

It may take 5 minutes or so for the application to be fully deployed and working. If it still isn't ready
after 5 minutes, there may be something wrong with the ECS task. You will need to go to the integration AWS account
to debug, or otherwise ask an infrastructure person.

For the sign in details and more information, see the review apps wiki page.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant