Skip to content

Improve observability of GitHub workflows - #349

Draft
whi-tw wants to merge 2 commits into
mainfrom
observe-workflow-failures
Draft

whi-tw wants to merge 2 commits into
mainfrom
observe-workflow-failures

Conversation

@whi-tw

@whi-tw whi-tw commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this pull request solve?

Trello card: https://trello.com/c/jPeqHHPu/772-improve-the-observability-of-our-github-workflows

There is no view of how often CI fails or slows down over time. This adds a weekly "CI digest" workflow: how often this repo's workflows failed in the last week compared with the previous 12 weeks, which jobs and specs failed, and how long runs take. It calls the shared workflow added in govuk-forms/forms-deploy#2365, where the digest is described in full.

It is a dry run for now. Each run writes the digest to its job summary and nothing is posted to Slack until we have seen a few weeks of output. Going live is a one-line change in forms-deploy, with nothing to change here.

It also sets timeout-minutes on every job that runs on a runner, so a hung job fails fast instead of running for up to six hours. The limits are generous and well above the observed run times, so they only catch hangs. Jobs that only call a reusable workflow cannot set one; their limits come from forms-deploy. Once govuk-forms/forms-deploy#2332 merges, the shared workflow lint fails any job that has none.

Earlier commits on this branch called the shared workflows from a forms-deploy branch, and those runs passed; they can be found in this branch's Actions history. The digest only runs from the default branch, so its first run in Actions will be after merge.

Before merging

Posting to Slack needs the org secret SLACK_WEBHOOK_GOVUK_FORMS_DEPLOYMENTS to be available to this repo. That is only needed once the digest goes live, not to merge this.

Things to consider when reviewing

  • Ensure that you consider the wider context.
  • Is it clear what the code is doing?
  • Do the commit messages explain why the changes were made?
  • Has all relevant documentation been updated?

@whi-tw
whi-tw force-pushed the observe-workflow-failures branch 2 times, most recently from 1033d1a to 1137335 Compare October 20, 2026 13:16
@whi-tw
whi-tw requested a review from theseanything October 20, 2026 13:21
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from 1137335 to 7a9dbbf Compare October 20, 2026 13:29
There is no view of how often CI fails or slows down over time, so this
runs a weekly summary by calling the shared workflow in forms-deploy. It
is a dry run for now: the digest goes to the job summary, not Slack.
A hung job otherwise runs for GitHub's six hour default. Recent runs
finish in under a minute, so 15 minutes only catches hangs.
@whi-tw
whi-tw force-pushed the observe-workflow-failures branch from 7a9dbbf to 957703d Compare October 29, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant