Skip to content

Measure lead time per commit, as DORA defines it - #69

Merged
trick77 merged 2 commits into
masterfrom
worktree-dora-lead-time
Sep 8, 2026
Merged

trick77 merged 2 commits into
masterfrom
worktree-dora-lead-time

Conversation

@trick77

@trick77 trick77 commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Lead time was measured as "the newest commit in each release" — it answers how stale the freshest change was, not DORA's "how long from commit to running in production". On the production export the difference is sevenfold, and the proxy is the flattering one:

proxy (newest commit) per commit (DORA)
prod p50 26.7 h 193.6 h (8 days)
prod p90 90.8 h 505.6 h (21 days)
intg p50 1.2 h 4.0 h

The gap between intg and prod is the actual finding: delivery is fast, the wait sits entirely in front of production.

How

Three views, layered so the fragile part is swappable:

  • commit_sightings — expands the commits[] array that push payloads already carry (id, authorTimestamp, parents, author + type) into one row per commit on master. Measured: the array is capped at 5 per push, which bites 0.6 % of master pushes.
  • deploy_commit_ranges — which commit range a deploy shipped, read today from the release-note bump links. This is the only view image_ref replaces, with "between the previous and the current successful deploy of this app" — no release notes, both release paths.
  • lead_time_changes — each commit against the first deploy per environment that carried it, so a release rolling out to four Apps counts each change once.

The SQL lives in riptide_collector/views.py, imported by the migration and by the tests, so the definition that ships is the definition under test.

What counts as a change

Merge commits and service-account commits are classified, not dropped; the documented default excludes both — they are artifacts of shipping rather than changes that shipped. Bots get their own line instead of being blended: Renovate authored 87 % of commits reaching production here, so a blended median (209.3 h) describes the bot's cadence, not the team's.

Verification

  • 242 tests pass, ruff + basedpyright clean. Seven new tests seed a synthetic release through testcontainers and assert the range boundaries, the once-per-change rule across two prod Apps, per-environment separation, and the classification columns.
  • Migration applies, rolls back and re-applies against Postgres 17.
  • Against the loaded export the views reproduce the prototype exactly: prod 984 changes / p50 193.6 h / p90 505.6 h, intg 1 375 / 4.0 h.

Docs: docs/dora-lead-time.md (definition, filters, limits), README lead-time row rewritten, one AGENTS.md line so the proxy does not come back.

Lead time was the newest commit in each release, which answers "how stale was
the freshest change" rather than "how long from commit to running". On the
production export the difference is sevenfold: 26.7 h as a proxy against a real
193.6 h median, and the proxy is the flattering one.

Three views do it properly. commit_sightings expands the commits array that
push payloads already carry - id, author timestamp, parents, author type - into
one row per commit on master. deploy_commit_ranges says which commit range a
deploy shipped, today from the release-note bump links; that is the one view
image_ref will replace, with "between the previous and current deploy of this
app". lead_time_changes maps each commit to the first deploy per environment
that carried it, so a release rolling out to four Apps counts each change once.

The SQL lives in riptide_collector.views so the migration and the tests run the
same text - an untestable metric definition drifts. Merge commits and
service-account commits are classified, not dropped, and the documented default
excludes both: they are artifacts of shipping, not changes that shipped. Bots
get their own line rather than being blended, since they authored 87 percent of
commits reaching production.

Measured against the export: prod human p50 193.6 h / p90 505.6 h over 984
changes, intg p50 4.0 h. The gap between the two is the finding - delivery is
fast, the wait sits in front of production.
- Join the App repo by exact slug instead of LIKE, and widen the slug pattern
  to allow '_' and '.', which are legal and would otherwise yield no ranges at
  all - with LIKE, '_' would also have matched any character.
- Declare the new index on the model too, so autogenerate does not propose
  dropping it.
- Doc query: match bots on login and display name, the same rule the rest of
  riptide uses, and with NOT EXISTS so an unresolved author is kept rather than
  dropped by NULL propagation.
- Coverage query counted bumps, not deploys, which understated the unresolved
  share. Counts distinct deploys now: 365 of 1007 unresolved on the export.
- Write down the two boundary assumptions rather than leaving them implicit:
  boundary SHAs and the GitOps release commit are push tips, and a missing
  boundary costs a whole release window, not one change, because the next
  range starts where this one ended.
- Test a two-commit push, which nothing covered: both commits are attributed
  to the same release and each keeps its own commit timestamp.
- README no longer suggests stratifying by change_type, which these commits
  do not carry.
@trick77
trick77 merged commit b4d99ec into master Sep 8, 2026
3 checks passed
@trick77
trick77 deleted the worktree-dora-lead-time branch September 8, 2026 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant