Measure lead time per commit, as DORA defines it

AGENTS.md

@@ -24,6 +24,7 @@ podman-compose up                    # Postgres + migrations + app on :8000
 - **Metrics computed on read.** No aggregation tables, no rollup jobs in v1. Schema additions preserve raw events.
 - **Correlation, in priority order.** Bitbucket↔Pipeline: `commit_sha` (App-repo SHA both sides, deterministic). Argo CD: the **full image reference** — senders report `pipeline_events.image_ref` (`registry/path:tag`), Argo stores the same strings in `payload->'images'`. `argocd_events.revision` is the GitOps-repo SHA (four Apps of one service share one) and matches neither other source. Image **tags are not SHAs** — measured: 0 of 4 936 refs, all semver; never parse a SHA out of a tag. Pre-`image_ref` rows: read-time fallback in `docs/correlating-deploys-to-commits.md`. Never `service_id` or name mappings.
 - **`repo:refs_changed` is ref movement, not developer activity.** Measured: of 16 295 master-ref events ~15 000 were release tooling (maven/gradle release plugins, component-version job, Renovate); the 1 210 human-authored ones were merge commits already counted as `pr:merged`. Read activity and `change_type` off PR events — a `master` push has no branch prefix, so change mix over all events reads 83 % `other`. Never infer intent from an event-type name; check `author` and the commit message.
+- **Lead time is per commit, against the first deploy that carried it** — the `lead_time_changes` view (over `commit_sightings` + `deploy_commit_ranges`). Exclude merge commits and service-account commits; report bots as their own line, never blended (87 % of commits reaching prod were Renovate's). Never quote the newest-commit-per-release shortcut as lead time: measured 26.7 h against a real 193.6 h.
 - **`change_type` on Bitbucket events only.** Don't denormalise onto pipeline / Argo rows; join at read time.
 - **Automation detection is config-last.** Order: configured `automation` authors (matched against login *and* display name, case-insensitive) → acting user's `type == "SERVICE"` from the payload → `*-bot` name shape. Senders also declare themselves (`reviewer_handle` / `actor_handle` + account kind, read-time filter). Only accounts nobody reports get a config entry. `automation` is org-wide, at the config root.
 - **CI events are source-tagged, not source-routed.** Every CI lands in `pipeline_events` via `POST /webhooks/pipeline`, told apart by `source`. No per-CI tables or endpoints. Dedup key `source#pipeline_name#run_id#phase`.

README.md

@@ -87,7 +87,7 @@ the data captured in v1.
 | Metric | How it's computed |
 |---|---|
 | **Deployment frequency** | `COUNT(DISTINCT revision)` of `argocd_events` per `team` / time window where `operation_phase = 'Succeeded' AND environment = 'prod'`. Count revisions, not rows: one release reconciles every Argo App that shares the GitOps revision, so `COUNT(*)` per `app_name` overcounts releases. Group by `app_name` for the per-App drill-down. |
-| **Lead time for changes** | Deploy → build → commit, joined on the image reference: `argocd_events.payload->'images'` contains the full image refs Argo rendered, `pipeline_events.image_ref` is what the build published. From the pipeline row, `commit_sha` gives the App-repo commit; its first sighting in `bitbucket_events` starts the clock, the `argocd_events.occurred_at` of the prod deploy ends it. Stratify by `bitbucket_events.change_type` (feature / hotfix / bugfix / …) to separate hotfix from feature lead time. `argocd_events.revision` is the GitOps-repo SHA and does **not** join to `commit_sha` — see [Correlating deploys back to commits](docs/correlating-deploys-to-commits.md), which also documents the read-time fallback for events collected before senders reported `image_ref`. |
+| **Lead time for changes** | Per commit, not per release: `SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY lead_time) FROM lead_time_changes WHERE environment='prod' AND NOT is_merge AND NOT author_is_service_account`. The `lead_time_changes` view maps every commit that shipped to the **first** deploy that carried it, so a release rolled out to four Apps counts each change once. Exclude merge commits and release tooling, and report bots on their own line — see [Lead time for changes (DORA)](docs/dora-lead-time.md) for why, and for the coverage limits. Group by `team` or `repo_full_name`; `change_type` is not available here, since these commits come from master pushes, which carry no branch prefix. Do **not** substitute the newest commit per release: it answers "how stale was the freshest change" and reads ~7× too fast. |
 | **PR cycle time** | `pullrequest:fulfilled.occurred_at − pullrequest:created.occurred_at` per PR id. |
 | **Time to first review** *(DX Core 4 "code review pickup time")* | Two-part computation per PR — see the SQL block below the table. Clock-start = `COALESCE(pr:ready_for_review, pr:opened)`: PRs opened ready start at `pr:opened`; PRs opened as drafts start at the synthetic `pr:ready_for_review` (emitted by the parser when a `pr:modified` payload carries `previousDraft=true, draft=false`). Engagement = first reviewer touch (`pr:comment:added`, `pr:reviewer:approved`, `pr:reviewer:unapproved`, `pr:reviewer:needs_work`, `pr:reviewer:updated`) where `author != pr_opener AND NOT is_automated AND occurred_at >= clock-start`. The five-event reviewer union covers every touch Bitbucket DC emits (silent approvals, retracted approvals, "needs work" flips, bare reviewer-status changes); the `occurred_at >= clock-start` guard drops early-feedback comments solicited during the draft phase, which would otherwise produce negative pickup times. `NOT is_automated` strips bot comments (noergler / Renovate / etc.) — every review-time bot must be recognisable, otherwise its instant comment drives the metric toward zero. Detection matches the `automation` config block against **both** the login handle (`author`) and the display name (`author_display_name`), because a bot is often provisioned as an ordinary user account whose login says nothing. `is_automated` is decided at ingest, so rows written before a bot was recognised stay marked human: the `non_human_identities` CTE in the query below filters those at read time from the accounts senders declare (`reviewer_handle` + `reviewer_account_kind` on noergler rollups, `actor_handle` + `actor_account_kind` on pipeline events) — riptide stores the declaration, it carries no account names of its own. |
 | **Build success rate** | `pipeline_events` with `phase = 'COMPLETED'` grouped by `status`. Slice by `source` to compare Jenkins vs Tekton, by `pipeline_name` / `team` for ownership. |
@@ -312,4 +312,6 @@ See [`docs/`](docs/) for setup and onboarding guides:
 - [Setup: ArgoCD notification](docs/setup-argocd-notification.md)
 - [Setup: Noergler notification](docs/setup-noergler-notification.md)
 - [Onboarding a team](docs/onboarding-a-team.md)
+- [Lead time for changes (DORA)](docs/dora-lead-time.md)
+- [Correlating deploys back to commits](docs/correlating-deploys-to-commits.md)
 - [OpenShift manifests](openshift/README.md)

docs/dora-lead-time.md

@@ -0,0 +1,110 @@
+# Lead time for changes (DORA), per commit
+
+**How long from a developer committing to that change running in production.**
+One row per change, not per release: DORA's second key metric.
+
+Riptide ships three views for it. Query `lead_time_changes`; the two below it
+exist so the fragile part stays swappable.
+
+| View | One row per | Purpose |
+|---|---|---|
+| `commit_sightings` | commit seen on a master push | expands `payload->'commits'`, with timestamps and classification |
+| `deploy_commit_ranges` | (deploy, app repo) | which commit range a deploy shipped |
+| `lead_time_changes` | (environment, commit) | the change and the **first** deploy that carried it |
+
+## The metric
+
+```sql
+-- Lead time to production, human-authored changes, last 90 days
+SELECT
+    percentile_cont(0.5) WITHIN GROUP (ORDER BY lead_time) AS p50,
+    percentile_cont(0.9) WITHIN GROUP (ORDER BY lead_time) AS p90,
+    count(*) AS changes
+FROM lead_time_changes
+WHERE environment = 'prod'
+  AND NOT is_merge
+  AND NOT author_is_service_account
+  -- Match the login AND the display name, the same rule the rest of riptide
+  -- uses: a bot is often provisioned with a nondescript login. NOT EXISTS
+  -- rather than NOT IN, so a commit with an unresolved author is kept rather
+  -- than silently dropped by NULL propagation.
+  AND NOT EXISTS (
+        SELECT 1 FROM unnest(:automation_handles) AS h(handle)
+        WHERE lower(h.handle) IN (lower(author), lower(author_display_name))
+      )
+  AND first_deployed_at > now() - interval '90 days';
+```
+
+Group by `team`, `repo_full_name`, or `date_trunc('week', first_deployed_at)`
+for the breakdowns. Swap `environment` for the stage you treat as production.
+
+## What counts as a change
+
+The views classify; the query decides. The defaults that make the number mean
+what people think it means:
+
+- **Exclude merge commits** (`NOT is_merge`). A squash-and-merge lands the change
+  once; counting the merge commit too double-counts it.
+- **Exclude release tooling** (`NOT author_is_service_account`, plus your CI
+  account if it is not flagged `SERVICE` by the git host). `[maven-release-plugin]
+  prepare for next development iteration`, `[gradle-release] …` and
+  component-version commits are created *by* the release, so their lead time is
+  near zero and they drag the median down.
+- **Report bots separately, never blended.** Dependency updates are real changes
+  that ship, but in one measured dataset Renovate authored 87 % of all commits
+  reaching production. Blending makes the median describe the bot's cadence
+  (p50 209.3 h) rather than the team's (p50 193.6 h). Two lines, always.
+
+## Reading it
+
+Lower is better, and the split between environments is where the signal is. In
+the dataset this was built on, human changes reached **intg in 4.0 h (p50)** but
+**production in 193.6 h (p50), p90 505.6 h** — the delivery pipeline is fast and
+the wait is entirely in front of production, in release scheduling. A single
+blended number would have hidden that.
+
+Beware the tempting shortcut this replaces: taking the *newest* commit in each
+release and calling that lead time. It answers "how stale was the freshest
+change" and reads 26.7 h on the same data — 7× too flattering.
+
+## Limits, so nobody over-reads the number
+
+- **`authored_at` is the commit's own timestamp**, which a rebase rewrites.
+  That is DORA's "code committed"; `committed_at` sits next to it for comparison.
+- **Coverage is bounded by range resolution.** A deploy whose commit range cannot
+  be resolved contributes nothing — it is absent, never counted as fast. Count
+  deploys, not bumps: one deploy fans out to a row per bumped component.
+
+  ```sql
+  SELECT count(DISTINCT a.id) AS deploys,
+         count(DISTINCT a.id) FILTER (WHERE r.deployed_at IS NULL) AS unresolved
+  FROM argocd_events a
+  LEFT JOIN deploy_commit_ranges r
+         ON r.app_name = a.app_name AND r.deployed_at = a.occurred_at
+  WHERE a.operation_phase = 'Succeeded';
+  ```
+
+- **A missing boundary commit costs a whole release, not one change.** The range
+  needs both endpoint SHAs, so if either is absent — the `commits[]` cap below,
+  or an ingest gap — every change in that window disappears. It does not recover
+  later either: the next release's range starts at *this* release's head, so the
+  window is skipped, not deferred.
+- **Bitbucket caps `commits[]` at 5 per push.** Measured, that bites 0.6 % of
+  master pushes. For changes it keeps it is harmless, but via the point above a
+  dropped commit that happens to be a release boundary costs its whole window.
+- **Boundary commits are assumed to be push tips.** Range membership is decided
+  by push time, so every commit of a push lands on one side of the boundary.
+  That is exact when the boundary SHA is the tip of its push, which is what a
+  release cut normally is. Where it is not, commits sharing that push are
+  attributed to the neighbouring release. Likewise the GitOps release commit
+  must be a push tip to be found at all — a release split across two pushes
+  where Argo reports only the second revision leaves the first unresolvable.
+  Both go away with `image_ref`-based ranges.
+- **Ranges come from release notes today.** `deploy_commit_ranges` reads the
+  compare links a release-note generator writes into the GitOps commit, so a
+  release landed as a direct version bump is invisible to it. That is the view to
+  replace once CI senders report `pipeline_events.image_ref`: the range then
+  becomes "between the previous and the current successful deploy of this app",
+  which needs no release notes at all. See
+  [Correlating deploys back to commits](correlating-deploys-to-commits.md).
+- **Ingestion is forward-only.** The metric covers what riptide has seen.

migrations/versions/0004_lead_time_views.py

@@ -0,0 +1,45 @@
+"""views for per-commit DORA lead time
+
+Lead time was a proxy: one App-repo commit per release (the newest), which
+answers "how stale was the freshest change" rather than DORA's "how long from
+commit to running in production". On real data the difference is 7x — prod p50
+26.7 h as a proxy against 193.6 h per commit — and the proxy is the flattering
+one, so it is replaced rather than kept alongside.
+
+The view SQL lives in `riptide_collector.views` so this migration and the tests
+execute the same definition. The index exists because the range lookups are per
+repo and per time window: without it the metric query does not finish on a
+table of any size.
+
+Revision ID: 0004
+Revises: 0003
+Create Date: 2026-09-08
+
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+
+from riptide_collector.views import CREATE_VIEWS, DROP_VIEWS
+
+revision: str = "0004"
+down_revision: str | None = "0003"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.create_index(
+        "ix_bitbucket_events_repo_branch_occurred",
+        "bitbucket_events",
+        ["repo_full_name", "branch_name", "occurred_at"],
+    )
+    for statement in CREATE_VIEWS:
+        op.execute(statement)
+
+
+def downgrade() -> None:
+    for statement in DROP_VIEWS:
+        op.execute(statement)
+    op.drop_index("ix_bitbucket_events_repo_branch_occurred", table_name="bitbucket_events")

src/riptide_collector/models.py

@@ -29,6 +29,14 @@ class BitbucketEvent(Base):
         Index("ix_bitbucket_events_repo_full_name", "repo_full_name"),
         Index("ix_bitbucket_events_pr_id", "pr_id"),
         Index("ix_bitbucket_events_commit_sha", "commit_sha"),
+        # Serves the lead-time views: repo equality + branch equality +
+        # occurred_at range, which is exactly how a release window is looked up.
+        Index(
+            "ix_bitbucket_events_repo_branch_occurred",
+            "repo_full_name",
+            "branch_name",
+            "occurred_at",
+        ),
         Index(
             "ix_bitbucket_events_jira_keys_gin",
             "jira_keys",