Fix bot detection and deploy-to-commit correlation - #66
Merged
Merged
Conversation
Both found by measuring 3.5 months of production events against the documented metrics. Bot detection matched only the login handle, so a review bot provisioned as an ordinary user account counted as a human reviewer and pulled the DX Core 4 pickup time down to seconds. detect_automation_source now matches the configured authors against the display name too, the parser carries it, and it is stored as bitbucket_events.author_display_name. Noergler can also self-report the account it posts under (reviewer_handle), so an installation does not have to configure the handle at all. Argo CD correlation never worked as documented: image tags are versions, not commit SHAs (0 of 4936 image refs in production carried a SHA), and argocd_events.revision is the GitOps-repo SHA, matching no pipeline row. The contract is now the full image reference: senders report pipeline_events.image_ref, Argo CD stores the same strings in payload->'images', and the join is exact. docs/correlating-deploys-to-commits.md documents it plus a read-time fallback for events collected before senders reported it. Also: PR size comes from noergler rollups, since Bitbucket DC webhooks carry no diff stats and the bitbucket_events columns are always NULL; the noergler setup doc still described the pre-0002 per-run payload.
- Tekton: pass image-ref as a Pipeline param. A finally task consuming an uninitialized task result is skipped, so a failed build would have stopped notifying riptide entirely, losing the rows the build success rate needs. - reviewer_handle is now actually read: the pickup-time query in the README derives a bot_identities set from it, which is what fixes historical rows that were tagged human at ingest. - Declare ix_pipeline_events_image_ref on the model too, so metadata-created schemas match the migration. - Filter the deploy-to-commit join to COMPLETED/SUCCESS runs; a sender that reports the image on every phase would otherwise fan each deploy out. - Reviewer precision: findings on both sides of the ratio. Counting disagreements against PRs can go negative. - Accept an empty image_ref / reviewer_handle as null instead of 422: an unset template param arrives as "", and rejecting it dropped the whole event over an optional field. - Match automation handles case-insensitively; a display name is human-formatted and a case-only miss reproduces the bug being fixed. - total_cost_usd is optional: an unpriced model must not cost the whole PR rollup. Senders omit the cost instead of dropping the event or sending 0. - AGENTS.md: noergler dedup keys were still the pre-0002 per-run ones.
The cost-by-model query unnested models_used but summed per-PR totals, so a two-model PR credited its full spend to both. The rollup carries no per-model split, so the query now reports model involvement and spend separately. The bot-identity filter used NOT IN, which is NULL for a row with no author and would drop those reviewer touches once any reviewer_handle exists.
A reported reviewer account only tells riptide who acted, not what that account is. Senders now declare both: reviewer_handle plus reviewer_is_bot. Read-time bot filtering keys off the declaration, so riptide holds no bot names of its own and a reviewer that posts as a person is not miscounted. The handle stays necessary as the join key: review comments arrive from Bitbucket, where the reviewer is an ordinary user.
A boolean only answered 'not human'. Senders now declare an account kind - bot, service or human - because the two non-human kinds read differently: a bot authors work of its own and its velocity is worth charting, while a service account authors nothing and should simply not appear in human activity. 'agent' was avoided as a name, it already means a build agent here. Pipeline senders can declare their git account the same way, which is what finally removes a CI service account from human metrics. Read-time filtering unions every sender's declarations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both problems found by measuring 3.5 months of production events against the documented metrics.
Bot detection matched only the login handle. A review bot provisioned as an ordinary user account (bot name only in
displayName) counted as a human reviewer and pulled DX Core 4 pickup time down to seconds.detect_automation_sourcenow also matches the configured authors against the display name, the parser carries it, and it is stored asbitbucket_events.author_display_name. Noergler can additionally self-report the account it posts under (reviewer_handle), so an installation does not have to configure the handle by hand.Argo CD correlation never worked as documented. Image tags are versions, not commit SHAs (0 of 4936 image references in production carried one), and
argocd_events.revisionis the GitOps-repo SHA, matching no pipeline row. The contract is now the full image reference: senders reportpipeline_events.image_ref, Argo CD stores the same strings inpayload->'images', and the join is exact.docs/correlating-deploys-to-commits.mddocuments it plus a read-time fallback for events collected before senders reported it.Also: PR size now comes from the noergler rollups (Bitbucket DC webhooks carry no diff stats, so the
bitbucket_eventscolumns are always NULL), and the noergler setup doc still described the pre-0002 per-run payload.Migration 0003 adds three nullable columns:
bitbucket_events.author_display_name,pipeline_events.image_ref,noergler_events.reviewer_handle.Verified: 215 tests pass, ruff + basedpyright clean, migration up/down/up against Postgres 17, and the documented fallback query reproduces the measured 1886 deploy-to-commit links on the real export.