fix(riptide): report the reviewer account, stop dropping unpriced rollups - #186
Merged
Merged
Conversation
…lups Two problems found while measuring 3.5 months of riptide data. We post reviews under an ordinary Bitbucket account, so riptide could not tell our comments apart from a human's. Being first to comment on almost every PR within a minute, we drove its code-review pickup-time metric from ~18 minutes to 36 seconds. The rollup now carries reviewer_handle (BITBUCKET_USERNAME) and riptide filters us out at read time. A rollup with no cost was dropped entirely and logged at debug, so a model the gateway cannot price silently withheld the whole PR record - outcome, diff size, tokens and runs included, none of which need a price. The rollup is now always sent, with the cost omitted rather than zeroed, and the unpriced case logs a warning naming the models. Needs riptide with optional total_cost_usd and reviewer_handle support.
A collector predating reviewer_handle rejects every rollup, not just the ones with a new field, and a rejected rollup is never retried.
riptide should not have to assume that a reported account is automation, so we say so explicitly. The handle stays because the review comments reach riptide from Bitbucket, where we are just another user.
riptide now distinguishes bot from service accounts; we are a bot.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two problems found while measuring 3.5 months of riptide data.
We looked like a human reviewer. Reviews are posted under an ordinary Bitbucket account, so riptide could not tell our comments apart from a person's. Being first to comment on nearly every PR within a minute, we pulled its code-review pickup-time metric from a real ~18 minutes down to 36 seconds. The rollup now carries
reviewer_handle(BITBUCKET_USERNAME), which riptide uses to filter our comments out at read time.An unpriced model silently withheld the whole PR record.
emit_pr_completedreturned early whentotal_cost_usdwasNone, logging at debug. Outcome, diff size, tokens and run count need no price, but they were dropped with it, and nothing surfaced that it was happening. The rollup is now always sent with the cost omitted (never zeroed, which would understate spend), and the unpriced path logs a warning naming the models.Deploy riptide first
trick77/riptide#66 must be deployed before this version. The riptide rollup schema forbids unknown fields, so a collector without
reviewer_handlerejects every rollup with 422, not only the ones carrying a new field. Rejected rollups are not retried: the PR is stamped as emitted when it is claimed, so everything closed in that window is lost.Tests: 649 pass locally.