Skip to content

Add DeputyDev v1 offline benchmark results - #59

Open
ankit27755 wants to merge 1 commit into
withmartian:mainfrom
ankit27755:add-deputydev-v1-offline-results
Open

Add DeputyDev v1 offline benchmark results#59
ankit27755 wants to merge 1 commit into
withmartian:mainfrom
ankit27755:add-deputydev-v1-offline-results

Conversation

@ankit27755

Copy link
Copy Markdown

Adds DeputyDev v1 to the 50-PR offline code-review benchmark, including GPT-5.5 judged results.

What is included

  • one deputydev-v1 review entry for each of the 50 benchmark tasks
  • 267 raw review comments
  • 224 extracted candidates
  • GPT-5.5 candidates, dedup groups, and evaluations
  • DeputyDev registration in the evaluated-tools list and dashboard
  • support for collecting reviews posted through a human GitHub account
  • a focused downloader regression test

GPT-5.5 result

  • TP: 74
  • FP: 145
  • FN: 63
  • micro precision: 33.79%
  • micro recall: 54.01%
  • skipped evaluations: 0
  • judge errors: 0

Validation

  • exactly 50 DeputyDev v1 benchmark reviews
  • exactly 50 candidate and evaluation entries
  • 43 dedup-group entries
  • only deputydev-v1 is present as the tool key in the GPT-5.5 result files
  • downloader tests: 8 passed
  • focused Ruff checks passed
  • generated dashboard includes the gpt-5.5 model and DeputyDev v1

The source clone repository names and PR URLs retain their original internal run identity for provenance, while the submitted public benchmark tool identity is deputydev-v1.

@ashleyzhang01

ashleyzhang01 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

hey @ankit27755, can you provide the Github org of the repos/PRs used for your run of the benchmark?

@ankit27755

Copy link
Copy Markdown
Author

hey @ankit27755, can you provide the Github org of the repos/PRs used for your run of the benchmark?

@ankit27755 ankit27755 closed this Aug 30, 2026
@ankit27755 ankit27755 reopened this Aug 30, 2026
@ankit27755

Copy link
Copy Markdown
Author

hey @ankit27755, can you provide the Github org of the repos/PRs used for your run of the benchmark?

Hey @ashleyzhang01 :
GitHub organization: https://github.com/dd-benchmarking
The organization also contains repositories from earlier internal benchmark runs. The 50 repositories used for the submitted DeputyDev V5 result are specifically those matching:

__deputydev-v5__PR__20260819
You can filter them here:
https://github.com/orgs/dd-benchmarking/repositories?q=deputydev-v5

Each recreated benchmark PR is PR #1 in its corresponding repository.
Example:
dd-benchmarking/grafana__grafana__deputydev-v5__PR80329__20260819#1

Repositories containing identifiers such as deputydev-v2, deputydev-v2-gpt-5-6, or other run dates were from previous experiments and were not used for the submitted V5 score. I can also provide the exact list of 50 PR URLs if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants