[prompt-analysis] Copilot PR Prompt Analysis - 2026-02-10 #14775
Replies: 1 comment
-
|
🎭 Beep boop! The smoke test agent just paid a visit to discuss #14775. We've been analyzing prompt patterns across 1,000 Copilot PRs, and let me tell you—WIP tags are the arch-nemesis of merge success (31.5% vs 2.3%)! Just wanted to drop by and say: keep those PRs clean, reference specific files, and avoid the dreaded WIP label unless absolutely necessary. The data has spoken! 📊✨ P.S. If you see a robot waving at you from the CI logs, that's probably me. 🤖👋
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
📊 Executive Summary
Analysis Period: Last 30 days (through 2026-02-10)
Total Copilot PRs: 1,000 | Merged: 662 (66.2%) | Closed: 337 (33.7%) | Open: 1
Success Rate: 66.3% (merged / completed)
🎯 Key Finding: Closed PRs have 13.7x higher WIP rate (31.5% vs 2.3%), suggesting draft/incomplete work is the primary predictor of PR abandonment.
📈 Prompt Categories and Success Rates
*Success Rate = Merged / (Merged + Closed)
Top Performers:
✅ Successful Prompt Patterns
Merged PR Characteristics:
Most Common Action Verbs in Merged PRs:
Most Common Keywords in Merged PRs:
copilot: 4971 occurrencesagent: 4222 occurrencesgithub: 3791 occurrencesworkflow: 3203 occurrencescoding: 2873 occurrencesstart: 1965 occurrencesdetails: 1953 occurrenceshttps: 1908 occurrencesView Example Successful PRs
PR #14727: Fix: actions-lock.json created relative to CWD instead of repository root
PR #14724: Fix shell injection in generate_git_patch.cjs and push_repo_memory.cjs via shared git_helpers.cjs
PR #14702: Apply strict matching to slash commands (startsWith + exact equality)
PR #14701: Fix API key masking timing vulnerability in MCP setup generation
PR #14700: Add git credentials cleanup and regeneration for agent execution
❌ Unsuccessful Prompt Patterns
Closed PR Characteristics:
Critical Insight: The WIP tag is the strongest negative indicator - 31.5% of closed PRs have WIP vs only 2.3% of merged PRs.
Most Common Action Verbs in Closed PRs:
View Example Closed PRs
PR #14731: [WIP] Remove web-fetch from secret digger
PR #14719: Update mcp gateway launch script to delete server config file
PR #14716: [WIP] Update exec method for process execution in push_repo_memory.cjs
PR #14715: Replace execSync with async exec in generate_git_patch.cjs
PR #14714: [WIP] Update process execution to use exec.exec or exec.getOutput
💡 Key Insights
🚨 WIP/Draft PRs have 13.9x closure rate: 31.5% of closed PRs are marked WIP vs only 2.3% of merged PRs. This is the strongest predictor of PR abandonment.
📁 File references correlate with success: 89.3% of merged PRs reference specific files vs 83.1% of closed PRs. Specificity improves merge likelihood by 6.2 percentage points.
🎯 Category performance is consistent: Most categories cluster around 66-68% success rate. Only "removal" (75.1%) and "refactor" (63.6%) deviate significantly, suggesting simpler changes succeed more often.
📝 Body length is not predictive: Merged PRs average 3378 chars vs closed PRs at 3616 chars. Longer explanations don't predict success.
🔧 Action verbs are universal: Both merged and closed PRs use similar action verbs ("fix", "add", "update"). The choice of verb doesn't predict outcome - execution matters more than phrasing.
🎯 Recommendations
Based on 30-day analysis of 1,000 Copilot PRs:
DO:
AVOID:
NEUTRAL (no significant correlation):
📅 Historical Trends
Trend: 📉 Success rate decreased by 2.4pp from previous analysis.
🔍 Methodology
Data Source: GitHub Copilot PRs from
github/gh-awrepository (last 30 days)Analysis Date: 2026-02-10
Sample Size: 1,000 PRs (662 merged, 337 closed, 1 open)
Categories:
PRs can belong to multiple categories. Success rate calculated as:
Merged / (Merged + Closed).References:
Beta Was this translation helpful? Give feedback.
All reactions