-
Notifications
You must be signed in to change notification settings - Fork 106
Description
Executive Summary
As a Claude Code user reviewing the gh-aw documentation, I found that Claude users can successfully adopt gh-aw, but the documentation has a Copilot-first presentation that creates unnecessary friction. The underlying system is engine-agnostic and well-designed, but the documentation structure suggests Copilot is required or preferred, when in reality all three engines (Copilot, Claude, Codex) are equally supported.
Key Finding: gh-aw is fully functional with Claude, with excellent technical documentation for setup and configuration. However, the onboarding experience creates the misleading impression that Copilot is the primary or default choice, which may discourage Claude users from exploring this tool.
Persona Context
I reviewed this documentation as a developer who:
- ✅ Uses GitHub for version control
- ✅ Uses Claude Code as primary AI assistant
- ❌ Does NOT use GitHub Copilot
- ❌ Does NOT use Copilot CLI
- ❌ Does NOT have Copilot subscription
Question 1: Onboarding Experience
Can a Claude Code user understand and get started with gh-aw?
Answer: Yes, but with initial confusion about engine options.
The Quick Start guide (docs/src/content/docs/setup/quick-start.mdx) includes engine selection in the interactive wizard (gh aw add-wizard), which is excellent. However, the Prerequisites section creates unnecessary barriers:
Current State:
## Prerequisites
Before installing, ensure you have:
- **AI Account** - a [GitHub Copilot](https://github.com/features/copilot) subscription, or an [Anthropic Claude](https://www.anthropic.com/) or [OpenAI Codex]((openai.com/redacted) API keyProblem: Lists Copilot first and uses "subscription" language that implies it's the primary option. A Claude user reading this might think:
- "Do I need Copilot to use this?"
- "Is Claude support experimental or secondary?"
- "Will I get a degraded experience without Copilot?"
Specific Issues Found:
- Issue 1: README.md (lines 1-50) - No mention of Claude or engine options at all. First-time users see no indication this works with Claude
- Issue 2:
docs/src/content/docs/introduction/how-they-work.mdx(line ~20) - States "Workflows support GitHub Copilot (default), Claude by Anthropic, and Codex" - labeling Copilot as "default" suggests it's preferred - Issue 3: Quick Start Prerequisites list Copilot first, creating perception hierarchy
Recommended Fixes:
- Reorder prerequisites to be neutral: "AI Account - Choose one: Anthropic Claude API key, GitHub Copilot subscription, or OpenAI Codex API key"
- Add engine info to README: Add a section showing all three engines with equal emphasis
- Remove "default" language: Say "Workflows support three engines: GitHub Copilot, Claude by Anthropic, and Codex" without labeling any as default
Question 2: Inaccessible Features for Non-Copilot Users
What features or steps don't work without Copilot?
Answer: None. All core features work with Claude.
This is excellent news! After thorough analysis, I found:
Features That Work Without Copilot:
- ✅ All workflow triggers (schedule, workflow_dispatch, issue events, PR events, etc.)
- ✅ All tools (github, bash, edit, playwright, web-fetch, web-search)
- ✅ All MCP servers (engine-agnostic protocol)
- ✅ All safe-outputs (create-issue, add-comment, create-pull-request, etc.)
- ✅ Compilation and CLI commands
- ✅ Network controls and sandboxing
- ✅ Repository initialization and workflow management
Features That Are Copilot-Specific:
- Custom agents (
.github/agents/*.agent.md) - These are Copilot-specific prompts, documented indocs/src/content/docs/reference/copilot-custom-agents.md - That's it! This is the only Copilot-specific feature.
Missing Documentation:
- No clear statement that "everything works with Claude except custom agents"
- No feature comparison table showing engine capabilities
- No guidance on equivalent Claude features for custom agents
Architecture Wins:
- MCP protocol ensures tool compatibility across engines
- Safe-outputs are engine-agnostic
- Network controls work identically for all engines
- Compilation and validation apply to all engines equally
Question 3: Documentation Gaps and Assumptions
Where does the documentation assume Copilot usage?
Answer: Primarily in presentation and language, not functionality.
Copilot-Centric Language Found In:
-
File:
README.md- Issue: Zero mentions of Claude or Codex engines
- Impact: First-time visitors see no indication of engine choice
- Line: Throughout entire file
-
File:
docs/src/content/docs/introduction/how-they-work.mdx- Issue: "Workflows support GitHub Copilot (default), Claude by Anthropic, and Codex"
- Impact: Suggests hierarchy where Copilot is preferred
- Line: ~20
-
File:
docs/src/content/docs/setup/quick-start.mdx- Issue: Prerequisites list Copilot subscription first
- Impact: Creates perception that Copilot is primary engine
- Line: ~30-40
-
File:
docs/src/content/docs/reference/engines.md- Issue: Section titled "Using Copilot CLI" comes first and states "is the default AI engine"
- Impact: Reinforces Copilot-first perception
- Line: ~15
Missing Alternative Instructions:
- No "Which engine should I choose?" guidance
- No comparison of engine capabilities, costs, or trade-offs
- No "Getting Started with Claude" quick path
- No explanation of why someone might prefer Claude over Copilot
Positive Counter-Examples:
- ✅ Authentication for Claude is clearly documented in
engines.md - ✅ Examples exist showing
engine: claudeworkflows - ✅ The
gh aw add-wizardcommand supports engine selection interactively - ✅ Secret setup for ANTHROPIC_API_KEY is well explained
Severity-Categorized Findings
🚫 Critical Blockers (Score: 0/10)
No critical blockers found. Claude users can successfully install, configure, and use gh-aw. The system is fundamentally engine-agnostic.
⚠️ Major Obstacles (Score: 3/10)
Obstacle 1: Quick Start Prerequisites Create False Dependency Impression
Impact: Significant friction in getting started - Claude users may think Copilot is required
Current State:
- **AI Account** - a [GitHub Copilot](https://github.com/features/copilot) subscription, or an [Anthropic Claude](https://www.anthropic.com/) or [OpenAI Codex]((openai.com/redacted) API keyWhy It's Problematic:
- Lists "subscription" for Copilot but "API key" for others, suggesting different tiers
- Copilot appears first, creating perceived hierarchy
- "or" conjunction suggests alternatives rather than equals
Suggested Fix:
- **AI Engine** - Choose one:
- [Anthropic Claude](https://www.anthropic.com/) API key
- [GitHub Copilot](https://github.com/features/copilot) subscription
- [OpenAI Codex]((openai.com/redacted) API keyAffected Files: docs/src/content/docs/setup/quick-start.mdx (line ~35)
Obstacle 2: README Doesn't Mention Engine Options
Impact: First impression suggests Copilot-only tool
Current State: README.md mentions workflows, Actions, AI, but never mentions Claude, Codex, or engine choice
Why It's Problematic:
- README is first thing developers read
- No indication this works with multiple engines
- Missed opportunity to attract non-Copilot users
- Creates assumption this is a Copilot-specific tool
Suggested Fix: Add section after "Overview":
## AI Engines
GitHub Agentic Workflows works with multiple AI engines:
- **Claude by Anthropic** - Full MCP support, excellent reasoning
- **GitHub Copilot** - Integrated GitHub authentication
- **OpenAI Codex** - OpenAI's code model
Choose the engine that fits your needs and existing tools. See [Engines documentation](https://github.github.com/gh-aw/reference/engines/) for setup.Affected Files: README.md (suggest adding after line ~50)
Obstacle 3: "Default Engine" Language Creates Hierarchy
Impact: Reinforces perception that Copilot is preferred or better supported
Current State: Multiple files state "GitHub Copilot (default)" or "Copilot CLI is the default AI engine"
Why It's Problematic:
- Suggests Copilot is the "normal" choice
- Makes Claude seem like an alternative/fallback
- Creates concern about support quality for non-default engines
- No technical reason one engine should be "default"
Suggested Fix: Remove "default" language throughout:
- "Workflows support three AI engines: GitHub Copilot, Claude by Anthropic, and Codex"
- "If no engine is specified in frontmatter, Copilot is used" (technical fact, not value judgment)
Affected Files:
docs/src/content/docs/introduction/how-they-work.mdx(line ~20)docs/src/content/docs/reference/engines.md(line ~15)
💡 Minor Confusion Points (Score: 5/10)
-
Issue 1:
add-wizardcommand shown in Quick Start but not documented in CLI reference (docs/src/content/docs/setup/cli.md) - where did this command come from? -
Issue 2: No engine comparison guidance - when would I choose Claude vs Copilot vs Codex? What are trade-offs? File: All engine docs
-
Issue 3: Example workflow distribution (71 Copilot vs 29 Claude vs 9 Codex) suggests Copilot preference even if unintentional - File:
.github/workflows/ -
Issue 4: Secret naming
COPILOT_GITHUB_TOKENis confusing - this token isn't Copilot-specific, it's a GitHub PAT. Name suggests Copilot connection when it's actually for GitHub API access. File:docs/src/content/docs/setup/quick-start.mdx -
Issue 5: Engines documentation orders Copilot first instead of alphabetical - creates subtle perception bias. File:
docs/src/content/docs/reference/engines.md
Engine Comparison Analysis
Available Engines
Based on my review, gh-aw supports these engines with the following documentation quality:
| Engine | Setup Docs | Examples | Auth Docs | Tool Support | Overall Score |
|---|---|---|---|---|---|
| Copilot | ⭐⭐⭐⭐⭐ Excellent | ⭐⭐⭐⭐⭐ (71 examples) | ⭐⭐⭐⭐⭐ Detailed | ⭐⭐⭐⭐ Custom agents | ⭐⭐⭐⭐⭐ 5/5 |
| Claude | ⭐⭐⭐⭐ Very Good | ⭐⭐⭐⭐ (29 examples) | ⭐⭐⭐⭐⭐ Clear | ⭐⭐⭐⭐⭐ Full MCP | ⭐⭐⭐⭐ 4/5 |
| Codex | ⭐⭐⭐ Good | ⭐⭐⭐ (9 examples) | ⭐⭐⭐⭐ Clear | ⭐⭐⭐⭐⭐ Standard | ⭐⭐⭐ 3/5 |
| Custom | ⭐⭐⭐ Good | ⭐⭐ Limited | ⭐⭐⭐ Basic | ⭐⭐⭐⭐ Standard | ⭐⭐⭐ 3/5 |
Key Observations:
- All engines have clear authentication documentation
- Copilot has most examples (but this may reflect repository history, not capability)
- Claude has excellent MCP support explicitly called out
- Tool support is engine-agnostic for standard tools
- Only Copilot has custom agents feature (Copilot-specific)
Documentation Quality by Engine
Strengths:
- ✅ Technical documentation is engine-neutral and accurate
- ✅ Authentication steps are clear for all engines
- ✅ Tool configuration works identically across engines
- ✅ No features are hidden or undocumented for Claude
Weaknesses:
- ❌ Marketing/onboarding documentation is Copilot-centric
- ❌ No comparison or selection guidance
- ❌ Example distribution favors Copilot
- ❌ "Default" language creates perceived hierarchy
Tool Availability Analysis
Tools Review
Engine-Agnostic Tools: ✅ All tools work with all engines
github:- GitHub API operations via MCPbash:- Shell command executionedit:- File editingplaywright:- Browser automationweb-fetch:- Web content retrievalweb-search:- Web search capabilitiesagentic-workflows:- Workflow introspectioncache-memory:- Persistent memoryrepo-memory:- Repository memory- All MCP servers (by design of MCP protocol)
Engine-Specific Tools:
- Copilot custom agents (
.github/agents/*.agent.md) - Only for Copilot engine- Documented in
docs/src/content/docs/reference/copilot-custom-agents.md - Allows custom prompt templates for Copilot CLI
- No equivalent for Claude (but Claude workflows can use detailed markdown instructions)
- Documented in
Verdict: 99% of tools are engine-agnostic. Only custom agents are Copilot-specific, which is clearly documented.
Authentication Requirements
Current Documentation
Quick Start guide covers authentication for:
- ✅ Copilot - Detailed instructions for
COPILOT_GITHUB_TOKEN(fine-grained PAT with copilot-requests scope) - ✅ Claude - Clear instructions for
ANTHROPIC_API_KEYinengines.md - ✅ Codex - Clear instructions for
OPENAI_API_KEYinengines.md
Authentication Quality Assessment
Excellent aspects:
- Each engine's secret is documented with exact steps
- Links to provider console/settings pages provided
gh aw secrets setcommand shown for all engines- Token requirements (scopes, permissions) clearly stated
Missing for Claude Users:
- No troubleshooting section for Claude API issues
- No mention of Claude API rate limits or costs
- No guidance on Claude model selection (though
model:field exists in extended config)
Secret Names
| Engine | Secret Name | Purpose | Documented |
|---|---|---|---|
| Copilot | COPILOT_GITHUB_TOKEN |
Fine-grained PAT with copilot-requests scope | ✅ Yes |
| Claude | ANTHROPIC_API_KEY |
Anthropic API key from console.anthropic.com | ✅ Yes |
| Codex | OPENAI_API_KEY |
OpenAI API key from platform.openai.com | ✅ Yes |
Confusing naming: COPILOT_GITHUB_TOKEN sounds Copilot-specific but is used for GitHub API access, not Copilot authentication. The name creates confusion about what it's for. Consider renaming to GITHUB_API_TOKEN_FOR_COPILOT or similar.
Example Workflow Analysis
Workflow Count by Engine
Engine: copilot - 71 workflows found
Engine: claude - 29 workflows found
Engine: codex - 9 workflows found
Analysis: Repository heavily favors Copilot examples (71 vs 29 vs 9). This creates perception that:
- Copilot is the "main" engine
- Some workflows might not work with Claude
- Claude support is less mature
Reality: The workflows are functionally identical - just change engine: copilot to engine: claude. The distribution likely reflects repository history and contributor preferences, not technical capability.
Quality of Examples
Copilot Examples:
- Diverse use cases (CI, security, documentation, triage)
- Well-commented and structured
- Show advanced features like custom agents
Claude Examples:
- Equally well-structured and functional
- Include complex workflows like
audit-workflows.mdandclaude-code-user-docs-review.md(this workflow!) - Demonstrate full tool and MCP support
- No functional differences from Copilot workflows
Recommendation: Add more Claude examples, or create a "reference implementations" section showing the same workflow with different engines to demonstrate equivalence.
Recommended Actions
Priority 1: Critical Documentation Fixes
No critical fixes needed - system is fully functional with Claude
Priority 2: Major Improvements
-
Update Quick Start Prerequisites - Rewrite to list engines equally without hierarchy
- File:
docs/src/content/docs/setup/quick-start.mdx(line ~35) - Change: "Choose one: Claude API key, Copilot subscription, or Codex API key"
- File:
-
Add Engine Section to README - Include engines in main README
- File:
README.md(after Overview section, ~line 50) - Add: Section listing all three engines with equal emphasis
- File:
-
Remove "Default" Language - Stop calling Copilot the "default engine"
- Files:
docs/src/content/docs/introduction/how-they-work.mdx,docs/src/content/docs/reference/engines.md - Change: "Workflows support three engines: Copilot, Claude, and Codex" (no default label)
- Files:
Priority 3: Nice-to-Have Enhancements
-
Create Engine Comparison Table - Help users choose the right engine
- New section in
docs/src/content/docs/reference/engines.md - Compare: cost, capabilities, authentication, use cases
- New section in
-
Add "Why Claude?" Guidance - Explain Claude benefits
- Benefits: Full MCP support, excellent reasoning, no GitHub-specific requirements
- When to choose: Complex analysis tasks, non-GitHub integrations, preference for Anthropic
-
Balance Example Workflows - Add more Claude examples
- Convert some Copilot workflows to Claude versions
- Create "reference implementations" showing same workflow with different engines
-
Document add-wizard Command - Missing from CLI reference
- File:
docs/src/content/docs/setup/cli.md - Add: Documentation for
gh aw add-wizardcommand
- File:
-
Consider Secret Naming - Make token names more intuitive
- Current:
COPILOT_GITHUB_TOKEN(confusing - is this for Copilot or GitHub?) - Better:
GITHUB_API_TOKENorGITHUB_PAT_FOR_AI_ENGINE
- Current:
-
Alphabetize Engine Documentation - Remove subtle ordering bias
- File:
docs/src/content/docs/reference/engines.md - Order: Claude, Codex, Copilot (alphabetical) instead of Copilot first
- File:
Positive Findings
What Works Well
Technical Implementation:
- ✅ Engine abstraction is excellent - change one line (
engine: claude) and everything works - ✅ MCP protocol ensures tool compatibility across all engines
- ✅ Safe-outputs are completely engine-agnostic
- ✅ Network controls and sandboxing work identically for all engines
Documentation Quality:
- ✅ Authentication for Claude is clearly documented with exact steps
- ✅
gh aw add-wizardincludes engine selection (excellent UX) - ✅ Multiple Claude example workflows exist and work perfectly
- ✅ Engines documentation (
docs/src/content/docs/reference/engines.md) has dedicated Claude section - ✅ No hidden features or undocumented requirements
Architecture Design:
- ✅ No hard dependencies on Copilot anywhere in the system
- ✅ Compilation and validation apply to all engines equally
- ✅ CLI commands work identically regardless of engine choice
- ✅ Tool allowlisting and security controls are engine-neutral
User Experience:
- ✅ Changing engines is as simple as editing one frontmatter field
- ✅ Secret setup is straightforward with
gh aw secrets setcommand - ✅ Error messages don't assume a specific engine
- ✅ Workflow compilation validates all engines equally
Conclusion
Can Claude Code Users Successfully Adopt gh-aw?
Answer: Yes, with moderate effort to navigate Copilot-centric presentation.
Reasoning:
The gh-aw system is fundamentally engine-agnostic and works excellently with Claude. The technical implementation, tool support, and core functionality are completely engine-neutral. A Claude user can:
- Install gh-aw (no issues)
- Run
gh aw add-wizardand select Claude (works perfectly) - Configure
ANTHROPIC_API_KEYsecret (well documented) - Use all tools, MCP servers, and safe-outputs (100% compatible)
- Create and run workflows (identical experience to Copilot)
However, the documentation presentation creates unnecessary friction:
- Prerequisites list Copilot first, creating false impression of requirement
- README doesn't mention engine options at all
- "Default engine" language suggests Copilot is preferred
- Example distribution (71 Copilot vs 29 Claude) suggests Copilot focus
These are presentation issues, not functional issues. The system works great with Claude, but the documentation makes it harder to discover this fact.
Overall Assessment Score: 7.5/10
Breakdown:
- Clarity for non-Copilot users: 7/10 - Good technical docs, but Copilot-centric presentation
- Claude engine documentation: 8/10 - Authentication and setup are clear, missing comparison guidance
- Alternative approaches provided: 8/10 - Tools and features work equally across engines
- Engine parity: 8/10 - Functional parity is excellent, documentation parity needs improvement
Next Steps
Immediate actions (high impact, low effort):
- Rewrite Quick Start prerequisites to list engines equally
- Add engine options to README (3-4 sentence section)
- Remove "default engine" language from How It Works
Medium-term improvements (high impact, medium effort):
4. Create engine comparison table
5. Add "Choosing an Engine" guidance section
6. Balance example workflow distribution
Long-term enhancements (nice to have):
7. Create reference implementations showing same workflow with different engines
8. Add engine-specific best practices sections
9. Consider more neutral secret naming conventions
Appendix: Files Reviewed
Complete List of Documentation Files Analyzed
Core Documentation
README.md- Main repository READMEdocs/src/content/docs/setup/quick-start.mdx- Quick Start guidedocs/src/content/docs/introduction/how-they-work.mdx- How It Worksdocs/src/content/docs/introduction/architecture.mdx- Security Architecturedocs/src/content/docs/reference/tools.md- Tools referencedocs/src/content/docs/setup/cli.md- CLI commandsdocs/src/content/docs/reference/engines.md- AI Engines documentationdocs/src/content/docs/reference/tokens.md- GitHub Tokens referencedocs/src/content/docs/reference/faq.md- FAQ
Example Workflows Sampled
.github/workflows/audit-workflows.md(Claude).github/workflows/claude-code-user-docs-review.md(Claude).github/workflows/smoke-claude.md(Claude)- Counts: 71 Copilot, 29 Claude, 9 Codex workflows analyzed
Additional Files
.github/aw/directory structure- Various guide documents in
docs/src/content/docs/guides/ - Total documentation files: 125
Report Generated: 2026-02-10
Workflow Run: 21875087522
Repository: github/gh-aw
Engine Used: claude (eating our own dog food! 🐕)
Stored Results: /tmp/gh-aw/cache-memory/claude-code-docs-review-2026-02-10.json
Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.
AI generated by Claude Code User Documentation Review
- expires on Feb 17, 2026, 5:24 PM UTC