Skip to content

🔍 Claude Code User Documentation Review - 2026-02-10 #14816

@github-actions

Description

@github-actions

Executive Summary

As a Claude Code user reviewing the gh-aw documentation, I found that Claude users can successfully adopt gh-aw, but the documentation has a Copilot-first presentation that creates unnecessary friction. The underlying system is engine-agnostic and well-designed, but the documentation structure suggests Copilot is required or preferred, when in reality all three engines (Copilot, Claude, Codex) are equally supported.

Key Finding: gh-aw is fully functional with Claude, with excellent technical documentation for setup and configuration. However, the onboarding experience creates the misleading impression that Copilot is the primary or default choice, which may discourage Claude users from exploring this tool.


Persona Context

I reviewed this documentation as a developer who:

  • ✅ Uses GitHub for version control
  • ✅ Uses Claude Code as primary AI assistant
  • ❌ Does NOT use GitHub Copilot
  • ❌ Does NOT use Copilot CLI
  • ❌ Does NOT have Copilot subscription

Question 1: Onboarding Experience

Can a Claude Code user understand and get started with gh-aw?

Answer: Yes, but with initial confusion about engine options.

The Quick Start guide (docs/src/content/docs/setup/quick-start.mdx) includes engine selection in the interactive wizard (gh aw add-wizard), which is excellent. However, the Prerequisites section creates unnecessary barriers:

Current State:

## Prerequisites

Before installing, ensure you have:

- **AI Account** - a [GitHub Copilot](https://github.com/features/copilot) subscription, or an [Anthropic Claude](https://www.anthropic.com/) or [OpenAI Codex]((openai.com/redacted) API key

Problem: Lists Copilot first and uses "subscription" language that implies it's the primary option. A Claude user reading this might think:

  1. "Do I need Copilot to use this?"
  2. "Is Claude support experimental or secondary?"
  3. "Will I get a degraded experience without Copilot?"

Specific Issues Found:

  • Issue 1: README.md (lines 1-50) - No mention of Claude or engine options at all. First-time users see no indication this works with Claude
  • Issue 2: docs/src/content/docs/introduction/how-they-work.mdx (line ~20) - States "Workflows support GitHub Copilot (default), Claude by Anthropic, and Codex" - labeling Copilot as "default" suggests it's preferred
  • Issue 3: Quick Start Prerequisites list Copilot first, creating perception hierarchy

Recommended Fixes:

  1. Reorder prerequisites to be neutral: "AI Account - Choose one: Anthropic Claude API key, GitHub Copilot subscription, or OpenAI Codex API key"
  2. Add engine info to README: Add a section showing all three engines with equal emphasis
  3. Remove "default" language: Say "Workflows support three engines: GitHub Copilot, Claude by Anthropic, and Codex" without labeling any as default

Question 2: Inaccessible Features for Non-Copilot Users

What features or steps don't work without Copilot?

Answer: None. All core features work with Claude.

This is excellent news! After thorough analysis, I found:

Features That Work Without Copilot:

  • ✅ All workflow triggers (schedule, workflow_dispatch, issue events, PR events, etc.)
  • ✅ All tools (github, bash, edit, playwright, web-fetch, web-search)
  • ✅ All MCP servers (engine-agnostic protocol)
  • ✅ All safe-outputs (create-issue, add-comment, create-pull-request, etc.)
  • ✅ Compilation and CLI commands
  • ✅ Network controls and sandboxing
  • ✅ Repository initialization and workflow management

Features That Are Copilot-Specific:

  • Custom agents (.github/agents/*.agent.md) - These are Copilot-specific prompts, documented in docs/src/content/docs/reference/copilot-custom-agents.md
  • That's it! This is the only Copilot-specific feature.

Missing Documentation:

  • No clear statement that "everything works with Claude except custom agents"
  • No feature comparison table showing engine capabilities
  • No guidance on equivalent Claude features for custom agents

Architecture Wins:

  • MCP protocol ensures tool compatibility across engines
  • Safe-outputs are engine-agnostic
  • Network controls work identically for all engines
  • Compilation and validation apply to all engines equally

Question 3: Documentation Gaps and Assumptions

Where does the documentation assume Copilot usage?

Answer: Primarily in presentation and language, not functionality.

Copilot-Centric Language Found In:

  1. File: README.md

    • Issue: Zero mentions of Claude or Codex engines
    • Impact: First-time visitors see no indication of engine choice
    • Line: Throughout entire file
  2. File: docs/src/content/docs/introduction/how-they-work.mdx

    • Issue: "Workflows support GitHub Copilot (default), Claude by Anthropic, and Codex"
    • Impact: Suggests hierarchy where Copilot is preferred
    • Line: ~20
  3. File: docs/src/content/docs/setup/quick-start.mdx

    • Issue: Prerequisites list Copilot subscription first
    • Impact: Creates perception that Copilot is primary engine
    • Line: ~30-40
  4. File: docs/src/content/docs/reference/engines.md

    • Issue: Section titled "Using Copilot CLI" comes first and states "is the default AI engine"
    • Impact: Reinforces Copilot-first perception
    • Line: ~15

Missing Alternative Instructions:

  • No "Which engine should I choose?" guidance
  • No comparison of engine capabilities, costs, or trade-offs
  • No "Getting Started with Claude" quick path
  • No explanation of why someone might prefer Claude over Copilot

Positive Counter-Examples:

  • ✅ Authentication for Claude is clearly documented in engines.md
  • ✅ Examples exist showing engine: claude workflows
  • ✅ The gh aw add-wizard command supports engine selection interactively
  • ✅ Secret setup for ANTHROPIC_API_KEY is well explained

Severity-Categorized Findings

🚫 Critical Blockers (Score: 0/10)

No critical blockers found. Claude users can successfully install, configure, and use gh-aw. The system is fundamentally engine-agnostic.


⚠️ Major Obstacles (Score: 3/10)

Obstacle 1: Quick Start Prerequisites Create False Dependency Impression

Impact: Significant friction in getting started - Claude users may think Copilot is required

Current State:

- **AI Account** - a [GitHub Copilot](https://github.com/features/copilot) subscription, or an [Anthropic Claude](https://www.anthropic.com/) or [OpenAI Codex]((openai.com/redacted) API key

Why It's Problematic:

  • Lists "subscription" for Copilot but "API key" for others, suggesting different tiers
  • Copilot appears first, creating perceived hierarchy
  • "or" conjunction suggests alternatives rather than equals

Suggested Fix:

- **AI Engine** - Choose one:
  - [Anthropic Claude](https://www.anthropic.com/) API key
  - [GitHub Copilot](https://github.com/features/copilot) subscription
  - [OpenAI Codex]((openai.com/redacted) API key

Affected Files: docs/src/content/docs/setup/quick-start.mdx (line ~35)

Obstacle 2: README Doesn't Mention Engine Options

Impact: First impression suggests Copilot-only tool

Current State: README.md mentions workflows, Actions, AI, but never mentions Claude, Codex, or engine choice

Why It's Problematic:

  • README is first thing developers read
  • No indication this works with multiple engines
  • Missed opportunity to attract non-Copilot users
  • Creates assumption this is a Copilot-specific tool

Suggested Fix: Add section after "Overview":

## AI Engines

GitHub Agentic Workflows works with multiple AI engines:
- **Claude by Anthropic** - Full MCP support, excellent reasoning
- **GitHub Copilot** - Integrated GitHub authentication
- **OpenAI Codex** - OpenAI's code model

Choose the engine that fits your needs and existing tools. See [Engines documentation](https://github.github.com/gh-aw/reference/engines/) for setup.

Affected Files: README.md (suggest adding after line ~50)

Obstacle 3: "Default Engine" Language Creates Hierarchy

Impact: Reinforces perception that Copilot is preferred or better supported

Current State: Multiple files state "GitHub Copilot (default)" or "Copilot CLI is the default AI engine"

Why It's Problematic:

  • Suggests Copilot is the "normal" choice
  • Makes Claude seem like an alternative/fallback
  • Creates concern about support quality for non-default engines
  • No technical reason one engine should be "default"

Suggested Fix: Remove "default" language throughout:

  • "Workflows support three AI engines: GitHub Copilot, Claude by Anthropic, and Codex"
  • "If no engine is specified in frontmatter, Copilot is used" (technical fact, not value judgment)

Affected Files:

  • docs/src/content/docs/introduction/how-they-work.mdx (line ~20)
  • docs/src/content/docs/reference/engines.md (line ~15)

💡 Minor Confusion Points (Score: 5/10)

  • Issue 1: add-wizard command shown in Quick Start but not documented in CLI reference (docs/src/content/docs/setup/cli.md) - where did this command come from?

  • Issue 2: No engine comparison guidance - when would I choose Claude vs Copilot vs Codex? What are trade-offs? File: All engine docs

  • Issue 3: Example workflow distribution (71 Copilot vs 29 Claude vs 9 Codex) suggests Copilot preference even if unintentional - File: .github/workflows/

  • Issue 4: Secret naming COPILOT_GITHUB_TOKEN is confusing - this token isn't Copilot-specific, it's a GitHub PAT. Name suggests Copilot connection when it's actually for GitHub API access. File: docs/src/content/docs/setup/quick-start.mdx

  • Issue 5: Engines documentation orders Copilot first instead of alphabetical - creates subtle perception bias. File: docs/src/content/docs/reference/engines.md


Engine Comparison Analysis

Available Engines

Based on my review, gh-aw supports these engines with the following documentation quality:

Engine Setup Docs Examples Auth Docs Tool Support Overall Score
Copilot ⭐⭐⭐⭐⭐ Excellent ⭐⭐⭐⭐⭐ (71 examples) ⭐⭐⭐⭐⭐ Detailed ⭐⭐⭐⭐ Custom agents ⭐⭐⭐⭐⭐ 5/5
Claude ⭐⭐⭐⭐ Very Good ⭐⭐⭐⭐ (29 examples) ⭐⭐⭐⭐⭐ Clear ⭐⭐⭐⭐⭐ Full MCP ⭐⭐⭐⭐ 4/5
Codex ⭐⭐⭐ Good ⭐⭐⭐ (9 examples) ⭐⭐⭐⭐ Clear ⭐⭐⭐⭐⭐ Standard ⭐⭐⭐ 3/5
Custom ⭐⭐⭐ Good ⭐⭐ Limited ⭐⭐⭐ Basic ⭐⭐⭐⭐ Standard ⭐⭐⭐ 3/5

Key Observations:

  • All engines have clear authentication documentation
  • Copilot has most examples (but this may reflect repository history, not capability)
  • Claude has excellent MCP support explicitly called out
  • Tool support is engine-agnostic for standard tools
  • Only Copilot has custom agents feature (Copilot-specific)

Documentation Quality by Engine

Strengths:

  • ✅ Technical documentation is engine-neutral and accurate
  • ✅ Authentication steps are clear for all engines
  • ✅ Tool configuration works identically across engines
  • ✅ No features are hidden or undocumented for Claude

Weaknesses:

  • ❌ Marketing/onboarding documentation is Copilot-centric
  • ❌ No comparison or selection guidance
  • ❌ Example distribution favors Copilot
  • ❌ "Default" language creates perceived hierarchy

Tool Availability Analysis

Tools Review

Engine-Agnostic Tools: ✅ All tools work with all engines

  • github: - GitHub API operations via MCP
  • bash: - Shell command execution
  • edit: - File editing
  • playwright: - Browser automation
  • web-fetch: - Web content retrieval
  • web-search: - Web search capabilities
  • agentic-workflows: - Workflow introspection
  • cache-memory: - Persistent memory
  • repo-memory: - Repository memory
  • All MCP servers (by design of MCP protocol)

Engine-Specific Tools:

  • Copilot custom agents (.github/agents/*.agent.md) - Only for Copilot engine
    • Documented in docs/src/content/docs/reference/copilot-custom-agents.md
    • Allows custom prompt templates for Copilot CLI
    • No equivalent for Claude (but Claude workflows can use detailed markdown instructions)

Verdict: 99% of tools are engine-agnostic. Only custom agents are Copilot-specific, which is clearly documented.


Authentication Requirements

Current Documentation

Quick Start guide covers authentication for:

  • Copilot - Detailed instructions for COPILOT_GITHUB_TOKEN (fine-grained PAT with copilot-requests scope)
  • Claude - Clear instructions for ANTHROPIC_API_KEY in engines.md
  • Codex - Clear instructions for OPENAI_API_KEY in engines.md

Authentication Quality Assessment

Excellent aspects:

  • Each engine's secret is documented with exact steps
  • Links to provider console/settings pages provided
  • gh aw secrets set command shown for all engines
  • Token requirements (scopes, permissions) clearly stated

Missing for Claude Users:

  • No troubleshooting section for Claude API issues
  • No mention of Claude API rate limits or costs
  • No guidance on Claude model selection (though model: field exists in extended config)

Secret Names

Engine Secret Name Purpose Documented
Copilot COPILOT_GITHUB_TOKEN Fine-grained PAT with copilot-requests scope ✅ Yes
Claude ANTHROPIC_API_KEY Anthropic API key from console.anthropic.com ✅ Yes
Codex OPENAI_API_KEY OpenAI API key from platform.openai.com ✅ Yes

Confusing naming: COPILOT_GITHUB_TOKEN sounds Copilot-specific but is used for GitHub API access, not Copilot authentication. The name creates confusion about what it's for. Consider renaming to GITHUB_API_TOKEN_FOR_COPILOT or similar.


Example Workflow Analysis

Workflow Count by Engine

Engine: copilot - 71 workflows found
Engine: claude  - 29 workflows found  
Engine: codex   - 9 workflows found

Analysis: Repository heavily favors Copilot examples (71 vs 29 vs 9). This creates perception that:

  1. Copilot is the "main" engine
  2. Some workflows might not work with Claude
  3. Claude support is less mature

Reality: The workflows are functionally identical - just change engine: copilot to engine: claude. The distribution likely reflects repository history and contributor preferences, not technical capability.

Quality of Examples

Copilot Examples:

  • Diverse use cases (CI, security, documentation, triage)
  • Well-commented and structured
  • Show advanced features like custom agents

Claude Examples:

  • Equally well-structured and functional
  • Include complex workflows like audit-workflows.md and claude-code-user-docs-review.md (this workflow!)
  • Demonstrate full tool and MCP support
  • No functional differences from Copilot workflows

Recommendation: Add more Claude examples, or create a "reference implementations" section showing the same workflow with different engines to demonstrate equivalence.


Recommended Actions

Priority 1: Critical Documentation Fixes

No critical fixes needed - system is fully functional with Claude


Priority 2: Major Improvements

  1. Update Quick Start Prerequisites - Rewrite to list engines equally without hierarchy

    • File: docs/src/content/docs/setup/quick-start.mdx (line ~35)
    • Change: "Choose one: Claude API key, Copilot subscription, or Codex API key"
  2. Add Engine Section to README - Include engines in main README

    • File: README.md (after Overview section, ~line 50)
    • Add: Section listing all three engines with equal emphasis
  3. Remove "Default" Language - Stop calling Copilot the "default engine"

    • Files: docs/src/content/docs/introduction/how-they-work.mdx, docs/src/content/docs/reference/engines.md
    • Change: "Workflows support three engines: Copilot, Claude, and Codex" (no default label)

Priority 3: Nice-to-Have Enhancements

  1. Create Engine Comparison Table - Help users choose the right engine

    • New section in docs/src/content/docs/reference/engines.md
    • Compare: cost, capabilities, authentication, use cases
  2. Add "Why Claude?" Guidance - Explain Claude benefits

    • Benefits: Full MCP support, excellent reasoning, no GitHub-specific requirements
    • When to choose: Complex analysis tasks, non-GitHub integrations, preference for Anthropic
  3. Balance Example Workflows - Add more Claude examples

    • Convert some Copilot workflows to Claude versions
    • Create "reference implementations" showing same workflow with different engines
  4. Document add-wizard Command - Missing from CLI reference

    • File: docs/src/content/docs/setup/cli.md
    • Add: Documentation for gh aw add-wizard command
  5. Consider Secret Naming - Make token names more intuitive

    • Current: COPILOT_GITHUB_TOKEN (confusing - is this for Copilot or GitHub?)
    • Better: GITHUB_API_TOKEN or GITHUB_PAT_FOR_AI_ENGINE
  6. Alphabetize Engine Documentation - Remove subtle ordering bias

    • File: docs/src/content/docs/reference/engines.md
    • Order: Claude, Codex, Copilot (alphabetical) instead of Copilot first

Positive Findings

What Works Well

Technical Implementation:

  • ✅ Engine abstraction is excellent - change one line (engine: claude) and everything works
  • ✅ MCP protocol ensures tool compatibility across all engines
  • ✅ Safe-outputs are completely engine-agnostic
  • ✅ Network controls and sandboxing work identically for all engines

Documentation Quality:

  • ✅ Authentication for Claude is clearly documented with exact steps
  • gh aw add-wizard includes engine selection (excellent UX)
  • ✅ Multiple Claude example workflows exist and work perfectly
  • ✅ Engines documentation (docs/src/content/docs/reference/engines.md) has dedicated Claude section
  • ✅ No hidden features or undocumented requirements

Architecture Design:

  • ✅ No hard dependencies on Copilot anywhere in the system
  • ✅ Compilation and validation apply to all engines equally
  • ✅ CLI commands work identically regardless of engine choice
  • ✅ Tool allowlisting and security controls are engine-neutral

User Experience:

  • ✅ Changing engines is as simple as editing one frontmatter field
  • ✅ Secret setup is straightforward with gh aw secrets set command
  • ✅ Error messages don't assume a specific engine
  • ✅ Workflow compilation validates all engines equally

Conclusion

Can Claude Code Users Successfully Adopt gh-aw?

Answer: Yes, with moderate effort to navigate Copilot-centric presentation.

Reasoning:

The gh-aw system is fundamentally engine-agnostic and works excellently with Claude. The technical implementation, tool support, and core functionality are completely engine-neutral. A Claude user can:

  1. Install gh-aw (no issues)
  2. Run gh aw add-wizard and select Claude (works perfectly)
  3. Configure ANTHROPIC_API_KEY secret (well documented)
  4. Use all tools, MCP servers, and safe-outputs (100% compatible)
  5. Create and run workflows (identical experience to Copilot)

However, the documentation presentation creates unnecessary friction:

  • Prerequisites list Copilot first, creating false impression of requirement
  • README doesn't mention engine options at all
  • "Default engine" language suggests Copilot is preferred
  • Example distribution (71 Copilot vs 29 Claude) suggests Copilot focus

These are presentation issues, not functional issues. The system works great with Claude, but the documentation makes it harder to discover this fact.

Overall Assessment Score: 7.5/10

Breakdown:

  • Clarity for non-Copilot users: 7/10 - Good technical docs, but Copilot-centric presentation
  • Claude engine documentation: 8/10 - Authentication and setup are clear, missing comparison guidance
  • Alternative approaches provided: 8/10 - Tools and features work equally across engines
  • Engine parity: 8/10 - Functional parity is excellent, documentation parity needs improvement

Next Steps

Immediate actions (high impact, low effort):

  1. Rewrite Quick Start prerequisites to list engines equally
  2. Add engine options to README (3-4 sentence section)
  3. Remove "default engine" language from How It Works

Medium-term improvements (high impact, medium effort):
4. Create engine comparison table
5. Add "Choosing an Engine" guidance section
6. Balance example workflow distribution

Long-term enhancements (nice to have):
7. Create reference implementations showing same workflow with different engines
8. Add engine-specific best practices sections
9. Consider more neutral secret naming conventions


Appendix: Files Reviewed

Complete List of Documentation Files Analyzed

Core Documentation

  • README.md - Main repository README
  • docs/src/content/docs/setup/quick-start.mdx - Quick Start guide
  • docs/src/content/docs/introduction/how-they-work.mdx - How It Works
  • docs/src/content/docs/introduction/architecture.mdx - Security Architecture
  • docs/src/content/docs/reference/tools.md - Tools reference
  • docs/src/content/docs/setup/cli.md - CLI commands
  • docs/src/content/docs/reference/engines.md - AI Engines documentation
  • docs/src/content/docs/reference/tokens.md - GitHub Tokens reference
  • docs/src/content/docs/reference/faq.md - FAQ

Example Workflows Sampled

  • .github/workflows/audit-workflows.md (Claude)
  • .github/workflows/claude-code-user-docs-review.md (Claude)
  • .github/workflows/smoke-claude.md (Claude)
  • Counts: 71 Copilot, 29 Claude, 9 Codex workflows analyzed

Additional Files

  • .github/aw/ directory structure
  • Various guide documents in docs/src/content/docs/guides/
  • Total documentation files: 125

Report Generated: 2026-02-10
Workflow Run: 21875087522
Repository: github/gh-aw
Engine Used: claude (eating our own dog food! 🐕)
Stored Results: /tmp/gh-aw/cache-memory/claude-code-docs-review-2026-02-10.json


Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

AI generated by Claude Code User Documentation Review

  • expires on Feb 17, 2026, 5:24 PM UTC

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions