Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
413723c
fix: resolve crashes in debug task, LiveView compat, and MCP timeline
lessthanseventy Mar 23, 2026
9f6024c
docs: add axe-core migration design
lessthanseventy Mar 23, 2026
4457e00
docs: add axe-core migration implementation plan
lessthanseventy Mar 23, 2026
a189bdc
feat: add axe-runner.js using Playwright + axe-core
lessthanseventy Mar 23, 2026
21002eb
feat: add AxeRunner Elixir wrapper for axe-core
lessthanseventy Mar 23, 2026
1211dbb
feat: swap mix excessibility from Pa11y to axe-core
lessthanseventy Mar 23, 2026
2007112
feat: add mix excessibility.snapshots for snapshot management
lessthanseventy Mar 23, 2026
d11f723
feat: add mix excessibility.check for arbitrary URL checking
lessthanseventy Mar 23, 2026
fcbbde3
chore: remove ChromicPDF dependency, screenshots now via Playwright
lessthanseventy Mar 23, 2026
e316da6
chore: remove Pa11y, installer now sets up Playwright + axe-core
lessthanseventy Mar 23, 2026
38e5dbb
chore: remove 8 MCP tools, 9 prompts, 2 resources
lessthanseventy Mar 23, 2026
a69b1ba
feat: add simplified MCP tools (a11y_check, debug)
lessthanseventy Mar 23, 2026
fe45e27
docs: update all references from Pa11y/ChromicPDF to axe-core/Playwright
lessthanseventy Mar 23, 2026
cf776c3
chore: final cleanup — formatting, screenshot support via Playwright
lessthanseventy Mar 23, 2026
921f0b0
feat: improve axe-runner reliability for remote URLs
lessthanseventy Mar 23, 2026
b38a899
chore: bump version to 0.11.0
lessthanseventy Mar 23, 2026
0ad6c4e
fix: exclude node_modules from hex package
lessthanseventy Mar 23, 2026
4f6d359
fix: remove broken doc links in README
lessthanseventy Mar 23, 2026
34e3cdf
feat: improve curl fallback with browser-like headers
lessthanseventy Mar 23, 2026
f8687c7
docs: add MCP elicitation + auto-check design doc
lessthanseventy Mar 25, 2026
4b64c7f
docs: add MCP elicitation + auto-check implementation plan
lessthanseventy Mar 25, 2026
d67aa4b
feat: add MCP elicitation module for structured user interaction
lessthanseventy Mar 25, 2026
e08b4b2
feat: wire elicitation protocol into MCP server
lessthanseventy Mar 25, 2026
20b4492
feat: add threshold-based elicitation to a11y_check tool
lessthanseventy Mar 25, 2026
174bad3
feat: add check_work composite MCP tool
lessthanseventy Mar 25, 2026
bbadc88
feat: replace .claude_docs with CLAUDE.md in installer
lessthanseventy Mar 25, 2026
9661fd4
docs: update README for check_work tool and auto-check workflow
lessthanseventy Mar 25, 2026
b34081e
refactor: reduce cyclomatic complexity in maybe_elicit
lessthanseventy Mar 25, 2026
5402774
fix: add extract_violations tests, elicitation comment, and cache eli…
lessthanseventy Mar 25, 2026
56e3b44
fix: formatting
lessthanseventy Mar 25, 2026
76d8d0e
style: format axe_runner curl args list
lessthanseventy Mar 25, 2026
b450988
Merge remote-tracking branch 'origin/main' into feat/axe-core-migrati…
lessthanseventy Mar 25, 2026
d81483d
fix: add Playwright to CI, fix a11y_check test assertion
lessthanseventy Mar 25, 2026
674a90c
fix: resolve credo --strict issues for CI
lessthanseventy Mar 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,16 @@ jobs:
if: matrix.elixir == '1.19'
run: mix format --check-formatted

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'

- name: Install npm dependencies and Playwright
run: |
cd assets && npm install
npx playwright install chromium --with-deps

- name: Compile with warnings as errors
run: mix compile --warnings-as-errors
env:
Expand Down
174 changes: 80 additions & 94 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,30 @@

**Accessibility Snapshot Testing for Elixir + Phoenix**

Excessibility helps you test your Phoenix apps for accessibility (WCAG compliance) by taking HTML snapshots during tests and running them through [Pa11y](https://pa11y.org/).
Excessibility helps you test your Phoenix apps for accessibility (WCAG compliance) by taking HTML snapshots during tests and running them through [axe-core](https://github.com/dequelabs/axe-core) via [Playwright](https://playwright.dev/).

## Why Excessibility?

- **Keep accessibility in your existing test feedback loop.** Snapshots are captured inside ExUnit, Wallaby, and LiveView tests, so regressions surface together with your functional failures.
- **Ship safer refactors.** Explicit baseline locking and comparison lets reviewers see exactly what changed and approve intentionally.
- **Debug CI-only failures quickly.** Pa11y output points to the failing snapshot, and the saved artifacts make it easy to reproduce locally.
- **Debug CI-only failures quickly.** axe-core output points to the failing snapshot, and the saved artifacts make it easy to reproduce locally.

## How It Works

1. **During tests**, call `html_snapshot(conn)` to capture HTML from your Phoenix responses, LiveViews, or Wallaby sessions
2. **After tests**, run `mix excessibility` to check all snapshots with Pa11y for WCAG violations
2. **After tests**, run `mix excessibility` to check all snapshots with axe-core for WCAG violations
3. **Lock baselines** with `mix excessibility.baseline` when snapshots represent a known-good state
4. **Compare changes** with `mix excessibility.compare` to review what changed and approve/reject
5. **In CI**, Pa11y reports accessibility violations alongside your test failures
5. **In CI**, axe-core reports accessibility violations alongside your test failures

## Features

- Snapshot HTML from `Plug.Conn`, `Wallaby.Session`, `Phoenix.LiveViewTest.View`, and `Phoenix.LiveViewTest.Element`
- Explicit baseline locking and comparison workflow
- Interactive good/bad approval when comparing snapshots
- Optional PNG screenshots via ChromicPDF
- Screenshots via Playwright
- Mockable system/browser calls for CI
- Pa11y configuration with sensible LiveView defaults
- axe-core accessibility checking with sensible LiveView defaults

## LLM Development Features

Expand Down Expand Up @@ -136,7 +136,7 @@ Automatically captures LiveView state throughout test execution and generates sc
mix excessibility.debug test/my_test.exs
```

See [CLAUDE.md](CLAUDE.md) for detailed usage.
See the project's `CLAUDE.md` file for detailed usage.

### Telemetry Implementation

Expand Down Expand Up @@ -172,14 +172,6 @@ test "manual capture", %{conn: conn} do
end
```

### Claude Documentation

Create `.claude_docs/excessibility.md` to teach Claude how to use these debugging features:

```bash
mix excessibility.setup_claude_docs
```

## MCP Server & Claude Code Skills

Excessibility includes an MCP (Model Context Protocol) server and Claude Code skills plugin for AI-assisted development.
Expand All @@ -192,55 +184,47 @@ The MCP server provides tools for AI assistants to run accessibility checks and

| Tool | Speed | Description |
|------|-------|-------------|
| `check_route` | Fast | Run Pa11y on a live route (requires app running) |
| `explain_issue` | Fast | Get explanation and fix suggestions for a WCAG violation code |
| `suggest_fixes` | Fast | Get Phoenix-specific code fixes for accessibility issues |
| `generate_test` | Fast | Generate test code with `html_snapshot()` calls for a route |
| `list_analyzers` | Fast | List available timeline analyzers |
| `get_timeline` | Fast | Read captured timeline showing LiveView state evolution |
| `a11y_check` | Slow | Run axe-core accessibility checks on snapshots or URLs |
| `check_work` | Slow | Run tests + a11y check + optional perf analysis (auto-check) |
| `debug` | Slow | Run tests with telemetry capture - returns timeline for analysis |
| `get_snapshots` | Fast | List or read HTML snapshots captured during tests |
| `analyze_timeline` | Fast | Run analyzers on captured timeline data |
| `list_violations` | Fast | List recent Pa11y violations from snapshots |
| `e11y_check` | Slow | Run tests and/or Pa11y accessibility checks on HTML snapshots |
| `e11y_debug` | Slow | Run tests with telemetry capture - returns timeline for analysis |
| `get_timeline` | Fast | Read captured timeline showing LiveView state evolution |
| `generate_test` | Fast | Generate test code with `html_snapshot()` calls for a route |

**Recommended workflow:**
### Auto-Check Workflow

1. `check_route` - Quick accessibility check on running app
2. `explain_issue` - Understand what violations mean
3. `generate_test` - Create test with `html_snapshot()` calls
4. `e11y_debug` - Run test to capture timeline
5. `analyze_timeline` - Find performance issues
The installer adds `CLAUDE.md` instructions that tell Claude to automatically run `check_work` after modifying code. This creates a seamless feedback loop:

1. Claude edits your code
2. `check_work` runs automatically (tests + a11y + optional perf analysis)
3. When critical violations are found, MCP elicitation presents a triage form for you to prioritize fixes
4. Minor issues are returned directly for Claude to fix silently
5. Clean results return immediately

**Automatic Setup:**

MCP server support is configured automatically when you run the installer:
The installer configures everything automatically:

```bash
mix excessibility.install
mix deps.get
```

This creates `.claude/mcp_servers.json` with the excessibility MCP server configuration.
This will:
- Add configuration to `config/test.exs`
- Install Playwright and axe-core via npm
- Register the MCP server with Claude Code
- Install the Claude Code skills plugin
- Add auto-check instructions to `CLAUDE.md`

Use `--no-mcp` to skip MCP setup if you don't need AI assistant integration.
Use `--no-mcp` to skip Claude Code integration.

**Manual Setup:**

Configure Claude Code's `mcp_servers.json`:

```json
{
"excessibility": {
"command": "mix",
"args": ["run", "--no-halt", "-e", "Excessibility.MCP.Server.start()"],
"cwd": "/path/to/your/project"
}
}
```bash
claude mcp add excessibility -s project -- mix run --no-halt -e "Excessibility.MCP.Server.start()"
claude plugins add deps/excessibility/priv/claude-plugin
```

The MCP server is now available in Claude Code.

### Claude Code Skills Plugin

Install the skills plugin for structured accessibility workflows:
Expand All @@ -253,9 +237,9 @@ claude plugins add /path/to/excessibility/priv/claude-plugin

| Skill | Description |
|-------|-------------|
| `/e11y-tdd` | TDD workflow with html_snapshot and Pa11y - sprinkle snapshots to see what's rendered, delete when done |
| `/e11y-debug` | Debug workflow with timeline analysis - inspect state at each event, correlate with Pa11y failures |
| `/e11y-fix` | Reference guide for fixing Pa11y/WCAG errors with Phoenix-specific patterns |
| `/e11y-tdd` | TDD workflow with html_snapshot and axe-core - sprinkle snapshots to see what's rendered, delete when done |
| `/e11y-debug` | Debug workflow with timeline analysis - inspect state at each event, correlate with axe-core failures |
| `/e11y-fix` | Reference guide for fixing axe-core/WCAG errors with Phoenix-specific patterns |

**Example workflow:**

Expand All @@ -266,18 +250,39 @@ claude plugins add /path/to/excessibility/priv/claude-plugin
# 1. EXPLORE - Add html_snapshot() calls to see what's rendered
# 2. RED - Write test with snapshot at key moment
# 3. GREEN - Implement feature, use snapshots to debug
# 4. CHECK - Run mix excessibility for Pa11y validation
# 4. CHECK - Run mix excessibility for axe-core validation
# 5. CLEAN - Remove temporary snapshots
```

### Optional: Hooks for Additional Automation

For belt-and-suspenders automation, you can also configure Claude Code hooks
to run tests after file edits. Add to your `.claude/settings.json`:

```json
{
"hooks": {
"PostToolUse": [
{
"matcher": "Edit",
"command": "mix test --failed"
}
]
}
}
```

This runs failing tests after each file edit, catching breakage immediately.
The `check_work` MCP tool handles accessibility and performance checking separately.

## Installation

Add to `mix.exs`:

```elixir
def deps do
[
{:excessibility, "~> 0.10", only: [:dev, :test]}
{:excessibility, "~> 0.11", only: [:dev, :test]}
]
end
```
Expand All @@ -297,8 +302,7 @@ mix excessibility.install --head-render-path /login

The installer will:
- Add configuration to `config/test.exs`
- Create a `pa11y.json` with sensible defaults for Phoenix/LiveView
- Install Pa11y via npm in your assets directory
- Install Playwright and axe-core via npm in your assets directory

## Quick Start

Expand Down Expand Up @@ -331,14 +335,14 @@ The installer will:
3. **Typical workflow:**

```bash
# Run specific test + Pa11y in one command
# Run specific test + axe-core in one command
mix excessibility test/my_test.exs
mix excessibility test/my_test.exs:42
mix excessibility --only a11y

# Or run tests separately, then check all snapshots
mix test # Generates snapshots in test/excessibility/
mix excessibility # Runs Pa11y against all snapshots
mix excessibility # Runs axe-core against all snapshots

# Lock current snapshots as known-good baseline
mix excessibility.baseline
Expand Down Expand Up @@ -372,7 +376,7 @@ It returns the source unchanged, so you can use it in pipelines.
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `:name` | `string` | auto-generated | Custom filename (e.g., `"login_form.html"`). Default is `ModuleName_LineNumber.html` |
| `:screenshot?` | `boolean` | `false` | Generate PNG screenshots (requires ChromicPDF) |
| `:screenshot?` | `boolean` | `false` | Generate PNG screenshots (via Playwright) |
| `:open_browser?` | `boolean` | `false` | Open the snapshot in your browser after writing |
| `:cleanup?` | `boolean` | `false` | Delete existing snapshots for the current test module before writing |

Expand Down Expand Up @@ -419,11 +423,10 @@ All configuration goes in `test/test_helper.exs` or `config/test.exs`:
| `:browser_mod` | No | `Wallaby.Browser` | Module for browser interactions |
| `:live_view_mod` | No | `Excessibility.LiveView` | Module for LiveView rendering |
| `:excessibility_output_path` | No | `"test/excessibility"` | Base directory for snapshots |
| `:pa11y_path` | No | auto-detected | Path to Pa11y executable |
| `:pa11y_config` | No | `"pa11y.json"` | Path to Pa11y config file |
| `:axe_runner_path` | No | auto-detected | Path to axe-runner.js script |
| `:head_render_path` | No | `"/"` | Route used for rendering `<head>` content |
| `:custom_enrichers` | No | `[]` | List of custom enricher modules (see [Telemetry Analysis](docs/telemetry-analysis.md)) |
| `:custom_analyzers` | No | `[]` | List of custom analyzer modules (see [Telemetry Analysis](docs/telemetry-analysis.md)) |
| `:custom_enrichers` | No | `[]` | List of custom enricher modules (see Timeline Analysis section above) |
| `:custom_analyzers` | No | `[]` | List of custom analyzer modules (see Timeline Analysis section above) |

Example:

Expand All @@ -438,57 +441,41 @@ Application.put_env(:excessibility, :excessibility_output_path, "test/accessibil
ExUnit.start()
```

## Pa11y Configuration
## axe-core Configuration

The installer creates a `pa11y.json` in your project root with sensible defaults for Phoenix/LiveView:
axe-core runs via Playwright and reports violations with structured data including `id`, `impact` (critical, serious, moderate, minor), `description`, `helpUrl`, and affected `nodes`.

```json
{
"ignore": [
"WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2"
]
}
```
You can disable specific rules via the `--disable-rules` flag:

The ignored rule (H32.2) is "Form does not contain a submit button" — a common false positive for LiveView forms that use `phx-submit` without traditional submit buttons.
```bash
mix excessibility --disable-rules=color-contrast
```

Add additional rules to ignore as needed for your project:
Or check a specific URL directly:

```json
{
"ignore": [
"WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2",
"WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail"
]
}
```bash
mix excessibility.check http://localhost:4000/my-page
```

## Screenshots

To enable PNG screenshots, start ChromicPDF in your test helper:

```elixir
# test/test_helper.exs
{:ok, _} = ChromicPDF.start_link(name: ChromicPDF)

ExUnit.start()
```

Then use `screenshot?: true` in your snapshots:
Screenshots are captured via Playwright when using `screenshot?: true`:

```elixir
html_snapshot(conn, screenshot?: true)
```

Screenshots are saved alongside HTML files with `.png` extension.
Screenshots are saved alongside HTML files with `.png` extension. Playwright is installed automatically as part of the npm dependencies.

## Mix Tasks

| Task | Description |
|------|-------------|
| `mix excessibility.install` | Configure config/test.exs, create pa11y.json, install Pa11y via npm |
| `mix excessibility` | Run Pa11y against all existing snapshots |
| `mix excessibility [test args]` | Run tests, then Pa11y on new snapshots (passthrough to mix test) |
| `mix excessibility.install` | Configure config/test.exs, install Playwright and axe-core via npm |
| `mix excessibility` | Run axe-core against all existing snapshots |
| `mix excessibility [test args]` | Run tests, then axe-core on new snapshots (passthrough to mix test) |
| `mix excessibility.check [url]` | Run axe-core on a live URL via Playwright |
| `mix excessibility.snapshots` | List and manage HTML snapshots |
| `mix excessibility.baseline` | Lock current snapshots as baseline |
| `mix excessibility.compare` | Compare snapshots against baseline, resolve diffs interactively |
| `mix excessibility.compare --keep good` | Keep all baseline versions (reject changes) |
Expand All @@ -498,7 +485,6 @@ Screenshots are saved alongside HTML files with `.png` extension.
| `mix excessibility.debug [test args] --format=package` | Create debug package directory |
| `mix excessibility.latest` | Display most recent debug report |
| `mix excessibility.package [test]` | Create debug package (alias for --format=package) |
| `mix excessibility.setup_claude_docs` | Create/update .claude_docs/excessibility.md |

## CI and Non-Interactive Environments

Expand Down
Loading
Loading