Playwright API Test Automation

Playwright-based API test suite for APIMatic's Docs as Code platform. Covers portal generation, snapshot diffing, chatbot quality, OpenAPI validation, and customer portal builds.

Test Suites

Suite	Spec file	Playwright project	What it tests
Docs as Code	docs-as-code.spec.ts	docs-as-code	Portal generation: baseUrl, LLMs.txt, portal.js config flags, nav structure, AI Copilot
AI Assist Menu	ai-assist-menu.spec.ts	docs-as-code	AI integration links (Cursor, Claude Code, VS Code) in generated portals
Async API	async-api.spec.ts	api	Customer portal generation via async endpoint (used in the Customer Portals workflow)
Portal Snapshot	portal-snapshot.spec.ts	api	File-level diff snapshot comparison of generated portal output
Chatbot	chatbot.spec.ts	api	Chatbot response quality via deterministic checks + GPT-4.1 rubric judge
Validation	validation.spec.ts	api	OpenAPI spec validation endpoint
Sync Customer	sync-customer.spec.ts	api	Customer portal sync flows
Docs as Code Negative	docs-as-code-negative.spec.ts	api	Error/failure scenarios for portal generation

Project Structure

tests/               Test specs
endpoints/           Typed API request helpers (PortalEndpoints, ValidateEndpoints)
utils/               Shared utilities (zip, file I/O, text normalisation)
data/                Input fixtures (OpenAPI specs zipped as test data)
snapshots/           Baselines for snapshot and chatbot tests
customer-builds/     Customer zip files used by the async-api tests
routes.ts            API route constants
playwright.config.ts Playwright configuration (projects, baseURL, auth header)
.github/workflows/   CI workflows (see below)

Prerequisites

• Node.js 20+
• npm

npm ci

Environment Variables

Set these before running tests locally (e.g. in a .env file):

Variable	Description
url	Base URL of the APIMatic API (dev or prod)
apiKey	APIMatic API key (X-Auth-Key header)
CHATBOT_URL	Full URL of the chatbot endpoint
OPENAI_API_KEY	OpenAI key used by the chatbot rubric judge
copilotKey	API Copilot key for AI configuration tests

Running Tests Locally

# Docs as Code suite (all tests except the heavy AI config test)
npx playwright test --project=docs-as-code --workers=4 --grep-invert "aiConfiguration for all languages"

# AI Configuration test (runs single-threaded due to external API usage)
npx playwright test --project=docs-as-code --workers=1 --grep "aiConfiguration for all languages"

# All api-project tests
npx playwright test --project=api --workers=4

# Single spec
npx playwright test tests/chatbot.spec.ts --workers=4

# Open HTML report after a run
npx playwright show-report

CI / GitHub Actions Workflows

All workflows are triggered manually via workflow_dispatch with an environment_name input (dev or prod).

Workflow	File	Runs
Docs as Code Tests	docs-as-code-tests.yml	docs-as-code project; heavy AI config test runs in a separate parallel job
Chatbot Tests	chatbot-tests.yml	chatbot.spec.ts; uploads actual responses as artifacts
Customer Portals Test	customer-portals.yml	async-api.spec.ts; pulls customer build zips from a private repo; Slack alert on failure
Portal Snapshot Tests	portal-snapshot-tests.yml	portal-snapshot.spec.ts; uploads diff files as artifacts; Slack alert on failure

Required Secrets

Secret	Used by
API_KEY_DEV	All workflows (dev environment)
API_KEY_PROD	All workflows (prod environment)
OPENAI_KEY	Chatbot Tests
ACTIONS_PAT	Customer Portals (to check out the private builds repo)
SLACK_BOT_TOKEN	Customer Portals, Portal Snapshot Tests (failure notifications)

Chatbot Test Approach

Tests in chatbot.spec.ts use a two-layer evaluation strategy:

1. Deterministic checks — keyword presence, code block requirements (fast, no LLM cost).
2. Rubric-based LLM judge — GPT-4.1 scores accuracy, completeness, code quality, and task success (0–3 each) against a stored baseline. Pass threshold: total ≥ 9 with no dimension below 1.

Baselines are stored under snapshots/chat-snapshots/. On the first run for a new prompt ID, the baseline is created automatically and the test fails once — rerun to pass.

Snapshot Tests

portal-snapshot.spec.ts generates portals and diffs the output file tree against baselines. Diff files are written to test-results/diffs/ and uploaded as CI artifacts for review.