Skip to content

ShivaniNR/qabot

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
qabot
qabot
 
 
tests
tests
 
 
.env.example
.env.example
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
pyproject.toml
pyproject.toml
 
 
uv.lock
uv.lock
 
 

Repository files navigation

qabot

Adversarial QA agent for web apps — powered by LLMs and browser automation.

qabot is a CLI tool that uses an AI agent to automatically test any web application. It navigates your app through a real browser, runs test cases you define in plain English, then goes adversarial — trying XSS, auth bypass, broken flows, and edge cases to find bugs before your users do.

It works with any web app — just point it at a URL.

How It Works

qabot runs three sequential testing phases, each powered by an independent LLM-driven browser agent:

┌─────────────────────────────────────────────────────────────────────┐
│                          qabot test <URL>                           │
└──────────────────────────────┬──────────────────────────────────────┘
                               │
                               ▼
                 ┌──────────────────────────┐
                 │    Phase 1: Discovery     │
                 │                          │
                 │  • Navigates all links   │
                 │  • Maps routes & forms   │
                 │  • Identifies auth pages │
                 │  • Caches results (TTL)  │
                 │                          │
                 │  Output: SiteMap         │
                 └────────────┬─────────────┘
                              │
                              ▼
                ┌───────────────────────────┐
                │  Phase 2: Directed Tests   │
                │                           │
                │  • Runs user-defined tests│
                │  • Fresh browser per test │
                │  • Step-by-step execution │
                │  • PASS/FAIL verdicts     │
                │                           │
                │  Output: TestResult[]     │
                └────────────┬──────────────┘
                             │
                             ▼
               ┌────────────────────────────┐
               │  Phase 3: Adversarial       │
               │                            │
               │  • XSS & injection attacks │
               │  • Boundary testing        │
               │  • Auth bypass attempts    │
               │  • Flow breaking           │
               │                            │
               │  Output: Finding[]         │
               └────────────┬───────────────┘
                            │
                            ▼
               ┌────────────────────────────┐
               │      Report Generation      │
               │                            │
               │  • Collects all results    │
               │  • Renders HTML report     │
               │  • Opens in browser        │
               │                            │
               │  Output: qabot-report/     │
               └────────────────────────────┘

Component Architecture

┌──────────────────────────────────────────────────────────────────┐
│                            CLI (cli.py)                          │
│                   click commands: test, explore                   │
└──────────────────────────────┬───────────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│                     Orchestrator (orchestrator.py)                │
│              LLM selection · browser profile · phase wiring      │
└───────┬──────────────────────┬───────────────────────┬──────────┘
        │                      │                       │
        ▼                      ▼                       ▼
┌───────────────┐  ┌───────────────────┐  ┌───────────────────────┐
│   Discovery   │  │     Directed      │  │     Adversarial       │
│  discovery.py │  │    directed.py    │  │    adversarial.py     │
│               │  │                   │  │                       │
│  Prompt:      │  │  Prompt:          │  │  Prompt:              │
│  discovery.md │  │  directed.md      │  │  adversarial.md       │
│               │  │                   │  │                       │
│  → SiteMap    │  │  → TestResult[]   │  │  → Finding[]          │
└───────┬───────┘  └─────────┬─────────┘  └───────────┬───────────┘
        │                    │                        │
        └────────────────────┼────────────────────────┘
                             │
                             ▼
              ┌──────────────────────────────┐
              │     ResultCollector           │
              │     collector.py              │
              │                              │
              │  Gathers all phase outputs   │
              │  → FullReport                │
              └──────────────┬───────────────┘
                             │
                             ▼
              ┌──────────────────────────────┐
              │     Report Generator          │
              │     generator.py              │
              │                              │
              │  Jinja2 → template.html      │
              │  → qabot-report/index.html   │
              └──────────────────────────────┘

Shared data models (models.py) are the only interface between components:

Model Purpose
SiteMap Discovered routes, forms, auth-gated pages
TestResult Verdict (PASS/FAIL), steps taken, severity, reproduction steps
AdversarialFinding Bug title, severity, description, reproduction steps
FullReport Combines all results; computes pass/fail counts and severity checks

Installation

Prerequisites: Python 3.11+, uv

Option 1: Install directly (no cloning needed)

pip install git+https://github.com/ShivaniNR/qabot.git

Then use it from anywhere:

qabot explore http://your-app.com --describe "My app"
qabot test http://your-app.com --config qabot.yaml --adversarial

Option 2: Clone the repo

git clone https://github.com/ShivaniNR/qabot.git
cd qabot
uv sync

Then prefix commands with uv run:

uv run qabot explore http://your-app.com --describe "My app"

Environment setup

Create a .env file (see .env.example):

ANTHROPIC_API_KEY=your-key-here

At least one LLM API key is required. Supported providers:

Provider Env Variable Default Model
Anthropic ANTHROPIC_API_KEY Claude Sonnet 4
OpenAI OPENAI_API_KEY GPT-4.1 Mini
Google GOOGLE_API_KEY Gemini 2.0 Flash

Quick Start

Adversarial-only scan (zero config)

uv run qabot explore http://your-app.com --describe "E-commerce app with products, cart, and checkout"

Full testing with config

uv run qabot test http://your-app.com --config qabot.yaml --adversarial

CLI Reference

qabot test <URL>

Run directed tests and optionally adversarial testing against a target URL.

Flag Type Default Description
--config, -c PATH — Path to qabot.yaml config file
--llm STRING auto-detect LLM model name (e.g., claude-sonnet-4-20250514, gpt-4.1-mini, gemini-2.0-flash)
--adversarial / --no-adversarial FLAG --no-adversarial Enable adversarial testing after directed tests
--max-steps INT 50 Maximum browser steps for adversarial mode
--rediscover FLAG off Force re-run discovery, ignore cache
--ci FLAG off CI mode: no browser popup, text summary to stdout
--fail-on CHOICE critical Exit code 1 if bugs at or above this severity (critical, major, minor)
--json-output FLAG off Output full results as JSON to stdout

qabot explore <URL>

Run adversarial-only exploration (no config file needed).

Flag Type Default Description
--llm STRING auto-detect LLM model name
--describe, -d STRING — Brief description of the app (helps the agent understand context)
--max-steps INT 50 Maximum browser steps
--focus, -f STRING (multiple) — Areas to prioritize (e.g., --focus auth --focus forms --focus checkout)
--auth KEY=VALUE (multiple) — Login credentials (e.g., --auth email=test@test.com --auth password=secret)
--login-url STRING /login Login page path
--rediscover FLAG off Force re-run discovery
--ci FLAG off CI mode
--fail-on CHOICE critical Severity threshold for exit code
--json-output FLAG off JSON output

Configuration File

Create a qabot.yaml to define test cases and settings:

url: http://127.0.0.1:8000
description: "E-commerce app with products, cart, checkout, signup, and login"

# Authentication (optional)
auth:
  login_url: /login
  credentials:
    email: test@example.com
    password: secret123

# Directed test cases — plain English instructions
tests:
  - name: "Signup with valid data"
    flow: "Go to /signup, enter name 'Alice', email 'alice@test.com', password 'pass123', click Sign Up, verify welcome message"

  - name: "Empty signup submission"
    flow: "Go to /signup, click Sign Up without filling any fields, verify that validation errors appear"

  - name: "Add product to cart"
    flow: "Go to /products, add Wireless Headphones with quantity 1, verify cart shows the item with correct price $49.99"

  - name: "Checkout with empty cart"
    flow: "Go to /checkout directly without adding items, verify it shows an error or redirects to cart"

# Adversarial settings
adversarial:
  enabled: true
  focus: ["forms", "cart", "checkout"]
  max_steps: 20

# Discovery cache settings
discovery:
  cache: true
  max_age: "1h"    # supports: s, m, h, d

Severity Levels

qabot categorizes all findings by severity:

Severity Examples
Critical XSS vulnerabilities, SQL injection, auth bypass, 500 errors, data corruption, negative prices
Major Features broken under edge cases, missing validation causing wrong behavior, broken error handling
Minor Cosmetic issues, missing validation messages, poor UX for unusual inputs

The --fail-on flag controls which severity triggers a non-zero exit code (useful for CI gates).

CI/CD Integration

uv run qabot test http://staging.your-app.com \
  --config qabot.yaml \
  --adversarial \
  --ci \
  --fail-on major
  • --ci — suppresses browser popup, prints text summary to stdout
  • --fail-on major — exits with code 1 if any major or critical bugs found
  • --json-output — machine-readable JSON output for pipeline parsing

Report Output

qabot generates a self-contained HTML report at qabot-report/index.html with:

  • Pass/fail summary for directed tests
  • Adversarial findings with severity and reproduction steps
  • Discovered sitemap (routes, forms, auth-gated pages)

The report is a single HTML file with inline CSS — no external assets, works offline, shareable as one file.

Try It Locally

A sample app with 5 intentional bugs is included for testing:

Bug Location
XSS vulnerability Signup name field renders unsanitized HTML
No form validation Signup accepts completely empty fields
Negative quantities Cart allows negative item quantities
Empty cart checkout Can place an order with nothing in cart
No 404 page Unknown routes return a blank 200 
# Terminal 1 — run the buggy sample app
uv run uvicorn tests.sample_app:app --port 8000

# Terminal 2 — unleash qabot on it
uv run qabot test http://127.0.0.1:8000 --config tests/qabot.yaml --adversarial

Attack Strategies

During adversarial testing, qabot attempts:

  • Input attacks — empty fields, long strings, XSS payloads (<script>alert('xss')</script>), SQL injection ('; DROP TABLE users; --), negative numbers, unicode/emoji
  • Flow breaking — skipping checkout steps, accessing auth pages without login, double-submitting forms, navigating away mid-flow
  • Boundary testing — max/min values, non-existent pages (404 handling), URL parameter manipulation
  • Auth & authorization — accessing other users' data via ID manipulation, admin pages without auth, default credentials

Tech Stack

Component Technology
Runtime Python 3.11+
Browser automation browser-use (Playwright + LLM)
CLI Click
Config & models Pydantic + PyYAML
Report rendering Jinja2
LLM providers Anthropic (Claude), OpenAI (GPT-4), Google (Gemini)
Package manager uv

License

MIT

About

Adversarial QA agent for web apps

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages