Skip to content

test-plan.md

Latest commit

 

History

History
90 lines (71 loc) · 3.93 KB

test-plan.md

File metadata and controls

90 lines (71 loc) · 3.93 KB
Raw

Test Plan

Phase 1

  • API unit/integration: /api/health, /api/ready, /api/metrics
  • DB/Redis check helpers
  • Frontend: SystemStatus happy path + API down + dependency unhealthy
  • Compose smoke (development): docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build then curl health/ready

Phase 2

  • Auth register/login/logout/me
  • Duplicate email / invalid email / weak password
  • Ownership isolation for agent sessions
  • Alembic upgrade + downgrade
  • Frontend login success/failure + dashboard + logout
  • CI Postgres/Redis services + alembic upgrade head

Phase 3

  • GitHub App JWT (RS256) + invalid key handling
  • OAuth connect/callback state (invalid, expired, replay)
  • Installations list + cross-user isolation
  • Repository discovery (mocked) + connect/disconnect ownership
  • Webhook signature + installation lifecycle + delivery idempotency
  • Frontend: not configured / connect / list / connect repo / disconnect / error / unauthorized / loading
  • Fresh clone without GitHub credentials still boots
  • Optional live GitHub App E2E (local credentials only; not required in CI)

Phase 4

  • Execution job API create/list/get/logs/cancel + IDOR
  • Path traversal / shell argv rejection
  • Worker claim + fixture checkout + remote sanitization unit tests
  • Docker sandbox integration (non-root, no socket, no secrets, timeout, cleanup)
  • Frontend executions UI tests
  • Compose worker service + sandbox image build
  • Optional live GitHub installation clone (credentials)

Phase 5

  • Agent run API + ownership/cancel
  • FakeLLM deterministic sandbox E2E (inspect/edit/test/diff)
  • Tool path escape tests + prompt-injection secret non-leak
  • Frontend agent panel tests
  • Optional live OpenAI coding task

Phase 6

  • WebSocket auth / anonymous / IDOR / revoked session
  • Event publisher sequence + malformed rejection
  • FakeLLM lifecycle event sequence via Redis Pub/Sub
  • Docker incremental command output streaming
  • Frontend live activity / XSS-safe output / reconnect / cancel
  • Docs: docs/realtime.md
  • Optional multi-replica WS soak (Compose remains single API)

Phase 7

  • Owner-only approval, rejection, duplicate approval, and cancellation boundary
  • SHA-256 diff and base-commit integrity checks
  • Deterministic local Git commit and push to a bare remote
  • Mocked PR creation and duplicate-delivery idempotency
  • Publication event protocol and frontend approval controls
  • Live GitHub App installation and PR creation (requires repository credentials)

Stage 6 — Observability and full-system reliability

  • Request/workflow correlation and canonical structured-field tests
  • Health/readiness, migration compatibility, and worker-heartbeat tests
  • Deterministic golden path with real PostgreSQL, Redis, Docker, local Git, fake LLM, and GitHub stub
  • Deterministic Redis, lease, validation, cancellation, revocation, tamper, stale-base, duplicate, WebSocket, GitHub, and sandbox fault tests
  • Synthetic-secret scan and baseline timing report (baseline only; no timing threshold is enforced)
  • Dependency advisory and migration-drift assessment (docs/dependency-audit.md)
  • Operations checklist: docs/operations.md

The Docker fixture checkout test resolves the repository root instead of relying on the current working directory, so it is valid from both the worker directory and the repository root.

Stage 7 — Product UX and demo readiness

  • Product-first landing page with workflow and trust-boundary explanation
  • Shared responsive navigation and accessible focus/reduced-motion defaults
  • Data-backed dashboard summary and contextual empty/loading states
  • Agent timeline, activity/output separation, bounded diff navigation, and approval context
  • Product-first README, architecture diagram, demo scenario, and screenshot plan