Skip to content

TESTING.md

Latest commit

 

History

History
416 lines (328 loc) · 13 KB

TESTING.md

File metadata and controls

416 lines (328 loc) · 13 KB
Raw

Comprehensive Testing Infrastructure

This document describes the complete testing infrastructure for the universal-agent-runtime project, designed to achieve and maintain 100% code coverage across all components.

Overview

The testing infrastructure provides:

  • 100% Code Coverage: Comprehensive testing for both Rust backend and TypeScript frontend
  • Real Service Integration: Tests run against actual PostgreSQL, SurrealDB, Redis, and LLM services
  • Multi-Level Testing: Unit, integration, API, and end-to-end tests
  • Local verification: Development and release checks run on operator-controlled local hosts
  • Docker Integration: Consistent testing environments via Docker Compose
  • Rich Reporting: HTML, JSON, XML, and LCOV coverage reports

Architecture

Test Categories

  1. Unit Tests

    • Rust: cargo test --lib --bins --tests
    • TypeScript: bun test
    • Fast, isolated component testing

  2. Integration Tests

    • Rust: Tests in tests/ directory with _integration suffix
    • Database integration with real PostgreSQL and SurrealDB
    • Redis caching and session management
    • Single-threaded execution for data consistency

  3. API Tests

    • HTTP endpoint testing via tests::api module
    • Request/response validation
    • Authentication and authorization flows
    • Error handling and edge cases

  4. End-to-End Tests

    • Playwright automation testing full user workflows
    • Cross-browser compatibility (Chromium focus)
    • UI interaction validation
    • Real application deployment testing

  5. Performance Tests

    • Load testing and benchmarking
    • Response time validation
    • Resource usage monitoring
    • Regression detection

Configuration Files

test-config.yaml

Central configuration for test execution parameters, service settings, and coverage thresholds.

docker-compose.test.yaml

Orchestrates test services with proper health checks:

  • PostgreSQL with pgvector extension
  • SurrealDB in memory mode
  • Redis Stack for caching
  • Unstructured API for document processing
  • Main application with test environment

Test Coverage Configuration

  • Rust: local cargo-llvm-cov; see docs/coverage-baseline.md for the 60% threshold and per-file baseline
  • TypeScript: local vitest --coverage using the v8 provider in frontend/vitest.config.ts
  • Playwright: V8 coverage integration
  • Local guard: tools/coverage-drift.sh reports per-file drift vs. docs/coverage-baseline.md; .grcovrc was removed (unused, superseded by cargo-llvm-cov)

Test Execution

Local Development

Quick Tests (5-10 minutes)

./tools/test-all.sh --quick
  • Smoke tests and unit tests only
  • Fast feedback for development

Full Test Suite (15-30 minutes)

./tools/test-all.sh --full
  • All test categories including E2E
  • Complete coverage analysis
  • Performance benchmarks

Sequential compatibility mode

./tools/test-all.sh --ci
  • Non-parallel local execution for constrained hosts
  • Comprehensive logging
  • Strict coverage requirements

Coverage Reports

Generate All Coverage Reports

./tools/coverage.sh --unified

Rust Coverage Only

./tools/coverage.sh --rust-only

TypeScript Coverage Only

./tools/coverage.sh --typescript-only

Open Reports in Browser

./tools/coverage.sh --unified --open

Docker-Based Testing

Full Docker Integration

docker-compose -f docker-compose.test.yaml up --build
docker-compose -f docker-compose.test.yaml run test-runner

Individual Services

# Start dependencies only
docker-compose -f docker-compose.test.yaml up -d postgres redis surreal

# Run tests against external services
./tools/test-all.sh --full

Verification execution policy

Run unit, integration, end-to-end, browser, conformance, regression, security, performance, load, stress, soak, and release-certification checks locally. Formatting, linting, typechecking, and coverage also run locally. GitHub Actions are reserved for deployment execution and deployment-specific validation; a build, package, image, or release step must not invoke a test suite directly or as a side effect.

The repository enforces this boundary with pnpm github-actions-policy:validate and the pre-commit hook in lefthook.yml. The tiered local commands in .claude/rules/rust.md and .claude/rules/typescript.md remain authoritative for when each check runs.

Test Environment Setup

Prerequisites

Local Development

# Rust toolchain with components
rustup component add rustfmt clippy llvm-tools-preview

# Coverage tools
cargo install cargo-llvm-cov

# Node.js and Bun
curl -fsSL https://bun.sh/install | bash

# Docker and Docker Compose
# Install via official documentation

# Playwright
npx playwright install --with-deps chromium

Local certification hosts

Install the required tools on each supported local build/certification host. Release receipts record the exact source commit and tool outputs used.

Environment Variables

Test Execution

# Database connections
DATABASE_URL=postgres://postgres:postgres@localhost:5431/uar_test
REDIS_URL=redis://localhost:6378
SURREAL_URL=ws://localhost:8001/rpc

# Test configuration
CONFIG_FILE=test-config.yaml
APP_ENVIRONMENT=test
TEST_MODE=true
COVERAGE=true

# Coverage instrumentation
CARGO_INCREMENTAL=0
RUSTFLAGS="-C instrument-coverage"
LLVM_PROFILE_FILE="tests/coverage/rust/coverage-%p-%m.profraw"

LLM Integration (Optional)

OPENAI_API_KEY=your_api_key_here
TAVILY_API_KEY=your_api_key_here

Directory Structure

tests/
├── integration/           # Rust integration tests
│   ├── api/              # API endpoint tests
│   ├── database/         # Database integration tests
│   └── services/         # Service integration tests
├── e2e/                  # End-to-end tests
│   ├── specs/            # Playwright test specifications
│   ├── pages/            # Page object models
│   └── utils/            # Test utilities
├── performance/          # Performance and benchmark tests
├── coverage/            # Coverage reports
│   ├── rust/            # Rust coverage data
│   ├── typescript/      # TypeScript coverage data
│   ├── e2e/             # E2E coverage data
│   └── unified/         # Combined coverage reports
├── config/              # Test-specific configuration
└── fixtures/            # Test data and fixtures

tools/
├── test-all.sh         # Main test execution script
├── coverage.sh         # Coverage report generator
└── ...

Coverage Tools and Formats

Rust Coverage

Primary Tool: cargo-llvm-cov Alternative: cargo-tarpaulin Formats: LCOV, HTML Configuration: .cargo/config.toml

TypeScript Coverage

Primary Tool: Bun's built-in coverage Alternative: c8, nyc Formats: HTML, JSON, LCOV, text Integration: Playwright V8 coverage for E2E

Unified Reporting

Tool: Custom HTML dashboard Features:

  • Multi-format report aggregation
  • Interactive coverage browsers
  • Historical trend analysis
  • Quality gate validation
  • Direct links to detailed reports

Mutation, fuzz, and property-based testing

Mutation testing

  • Tool: cargo-mutants
  • Run locally: cargo mutants --no-shuffle --features server-full
  • Reports: docs/mutation-history/
  • Summary: bash tools/mutation-summarize.sh <report-dir>

Fuzz testing

  • Tool: cargo-fuzz
  • Targets: fuzz/fuzz_targets/{chunker,rag_verification,mcp_message_parser,json_schema_validator}.rs
  • Run locally: cargo +nightly fuzz run chunker (requires nightly Rust and cargo-fuzz)

Property-based testing

  • Tool: proptest
  • Coverage: settings store serde roundtrip, retrieval RRF invariants, governance policy hot-reload semantics
  • Run locally: included in cargo test

Error handling and observability tests

  • Hot-path unwrap/expect guard: src/uar/api/, src/uar/runtime/, and src/server.rs deny clippy::unwrap_used / clippy::expect_used at the module level. Run cargo clippy --features server-full --no-deps to verify.
  • Central error type tests: src/uar/error.rs verifies UarError stable codes, JSON response shape, and tracing-error span capture.
  • Sentry feature build: cargo check --no-default-features --features server-full,sentry ensures the optional Sentry integration compiles.
  • Documentation: see docs/observability.md for operator-facing tracing and Sentry setup guidance.

Quality Gates

Automated Checks

  1. Format Compliance: cargo fmt, prettier
  2. Linting: clippy, eslint
  3. Type Safety: TypeScript strict mode
  4. Security: cargo audit, bun audit
  5. Coverage Thresholds: Configurable per environment
  6. Performance: Response time and throughput validation
  7. Docker Health: Service connectivity and readiness
  8. Conventional Commits: commitlint + lefthook for the JS workspace
  9. Mutation Testing: cargo-mutants nightly
  10. Fuzz and Property Tests: cargo-fuzz and proptest
  11. Hot-Path Error Handling: clippy::unwrap_used / clippy::expect_used denied in src/uar/api/, src/uar/runtime/, and src/server.rs

Manual Gates

  1. Code Review: Required for all PRs
  2. Architecture Review: For significant changes
  3. Security Review: For authentication/authorization changes
  4. Performance Review: For database schema or API changes

Troubleshooting

Common Issues

Docker Services Not Starting

# Check service health
docker-compose -f docker-compose.test.yaml ps

# View service logs
docker-compose -f docker-compose.test.yaml logs postgres
docker-compose -f docker-compose.test.yaml logs redis
docker-compose -f docker-compose.test.yaml logs surreal

# Reset containers
docker-compose -f docker-compose.test.yaml down -v
docker-compose -f docker-compose.test.yaml up -d --build

Coverage Tool Issues

# Verify tool installation
cargo llvm-cov --version

# Check profraw files generated
find . -name "*.profraw" -type f

# Clear coverage cache
rm -rf tests/coverage/
rm -f *.profraw

Test Failures

# Run specific test category
cargo test --test config_integration -- --nocapture
cargo test tests::api -- --test-threads=1 --nocapture

# Enable debug logging
RUST_LOG=debug cargo test

# Playwright debugging
npx playwright test --debug
npx playwright test --headed --slow-mo=1000

Performance Issues

# Check resource usage
docker stats

# Analyze slow tests
cargo test -- --nocapture | grep -E "(test|elapsed)"

# Profile test execution
time ./tools/test-all.sh --quick

Best Practices

Test Writing

  1. Isolation: Each test should be independent
  2. Determinism: Tests should produce consistent results
  3. Clarity: Test names should describe expected behavior
  4. Coverage: Test both happy paths and error conditions
  5. Performance: Keep unit tests fast (<10ms each)

Coverage Goals

  1. Completeness: Test all code paths including error handling
  2. Quality: Focus on meaningful tests, not just coverage percentage
  3. Maintenance: Keep tests updated with code changes
  4. Documentation: Use tests as behavioral documentation

Local verification integration

  1. Fast feedback: Run the current tier's focused local checks while editing.
  2. Comprehensive validation: Run the phase-completion tier locally.
  3. Release quality: Retain source-bound local certification receipts.
  4. Monitoring: Track local test execution times and success rates.

Metrics and Monitoring

Key Metrics

  • Test Execution Time: Track performance trends
  • Coverage Percentage: Monitor quality improvements
  • Failure Rate: Identify unstable tests
  • Resource Usage: Optimize local certification-host costs

Dashboards

  • Coverage Reports: tests/coverage/unified/index.html
  • Test Results: source-bound local verification receipts
  • Performance Trends: Artifact-based historical data

Alerts

  • Coverage Drop: Below threshold warnings
  • Test Failures: local command failure and retained logs
  • Performance Regression: Automated benchmark comparisons

Contributing

Adding New Tests

  1. Choose appropriate test category (unit/integration/e2e)
  2. Follow naming conventions (*_test.rs, *.test.ts, *.spec.ts)
  3. Ensure tests are deterministic and isolated
  4. Update coverage thresholds if needed
  5. Document complex test scenarios

Modifying Test Infrastructure

  1. Update configuration files as needed
  2. Test changes locally before committing
  3. Update documentation for new features
  4. Verify the local GitHub Actions policy guard still passes
  5. Consider backward compatibility

Performance Optimization

  1. Profile test execution to identify bottlenecks
  2. Optimize Docker image sizes and startup times
  3. Use caching effectively on local verification hosts
  4. Balance thoroughness with execution speed
  5. Monitor resource usage trends