Skip to content

TESTING.md

Latest commit

 

History

History
242 lines (180 loc) · 14.8 KB

TESTING.md

File metadata and controls

242 lines (180 loc) · 14.8 KB
Raw

Testing Guide

For LLM agents: Load the writing-tests skill (.opencode/skills/writing-tests/SKILL.md) before writing or modifying any test file. It contains the full mock isolation rules, CI pipeline structure, and anti-patterns. For agent operational safety and the broader engineering invariants of this repo (especially the test_runner broad-scope restriction and the _internals DI-seam pattern for mock isolation), read AGENTS.md at the repo root.

Writing stable tests? See docs/testing/test-stability.md — the runbook for the four root-cause classes of merge-group flaky tests (time-sensitive, coverage-sensitivity, cross-platform, subprocess) and the freezeClock / withIsolatedState helpers that prevent them. The bun run check:test-clock lint (CI quality job) fails on NEW time-touching tests that don't use the helper.

⚠️ Do NOT use the OpenCode test_runner tool to validate the full repo. It is for targeted agent validation with explicit files: [...] or small targeted scopes. scope: 'all' is gated behind the SWARM_ALLOW_FULL_SUITE=1 env var (intended for opt-in CI mirrors only; there is no allow_full_suite arg). Broad scopes can stall or kill OpenCode before the MAX_SAFE_TEST_FILES = 50 guard in src/tools/test-runner.ts fires. For repo validation, use the shell commands below — per-file isolation loops match CI behavior. See AGENTS.md invariant 6 for the full contract.

Quick Reference

Framework

All tests use bun:test. No Jest, Vitest, or other frameworks.

import { describe, test, expect, beforeEach, afterEach, mock, spyOn } from 'bun:test';

Running Tests

# Single file (always safe)
bun --smol test tests/unit/tools/diff.test.ts --timeout 30000

# Per-file loop (required for tools, services, agents — prevents mock poisoning)
for f in tests/unit/tools/*.test.ts; do bun --smol test "$f" --timeout 30000; done

# Batch run (safe only for directories without mock conflicts)
bun --smol test tests/unit/hooks --timeout 120000
bun --smol test tests/unit/cli --timeout 120000

Reproducible repository validation

Issue #2675's canonical validation command runs the complete, ordered surface inventory and writes the bounded JSON report under .swarm/:

# Full validation (all surfaces)
bun run validate:repo -- --mode full --diff-base origin/main

# Pull-request diff validation (no changed work is a distinct no-op result)
bun run validate:repo -- --mode diff --diff-base origin/main

In diff mode, the selected changes are the committed branch changes in <diff-base>...HEAD; unstaged and untracked work is excluded. The ten surfaces are quality, unit, integration, security, coverage, memory-recall-regression, package-check, smoke, php-validation, and rust-sandbox-runner. The report is schema-versioned (schemaVersion: 1) and records the runtime (bunVersion, platform, arch), exact array-form argv and cwd for every item, the origin/main diff base, per-surface results, and summary counts. A process must end in one of passed, failed, crashed, timed_out, missing, or skipped; a crash, timeout, missing item, or skipped item cannot be reported as a complete pass. The run-level status is passed, failed, incomplete, or no_op; no_op is reserved for a diff run with no discovered work and is not equivalent to passed.

Use --surface <name> (repeatable) or comma-separated --surfaces <name,...> to validate a deliberate subset; omitting both selects the full ten-surface matrix. If a selected item requires an unavailable runtime, it is recorded as skipped with a reason and the run is incomplete (nonzero exit), so a full matrix run never claims a pass while a required runtime is missing.

The default safety bounds are a 120000 ms test timeout, 180000 ms per-item wall timeout, 900000 ms whole-run timeout, and 65536 output bytes per channel (after redaction). The runner preloads scripts/ci/bun-32056-keepalive.ts and preserves each test-file path as a separate argv element, including spaces and quotes.

The historical 659/3,389 count is an unconfirmed historical count because its raw source/provenance was not retained; it must not be presented as a reproducible current validation result. New reports retain the raw per-item terminal statuses, timing, signal, bounded redacted output, and cleanup outcome needed to audit a run.

Do not run bun --smol test tests/unit/tools or tests/unit/hooks as a single batch. Mock modules leak across files in Bun's --smol mode, causing false failures. The CI uses per-file isolation loops for the 15 mock.module hook files (step 1a) and steps 4-6 (tools, services, state/agents), while other hook tests remain in batch groups (step 1b).

Bun v1.3.13+: The --isolate flag is available for local development to run each test file in a fresh global environment. However, CI currently uses --smol with per-file isolation loops, which achieves the same mock isolation goal. You may use --isolate locally, but the CI pipeline will continue using --smol with per-file loops for consistency.

Mock Isolation

Bun's --smol mode shares module cache between test files. A mock.module() call replaces the module globally for all files in the same process.

mock.restore() does NOT reliably restore mock.module mocks in Bun v1.3.11. Three layers of defense are required.

Always spread the real module when mocking Node built-ins:

import * as realChildProcess from 'node:child_process';
const mockExecFileSync = mock(() => '');
mock.module('node:child_process', () => ({
  ...realChildProcess,           // preserve ALL exports — mandatory
  execFileSync: mockExecFileSync, // override only what you need
}));

Always add afterEach(mock.restore()) for cross-module mocks. Even though unreliable in Bun v1.3.11, it provides best-effort cleanup. Exception — Windows EBUSY: Test files that spawn async child processes (pre-check-batch suite) must NOT call mock.restore() on Windows. Child process handles hold directory locks and trigger EBUSY errors. Skip affected tests with test.skipIf(process.platform === 'win32').

Mock cleanup enforcement: bun run check:mock-cleanup runs in CI (quality job) and enforces two checks:

  1. All mock.module calls have afterEach(mock.restore()) cleanup or file-scoped mockClear/mockReset pattern
  2. All mock.module('node:*', ...) calls spread real exports (e.g., ...realFs) to prevent test pollution

Allowlist growth ratchet (issue #1666): bun run check:invariants Check 4 ratchets scripts/mock-allowlist.txt closed against unapproved growth. Adding a new mock.module target requires a matching standalone marker line # APPROVED-NEW: <normalized-target> in the allowlist (preserved across regen by scripts/generate-mock-allowlist.sh). MOCK_ALLOWLIST_ENFORCE=0 soft-warns for a deliberate growth PR. Prefer the _internals DI seam for new code.

Run locally before pushing:

bun run check:mock-cleanup
bun run check:invariants
bun run check:pre-push

check:pre-push is the authoritative cross-platform aggregate for enforced repository drift and retention-registry citation validation. Unlike the plain drift:check report command, it exits nonzero for blocking drift.

Intentionally skipped on Windows (async child process handles cause EBUSY):

  • tests/unit/tools/pre-check-batch.test.ts
  • tests/unit/tools/pre-check-batch.adversarial.test.ts
  • tests/unit/tools/pre-check-batch-cwd.test.ts
  • tests/unit/tools/pre-check-batch-cwd.adversarial.test.ts
  • tests/unit/tools/pre-check-batch-contextdir-adversarial.test.ts
  • tests/unit/tools/pre-check-batch-sast-preexisting.test.ts
  • tests/unit/tools/pre-check-batch-secretscan-evidence.test.ts

Use lazy binding in source code so mocks can intercept:

// Good — mockable
import * as child_process from 'node:child_process';
function run() { return child_process.execFileSync('git', ['status']); }

// Bad — binds at load time, mock can't intercept
import { execFileSync } from 'node:child_process';

CI Pipeline Steps

Step Directories Isolation
1a hooks (mock.module files — 15 files) Per-file isolation (dedicated step)
1b hooks (remaining groups) Per-file loop per group
2 cli Batch
3 commands, config Batch
4 tools Per-file loop
5 services, build, quality, sast, sbom, scripts Per-file loop
6 adversarial, agents, background, context, diff, evidence, git, helpers, knowledge, lang, output, parallel, plan, session, skills, types, utils Per-file loop

Test File Size Limits

To prevent monolithic test files that cause mock isolation issues and slow CI:

  • Maximum 500 lines per test file (enforced in CI by scripts/check-test-file-cap.ts as a diff-scoped ratchet: new over-cap files and existing over-cap files that grew fail the gate; pre-existing over-cap files untouched by the PR are non-blocking; TEST_CAP_ENFORCE=0 soft-warns). Run the same gate locally on any platform, Windows included, with bun run check:test-file-cap.
  • delegation-gate.test.ts was split into 45 focused files (FR-006 SC-006.1) — all under 500 lines
  • When a test file exceeds 500 lines, split it by behavior/feature into focused files

New Behavioral Test Files (Phase 3–4 — Issue #1231 Structural Debt)

Phase 3 files:

File Tests Coverage
tests/unit/commands/sync-plan.test.ts 10 FR-007 (sync-plan command)
tests/unit/agents/sme.test.ts 24 (75% parameterized) FR-008 (SME delegation)
tests/unit/parallel/lean-turbo-acquire-locks.test.ts 14 FR-009 (Lean Turbo locking)
tests/unit/parallel/lean-turbo-plan-lanes.test.ts 16 FR-009 (lane planning)
tests/unit/parallel/lean-turbo-review.test.ts 13 FR-009 (Lean Turbo review)
tests/unit/parallel/lean-turbo-runner-status.test.ts 18 FR-009 (runner status)
tests/unit/tools/generate-mutants.test.ts 11 FR-009 (mutation testing)
tests/unit/config/set-qa-gates.test.ts 19 FR-009 (QA gate config)
tests/unit/config/get-qa-gate-profile.test.ts 9 FR-009 (QA gate profile)

Phase 4 files (FR-010/011/012 — previously untested hooks):

File Tests Coverage
tests/unit/hooks/conflict-resolution.test.ts — FR-010 (conflict-resolution hook)
tests/unit/hooks/curator-types.test.ts — FR-010 (curator types)
tests/unit/hooks/curator.test.ts — FR-010 (curator consolidated)
tests/unit/hooks/delegate-ack-collector.test.ts — FR-011 (delegate-ack-collector hook)
tests/unit/hooks/delegate-directive-injection.test.ts — FR-011 (delegate-directive-injection hook)
tests/unit/hooks/knowledge-reinforcement.test.ts — FR-011 (knowledge-reinforcement hook)
tests/unit/hooks/normalize-tool-name.test.ts — FR-011 (normalize-tool-name hook)
tests/unit/hooks/phase-complete-directive-gate.test.ts — FR-011 (phase-complete-directive-gate hook)
tests/unit/hooks/phase-directives.test.ts — FR-011 (phase-directives hook)
tests/unit/hooks/semantic-diff-injection.test.ts — FR-011 (semantic-diff-injection hook)

Phase 4 also consolidated knowledge-curator tests with shared fixtures (tests/unit/hooks/curator-test-fixtures.ts) and completed the vitest→bun:test migration across all 11 directories (cli, services, session, evidence, commands, build, lang, scripts, config, knowledge, context-map, hooks, tools).

Coverage Gate

CI enforces a minimum line-coverage threshold of 65.00% (recalibrated in issue #1778 H4 when the measured set grew to src/** + the orphan test trees; measured 73.41% at the time) on the merge queue. Coverage is measured using bun test --coverage with output configured in bunfig.toml:

# bunfig.toml
[test]
coverageReporter = ["lcov", "text"]
coverageDir = "./coverage"

The coverage gate is a required status check but only runs on merge_group events (not on every PR, for speed). Since issue #2341 it is sharded: a coverage-shard matrix (6 ubuntu shards, the same round-robin partition as the unit job) each measures its partition per-file under coverage and uploads its merged lcov, and the dependent coverage aggregator job merges all shard reports and enforces the threshold once over the union, failing closed if any shard report is missing — a dropped shard can never silently shrink the measured set. To measure coverage locally (unsharded, full set + inline threshold):

bun test --coverage tests/unit/ --timeout 60000

The coverage report is output to ./coverage/ as lcov.info and text summary.

Adversarial Tests

Adversarial tests (tests/adversarial/) verify security boundaries against crafted malicious inputs. They cover:

  • FR-003 / SC-003.1–SC-003.7: Subprocess injection — command injection vectors via shell.safeify, bunSpawn routing, PATH traversal, null-byte injection, argument injection
  • FR-004 / SC-004.1–SC-004.4: Guardrail bypass attempts — prompt injection, capability escalation, schema override, context capsule exfiltration
  • FR-005 / SC-005.1–SC-005.3: Evidence spoofing — plan mutations, phase伪造, retrospective fabrication

Run adversarial tests with the same isolation rules as unit tests:

# Per-file loop (required — adversarial tests may mock global state)
for f in tests/adversarial/*.test.ts; do bun --smol test "$f" --timeout 30000; done

Adversarial tests use _internals DI seams for mocking (avoiding mock.module cross-file leakage):

  • src/hooks/knowledge-migrator.ts:_internals — exposes writeSentinel, mkdir, writeFile, existsSync, readFileSync, readFile for evidence-spoofing tests
  • src/evidence/manager.ts:_internals — exposes validateEvidence for evidence integrity tests

Cross-Platform

  • Use path.join(), never string concatenation with /
  • Use os.tmpdir(), never hardcoded /tmp
  • Mock validateDirectory from path-security.ts when tests use Windows temp paths
  • On Windows, if %USERPROFILE% (or another ancestor of %TEMP%) already contains both .swarm/ and a project indicator such as .opencode/, evidence-writing tests can fail the .swarm containment guard before they reach your temp project root. For per-file tests/unit/tools/*.test.ts publication evidence, either clear the parent .swarm/ contamination first or run the temp-root setup from a directory tree that is outside that contaminated ancestor chain.

Full Details

See .opencode/skills/writing-tests/SKILL.md for the complete guide including:

  • All mock isolation rules and patterns
  • File placement conventions
  • Test quality standards (DO and DO NOT)
  • Cross-platform process spawning rules
  • Pre-submission checklist