Implementation milestones

[lo1tuma]

Implementation Milestones

• Monorepo Foundation

  • Workspace Layout Implement workspace layout foundation #195
  • Dependencies And Tooling: update dependencies, scripts, TS config, lint/test/build wiring, and CI for the new shared-source layout

• First Usable Engine

  • Public @overkill-dev/engine Entry Point: one supported import surface for defining suites/tests and executing them without repo-internal imports
  • Direct Engine Execution Flow: create suites/tests, execute them through the public API, and get a result back
  • Structured Outcomes And Run Result: stable Pass / Fail / Skip / Inconclusive protocol plus run summary/result shapes
  • Stable Identity And Run Counts: engine-owned test/case identity, discovered vs executed counts, and orphan detection
  • Reporter Event Stream Contract: engine-owned structured event surface for reporters and machine consumers
  • Engine Self-Tests Through Public API: rewrite engine tests so they exercise the public engine entry point, not internal source paths

• First Human Output

  • Reporter Lifecycle And Sink Semantics: real-time vs final-result reporters plus sink declarations as required by the concept
    • Define RealTimeReporter and FinalResultReporter as separate lifecycles
    • Require declared reporter sinks and validate direct-engine conflicts before run-start
    • Document lifecycle and sink behavior on the engine package surface
  • @overkill-dev/reporter-line: first standalone human reporter package exposed through its own facade
  • Human Failure Rendering: useful terminal rendering for pass/fail/summary output
  • @overkill-dev/reporter-dot: minimal progress reporter for compact local/CI output

• Structured Assertions

  • Engine-Owned Assertion Protocol: keep assertion semantics, injected case.assert / case.require, and assertion counting in @overkill-dev/engine
  • Built-In Assertion Catalog: first useful built-in assertion set
  • assert / require Semantics: define non-gating vs gating assertion behavior
  • done(), plan(n), And Zero-Assertion Detection: explicit assertion-count contract and default no-assertions failure
  • Custom Assertion Protocol: decide the public shape for custom nodes, composite assertion boundaries, and foreign assertion bridges
  • Assertion Source Locations: capture call-site locations for built-in, custom, composite, and foreign assertion failures
    • Shared Source Location Relativization: centralize project-root path rendering so every reporter benefits from one path policy
    • Adapt reporters to display source locations
    • Helper And Macro Location Forwarding: provide an explicit API for wrapped assertions to report the caller location
  • Failed Check And Diff Model: structured mismatch payloads and diff shapes
  • Bounded Assertion Serialization: cap depth, node count, entry count, string bytes, and serialized bytes before reporter delivery
  • Error Matcher Contract: explicit matcher shape for throws / rejects
  • @overkill-dev/assert: reusable assertion-extension helpers such as defineCompositeAssertion(...), foreign-assertion bridges, and packaged assertion adapters

• Doubles

  • @overkill-dev/doubles: first doubles package facade
  • testDouble(): primary public double creation API
  • Rule-Based Behaviors: configurable responses and behavior rules
  • Typed Pattern Matchers aka integrate ts-pattern
  • Call History Introspection: inspect calls/results directly
  • Doubles Assertions: doubles-oriented assertion extensions plugged into the engine-owned assertion context

• First Dogfooding And Publishing

  • Publish @overkill-dev/engine: first real package publication from the new layout once the engine flow is usable
  • Publish @overkill-dev/reporter-line: first reporter publication once it works end to end
  • Publish @overkill-dev/assert: publish reusable assertion-extension helpers once the extension model is usable
  • Publish @overkill-dev/doubles: publish the doubles package once the core API is usable
  • Dogfood Through Engine API: use the published engine, assertion-helper, doubles, and reporter packages in Overkill’s own tests/tools directly
  • Remove Internal-Only Test Wiring: stop depending on repo-internal execution paths where the published engine API is sufficient

• Minimal Runner

  • @overkill-dev/run: first orchestration package facade
  • RunRequest, ResolvedRun, And RunFacts API: typed request, resolution, and serializable run facts surface mirrored by the CLI
  • Reporter Completion Errors: decide how run-end, onFinish, and final-result reporter failures are surfaced after RunResult is created
  • Command-Line Business Logic API: parser, command intent model, config loading flow, command dispatch, fallback error reporting, and exit-code mapping are owned by @overkill-dev/run
  • Command-Selected Module Loading: command modules load only for the selected command path so ordinary microtest startup does not load benchmark, baseline writer, coverage, or reporter-heavy modules
  • Named Profile Model: support choosing the runner profile used by a run request, including the existing microtest profile
  • Reporter Sink Exit Codes: map configuration failures such as sink conflicts to stable CLI exit codes
  • Managed Output Sink Coordination: validate raw vs managed stdout / stderr reporter sinks, render managed line intents through the default plain renderer, and add @overkill-dev/reporter-brief plus @overkill-dev/output-renderer-github-actions
  • Resource Budget Resolution: carry Node-first heap, RSS, resident-growth, and active-resource budgets through grouped resourceUsage policy in config, RunRequest, and RunFacts
  • Supervised Resource Exhaustion Attribution: enforce budgeted microtest profiles through processModel: 'supervised-process', stream active test identity, kill wedged children on hard timeout or budget breach, and record resource-exhausted with the active case
  • Run A Specific File Or Module: one complete public run flow for explicit inputs
  • Internal Test Export Cleanup: migrate Overkill test modules from exported testSuite to the public testNode convention
  • Full Config Loading: finish config discovery, import, schema validation, and any final decision on default vs named exports for defineConfig
  • Runner Profile Naming Model: support project-owned runner profile names such as backend-http or ui-browser without deriving behavior from the name
  • Test Family Discriminator: require profiles.<name>.testFamily and use it as the discriminant for profile config
  • Microtest Profile Validation: validate the initial testFamily: 'microtest' profile surface and reject fields that are not valid for microtests
  • Microtest Execution Model: model processModel as in-process or supervised-process for microtests
  • Microtest Scheduling Model: model concurrent and serial scheduling for microtests
  • Microtest Capability Restrictions: implement capability restrictions for microtest profiles only in the minimal runner; property profiles adopt the same restricted boundary later, while integration, browser, benchmark, and type-test profiles do not use this model
  • Cross-Process Custom Engine Boundary: design custom engine execution for supervised and other multi-process runs without parent-side user-module execution
  • Profile Reporter Resolution: support profile reporter overrides with global reporter fallback
  • CLI Fallback Diagnostics: return runner errors not sent to terminal-capable reporters, and map completed runs to the documented non-zero CLI exit status
  • CLI Run Command Surface: expose the @overkill-dev/run command implementation consumed by the @overkill-dev/test binary wrapper, keep bin: overkill out of @overkill-dev/run, write returned fallbackDiagnostics to stderr, apply the returned exit code, and parse textual run flags including --measure-resource-usage and --resource-budget <name=value> for activeResourceCount, javaScriptEngineHeapBytes, residentSetBytes, and residentSetGrowthBytesPerSecond

• Discovery, Metadata, And Selection

  • Explicit Module Listing: list explicit test files/modules through the runner
  • Profile File Discovery: implement files.include plus files.exclude with Node glob's separate exclude support; do not support negated include globs as a second exclusion mechanism
  • Metadata Model And Propagation: tags, kind, runtimes, capabilities, stability, ownership, and subtree propagation rules
  • Programmatic Selection API: typed filter helpers for IDEs, MCP servers, and code-based integrations
  • CLI Filter Grammar: stable selection/filter syntax for the runner
  • Selection And Filtering: select tests by file/name/id/tag/runtime/stability and similar stable handles
  • Ordering And Seeding: deterministic ordering plus seeded variation
  • Sharding: stable identity-based partitioning of the filtered run set

• Resources And Runtime Composition

  • @overkill-dev/resources: typed runtime/resource composition package
  • Typed Context Composition: compose runtimes/resources into test context explicitly
  • Lifecycle Scopes And Teardown: setup/teardown and sharing/isolation rules
  • Resource Lifecycle Scopes: support per-run, per-file, per-suite, per-case, and shared-per-worker lifetimes
  • Runtime Matrices: explicit runtime dimensions that feed identity and planning
  • Local Service Resources: model deterministic app servers, registries, browser test servers, and other local services as explicit resources
  • Runtime Scenarios: support named scenarios or runtime presets that affect runtime identity, artifacts, and replay metadata
  • Interaction Transcripts: support generic transcript recording for calls, HTTP, browser requests, subscriptions, and custom protocols
  • Runtime Attachments: allow fixtures and runtimes to attach structured artifacts such as screenshots, accessibility JSON, service logs, and transcripts
  • Execution Requirements Contribution: runtimes/resources can influence execution strategy without owning the runner

• Default Test Authoring

  • @overkill-dev/test Root Authoring Facade: add the high-level authoring surface on top of the existing binary package
  • @overkill-dev/test Binary Wrapper: ship the only first-party bin: overkill and delegate command behavior to @overkill-dev/run
  • Output-Capture CLI Flags: decide and implement the minimal textual surface for capture mode, including whether --no-capture should exist or remain config-only
  • File-Level Metadata Authoring: decide explicit module metadata export or root authoring semantics now that file-frame metadata exists internally
  • Standard Distribution Subpaths: expose config, reporters, bench, assert, resources, and baselines through explicit subpaths while preserving leaf package ownership
  • test(...) And suite(...): implement the first default authoring API on top of engine and export it from @overkill-dev/test
  • table(...): implement the first parameterized authoring surface and export it from @overkill-dev/test
  • defineMacro(...): implement the first preferred reuse mechanism for tests-as-values and export it from @overkill-dev/test
  • createTestFacade(...): implement authoring-surface composition without taking over assertion ownership and export it from @overkill-dev/test
  • runIfMain(...): supported direct-file execution companion for exported specs - already exists partially in engine, we should consider moving it to @overkill-dev/test package so we can also rely on default reporters and so on. But we probably also need to decide which testFamily and which profile is picked when using runIfMain
  • Lightweight Doubles Root Exports: expose lightweight doubles APIs and doubles assertion references from @overkill-dev/test
  • Root Import Boundary: after root authoring APIs exist, verify the root export does not expose or load config, reporters, resources, benchmark, baseline, coverage, or command implementation APIs
  • throwingTest: decide whether throwable test authoring belongs in the root facade or a dedicated compatibility export, then implement the chosen surface
  • skippedTest: refine the concept for skip authoring, including whether it applies to every test family or only beyond microtests, then implement the chosen surface
  • defineHarness(...): implement the authoring helper after the root facade and core test(...) / suite(...) APIs exist
  • Interaction Transcript Helpers: implement recordCalls(...), recordSink(...), and recordEvents(...) once transcript semantics are ready
  • Async-Control Helpers: implement scope.microtasks(), scope.immediate(), scope.flushAsync(), and scope.inFlight(...) as one async-control authoring milestone
  • Stable Suite Aliases: project-local re-export pattern such as #tests/micro
  • Standard Distribution Documentation: normal install path is npm install -D @overkill-dev/test

• Coverage

  • Microtest-Only Coverage Model: explicit, off-by-default coverage restricted to microtest profiles and activated through profile selection, not an ordinary --coverage run flag
  • V8 Coverage Integration: runner-owned native coverage flow and artifact emission
  • Coverage Config Surface: profile-scoped formats/include/exclude/thresholds/outputDir in microtest config
  • Coverage Profile Validation: reject coverage config on non-microtest profiles
  • Coverage Run Behavior: serial coverage execution for coverage-enabled microtest profiles and run-record integration

• Replay, Records, Baselines, And Debug

  • Run Records: persist RunRecord when the workflow needs durable run state
  • Last-Failed Selection: resolve --last-failed from persisted RunRecord failure case ids
  • Witness File Format: stable witness schema for replayable failures
  • Replay Commands: replay a recorded run or witness through public surfaces
  • @overkill-dev/baselines: shared baseline model and package facade
  • Baseline Verbs: update/apply/bootstrap/diff/list workflows owned by @overkill-dev/run
  • Snapshot Workflow: checked-in baseline compare/update flow
  • Failure Artifacts: structured logs, traces, snapshots, and artifact attribution rules
  • Debug Mode: richer diagnostics and retained runtime state for investigation

• Additional Reporters

  • @overkill-dev/reporter-tap: TAP-compatible real-time output package
  • @overkill-dev/reporter-json: canonical machine-readable final-result package
  • @overkill-dev/reporter-html: generic artifact/failure HTML report package
  • @overkill-dev/reporter-benchmark-html: benchmark-specific HTML report with workload/baseline/machine metadata

• Benchmarking

  • @overkill-dev/bench: first benchmark package facade
  • Standard Distribution Integration: include @overkill-dev/bench in the @overkill-dev/test standard dependency set while keeping benchmark measurement outside the root authoring import and ordinary microtest startup path
  • Benchmark Command Namespace: implement overkill bench run and overkill bench list
  • Benchmark Config Namespace: load benchmark.profiles.<name> from the project config without mixing benchmarks into runner profiles
  • Benchmark Profile Selection: select benchmark profiles from benchmark commands without supporting overkill run --profile benchmark
  • Benchmark Baseline Namespace: implement overkill bench baseline update/apply/bootstrap/diff/list for performance baselines
  • Workload Model: define benchmark inputs/workloads explicitly
  • Measurement Engines: collect timing and related metrics through explicit measurement surfaces
  • Calibration And Normalization: normalize benchmark results for comparability
  • Budget And Baseline Policy: compare results against checked-in expectations
  • Process And PTY Benchmarks: external process and terminal-oriented benchmark support
  • slo(...): explicit budget-oriented benchmark API
  • Dogfooded Startup Guardrails: after benchmark commands are usable, add Overkill benchmark checks for root @overkill-dev/test import cost and ordinary microtest CLI startup

• Integration Test Family

  • Integration Profile Model: support testFamily: 'integration' as a later runner profile family
  • Integration Runtime Environment: define temp directories, local services, process use, network use, and failure artifacts without applying microtest capability restrictions
  • Integration Resource Defaults: model app servers, registries, HTTP/IPC workflows, and richer fixtures as explicit resources
  • Integration Retry Policy: allow integration profiles to opt into retries with attribution-preserving artifacts
  • Integration Failure Artifacts: support transcripts, logs, service output, and scenario witnesses for integration resources

• Advanced Runner Execution

  • Worker Pool Execution: support processModel: 'worker-pool' for families that allow pooled workers
  • Process-Per-File Execution: support processModel: 'process-per-file' for integration-style file isolation
  • Supervised Integration Execution: support processModel: 'supervised-process' for integration profiles when a disposable child boundary is required
  • Single-Worker Serial Execution: support one dedicated worker or process for benchmark-like and resource-constrained runs
  • Worker Count Controls: implement default worker sizing, host caps, and --workers N
  • Frozen Plan Assignment: collect once in the orchestrator, freeze RunFacts, and assign planned case identities to workers or subprocesses without worker-side rediscovery
  • Execution Plan Resolution: resolve final execution from profile config plus resource/runtime execution requirements

• Higher Test Layers

  • withSimulation(...): first simulation-oriented runtime helper
  • @overkill-dev/property: property-testing package facade
  • Property Test Family: first-class testFamily: 'property' profile model with microtest-like boundaries
  • Property Capability Boundary: reuse the microtest capability-restriction model for property profiles; do not apply that model to integration, browser, benchmark, or type-test profiles
  • Property Profile Policy: separate property timeouts, generated-input witnesses, shrinking state, corpus replay, and no coverage
  • Relation-Style Property Checks: expose metamorphic relation helpers through @overkill-dev/property
  • @overkill-dev/model: model-based testing package
  • @overkill-dev/differential: differential testing package
  • @overkill-dev/linearizability: concurrency/consistency package
  • linearizability(...): first concurrency/consistency helper

• Builder Layer

  • Builder-Oriented Composition Surface: preserve an explicit layer for assembling higher-level testing APIs from lower-level contracts
  • Third-Party DSL Support: allow specialized test APIs to be built without forking the runner or engine

• Standard Distribution And Optional Packages

  • Standard Distribution Model: @overkill-dev/test is the one-package normal user install and standard distribution
  • Standard Stack Dependencies: @overkill-dev/test depends directly on the standard stack so users do not align package versions by hand
  • No Preset Packages: remove @overkill-dev/micro, @overkill-dev/default, @overkill-dev/integration, and @overkill-dev/all from the concept
  • Optional Package Policy: browser, property, model, differential, linearizability, type-test, mutation, ESLint, AWS CDK, contract, accessibility, and similar domain packages remain separate installs
  • Explicit Optional Integration: optional packages integrate through typed profiles, runtime/resource factories, reporters, baseline adapters, or authoring helpers
  • No Dynamic CLI Extension: installing an optional package does not mutate the overkill CLI, trigger package-name discovery, or widen top-level config through module augmentation

• Ecosystem Integrations

  • Type-Test Adapters: support TS type-test workflows through integrations rather than a custom engine
  • Stryker Plugin: mutation testing integration
  • @overkill-dev/eslint-rule-test: ESLint rule-testing package
  • @overkill-dev/eslint-plugin: static authoring rules such as orphan detection and related checks
  • @overkill-dev/aws-cdk: first package-specific assertion adapter wrapping @aws-cdk/assertions
  • Contract Adapters: other adapters for external assertion/testing ecosystems
  • Stable Machine-Readable APIs: keep reporter/results/planning data usable for IDE and MCP integrations

• Browser And Remote Expansion

  • Browser Runtime Package: first optional browser-oriented runtime/package
  • Browser Runtime Integration: browser support integrates through fixtures/resources and explicit project profiles, not a browser test family or CLI mutation
  • Visual Baselines: checked-in visual comparison artifacts
  • Accessibility Attachments: accessibility-oriented artifacts/reports
  • Browser CLI Extension Deferred: use project-named profiles such as ui-browser with ordinary overkill run, overkill list, and baseline verbs until a dedicated command need is proven
  • Remote Execution Planning: model remote/distributed execution in orchestration
  • Distributed Result Merging: merge results from remote/distributed runs