Replace fragmented tests with whole-product Rust conformance

[santhreal]

Problem

Veyyon's verification surface is fragmented across package-local TypeScript tests, ad hoc simulations, direct module replacements, source-oriented checks, and a small number of compiled-product probes. The pieces answer different questions, but they do not form one conformance system that can prove the shipped product behaves consistently across providers, tools, sessions, rendering, persistence, workers, configuration, and distribution.

The OpenAI-compatible clean-EOF defect demonstrates the gap. Existing simulations replaced provider modules after wire parsing and therefore could not reproduce a provider that emitted complete SSE output and closed the HTTP body without finish_reason or [DONE]. The normalized event tests remained green while production models failed. Similar gaps exist wherever tests bypass transport decoding, process boundaries, persistence, scheduling, or the compiled binary.

This issue tracks a clean cutover to one whole-product Rust conformance system. It is not a request to add another test layer beside the existing ones indefinitely. Every old contract must move into the new system, prove parity, and then leave the TypeScript/simulation surface.

Current detailed design: docs/internal/whole-product-rust-conformance.md.

Draft review surface: #878.

Required outcome

Create crates/veyyon-conformance, migrate production logic to Rust before claiming direct Rust coverage, exercise remaining TypeScript behavior through the compiled Veyyon product, fold the useful scenarios from packages/simulations into the conformance corpus, and delete the superseded TypeScript tests and simulation package after one-for-one contract parity is certified.

The finished system must:

• Materialize exactly 250,000 unique JSONL conformance cases.
• Enforce exactly 4,496 structured expected-error contracts.
• Execute at least 1,200 mutations and kill at least 1,000.
• Materialize exactly 245,000 direct-Rust cases and 5,000 compiled-product cases. Compiled cases launch the unmodified release artifact through process, PTY/ConPTY, real-workspace, and loopback-network boundaries.
• Enumerate registries and exported state spaces at runtime so new providers, tools, settings, routes, workers, and protocols fail closed until covered.
• Produce deterministic replay bundles with seed, input trace, filesystem fixture and observations, provider transcript, PTY stream, expected result, actual result, and minimized reproducer.
• Preserve no parallel fake implementation of production behavior.

Design principles

Production path, not a stand-in

A conformance case either calls migrated production Rust directly or launches the compiled Veyyon binary. Test helpers may emulate external systems such as an HTTP provider, filesystem faults, terminal input, and clocks. They must not reproduce provider parsers, session policy, tool dispatch, configuration resolution, or persistence logic.

Observable contracts

Assertions operate on:

• stdout and stderr bytes;
• exit status and bounded termination;
• emitted protocol packets;
• PTY cell grids and rasterized frames;
• filesystem and SQLite state;
• session trees and persisted messages;
• provider requests and responses;
• tool execution side effects;
• structured errors, retry decisions, and deadlines.

Internal call counts, private fields, implementation source text, and mock invocation assertions are not contracts.

Fail closed on new members

Each suite derives its member space from the production registry, enum, union, descriptor table, command registry, or generated schema. A newly added member must make conformance red until it receives an explicit policy and cases. Opt-outs are exact named sets and a second exemption changes the expected set.

Deterministic faults

Direct-Rust cases use virtual monotonic time and deterministic scheduling at the production crate's external I/O traits. Compiled-product cases use the production system clock, deterministic external event scripts, short real deadlines, and assertions for termination plus an upper bound. No compiled-product proof depends on syscall interposition, dynamic-library injection, privileged clock control, or exact elapsed time.

Independent oracles

Expected results come from declarative contracts, algebraic invariants, protocol standards, and state machines. They must not call the production function they are checking or derive expected output by observing the implementation.

Architecture

crates/veyyon-conformance owns:

• src/corpus/: canonical JSONL schema, semantic hashes, deduplication, sharding, and replay.
• src/generator/: grammar, boundary, combinatorial, pairwise, and property-based generators.
• src/oracle/: exact outcomes, algebraic invariants, protocol validators, and state-transition models.
• src/vmock/: deterministic cross-platform loopback HTTP/1.1 and HTTP/2 provider server with raw SSE streaming and transport faults.
• src/vfs/: trait-backed copy-on-write filesystem and fault injection for direct Rust; content-addressed real-workspace fixtures for compiled-product execution.
• src/vpty/: POSIX PTY and Windows ConPTY driving plus VT100/xterm parsing, terminal resize, ANSI capture, and cell-grid state.
• src/vclock/: virtual monotonic time and deterministic scheduling for direct Rust; bounded real-time contracts for compiled-product execution.
• src/render/: cell-grid comparison and raster proofs on grey (#1e2127) and black (#000000) grounds.
• src/model_check/: session, tool lifecycle, queue, lock, and worker state machines.
• src/fuzz/: libFuzzer and AFL++ targets for raw parsers and wire formats.
• src/mutation/: AST and bytecode mutation execution, sensitivity accounting, and critical-path zero-survivor rules.
• src/shrink/: hierarchical minimization of event traces, payloads, schedules, VFS trees, and terminal dimensions.
• src/report/: JUnit, SARIF, JSON summaries, replay bundles, and coverage inventories.

The harness launches the same compiled entrypoint users run. Worker selectors re-enter that entrypoint. Provider base URLs point to vmock over TCP loopback, and an external-network deny guard fails unexpected connections. Configuration and credentials are supplied through isolated profile directories. Compiled cases use unique real temporary workspaces; syscall preloading is not part of the design. No production module is replaced after parsing.

The launcher waits for child exit and PTY/ConPTY closure before deleting a
workspace. Windows retries only access-denied and sharing-violation cleanup
errors with bounded exponential backoff capped at five seconds per workspace;
other errors fail immediately, and a shard cannot pass with a nonempty cleanup
queue.

The manifest fixes the execution-target allocation at 245,000 direct-rust
cases and 5,000 compiled-product cases. It requires compiled coverage in every
subsystem and every source-enumerated boundary family. Direct cases exhaust
algorithmic, state-machine, scheduling, and fault dimensions; compiled cases
prove process wiring and operator-visible behavior. A case runs only the target
declared in its record.

Platform allocation is exact: 240,000 direct cases are
platform-independent, and Linux x86_64, Linux aarch64, macOS x86_64, macOS
aarch64, and Windows x86_64 each receive exactly 1,000 platform-specific direct
cases and 1,000 compiled-product cases. Platform-independent cases run once.
Every platform's compiled allocation covers all sixteen subsystems and every
applicable source-enumerated boundary family.

Canonical case record

Every JSONL row is a versioned, self-contained ConformanceCase with:

• schemaVersion;
• canonical caseId;
• generator version and deterministic seed;
• subsystem and contract IDs;
• expected structured error ID or null;
• production target kind, entrypoint, and artifact digest;
• normalized dimensions such as API, terminal signal, output shape, framing, and fault;
• target-dependent platform, clock mode, filesystem fixture, and provider fixture;
• ordered stimulus records;
• exact oracle values for exit status, output, stop reason, error identity, deadline, side effects, persisted state, and tool execution;
• source-derived registry members and requirement IDs covered by the case;
• synthetic/generated provenance.

caseId is the BLAKE3 digest of canonical semantic inputs and the oracle. It excludes generator metadata, execution observations, and provenance, preventing two generators from claiming distinct coverage for the same behavior. Fixtures are content-addressed. Unknown schema versions, missing fixture digests, duplicate semantic IDs, count drift, and secret-bearing fixtures fail materialization. Execution results are separate immutable reports and never overwrite committed oracles.

The identity payload is exactly subsystem, contract, target.kind,
dimensions, environment, stimulus, and oracle. It excludes caseId,
generator metadata, entrypoint and artifact digests, coverage labels,
provenance, and execution observations.

Corpus allocation

Subsystem	Cases	Exact errors
Rendering and terminal UI	20,000	384
AI providers and streaming	24,000	480
Tool execution runtime	26,000	512
Session and tree engine	20,000	320
Persistence and Mnemopi	16,000	256
Concurrency and agent mesh	14,000	256
Security and sandbox	14,000	384
CLI engine and modes	16,000	256
Installers and distribution	10,000	192
Native services, workers, and subprocesses	12,000	192
Configuration and settings	12,000	192
Context and compaction	14,000	224
Memory engine and vectors	12,000	160
Editing and Hashline engine	16,000	288
LSP client and diagnostics	10,000	160
Wire protocol and Argot	14,000	240
Total	250,000	4,496

The builder rejects duplicate semantic hashes, corpus-count drift, and target/platform allocation drift.

CI uses exactly eight runners: four Linux x86_64 runners and one runner for each
of Linux aarch64, macOS x86_64, macOS aarch64, and Windows x86_64. The dispatcher
routes platform:any to the Linux pool, routes platform-specific cases only to a
matching runner, and hashes within that pool. Each runner enforces a fixed
four-slot compiled-product worker queue. A complete compiled case includes
launch, execution, child exit, PTY/ConPTY closure, and normal workspace deletion
or bounded Windows cleanup handoff; a complete direct case includes fixture
reset and oracle evaluation. Wave 0 establishes p95 limits of 1.5 ms for direct
cases and 500 ms for compiled cases on every applicable platform. Before each
wave enters CI, three consecutive cold runs of every exact shard must each finish
in <= 144 seconds. Nominal p95 work is 126.5 seconds on each non-Linux runner and
121.625 seconds on each Linux runner. A runner fails at 180 seconds or with a
nonempty cleanup queue. The manifest rejects target/platform drift, missing
runner eligibility, shard skew, or calibration failure.

Provider conformance requirements

Provider coverage must begin at raw HTTP bytes and continue through parsing, accumulation, terminal classification, retry policy, persistence, and tool execution.

Source-derived provider matrix

At runtime:

1. Enumerate every registered model and provider descriptor.
2. Resolve its production API and compatibility policy.
3. Require a conformance policy for every resolved API.
4. Sweep every provider that resolves to openai-completions, openai-responses, anthropic-messages, Google/Gemini, Ollama, or another shipped transport.
5. Fail when a new provider or API has no policy.

Model-specific behavior belongs in descriptor data, not duplicated bespoke tests.

Raw wire dimensions

Generate real provider bytes across:

• LF and CRLF SSE records;
• comments, empty records, and multiple data: fields;
• one event per transport chunk and many events per chunk;
• every meaningful JSON split boundary;
• every UTF-8 split boundary;
• [DONE], explicit terminal records, usage-only records, and clean body EOF;
• sockets that remain open after an authoritative terminal record;
• HTTP/1.1 chunking and HTTP/2 data frames.

Output shapes

Cross terminal conditions with:

• empty output;
• whitespace-only output;
• text;
• reasoning only;
• reasoning followed by text;
• one or multiple complete tool calls;
• missing tool ID;
• missing tool name;
• empty, truncated, malformed, primitive, array, and object arguments;
• interleaved text and tools;
• interleaved reasoning and tools;
• parallel tool-call deltas arriving out of order.

Terminal and fault conditions

Cover:

• finish_reason: stop;
• finish_reason: tool_calls;
• finish_reason: length;
• content filtering and provider refusal;
• usage before, with, and after terminal output;
• [DONE] with and without a semantic terminal record;
• clean HTTP EOF;
• empty EOF;
• ECONNRESET, ETIMEDOUT, DNS, TLS, and caller cancellation;
• first-event and next-event timeout;
• malformed SSE, truncated JSON, and invalid UTF-8;
• 401, 403, 408, 409, 429, 500, 502, 503, and 504 responses;
• Retry-After seconds and dates;
• retry exhaustion and replay-unsafe partial batches.

Provider invariants

• Clean EOF succeeds only for semantically self-contained output.
• Reasoning-only clean EOF enters bounded incomplete-output recovery rather than becoming a successful empty answer.
• Empty EOF remains retryable incomplete-stream failure.
• A transport exception never becomes clean EOF.
• No incomplete tool call reaches execution.
• Complete parallel tool batches preserve IDs, ordering, arguments, and replay safety.
• A terminal frame ends boundedly even if the server keeps the socket open.
• Retry policy preserves the original structured error and never duplicates committed output.
• Persisted history contains exactly the delivered attempt, not discarded retry fragments.

Fleet-derived regression intake

Production incidents are reduced to synthetic structural fixtures. Raw sessions and conversational content never enter the repository. Each fixture is identified by a semantic key over:

API + compatibility policy + terminal signal + output shape + framing + fault + expected outcome.

A new incident first checks for an existing semantic key. New shapes enter the generated corpus and receive a minimized replay case.

Other subsystem requirements

Tools

Enumerate the live tool registry. Exercise valid, invalid, partial, streaming, timed-out, canceled, and permission-gated calls through the compiled product. Assert schema rejection occurs before side effects. Assert every started call settles exactly once.

Sessions and persistence

Model fork, resume, branch switching, compaction, retry replacement, interrupted tool batches, stale serialized versions, and crash recovery. Persisted shape changes require version changes and stale-copy rejection cases.

Rendering

Drive real components. Compare ANSI bytes, terminal cell grids, and rasterized output on grey and black grounds. Sweep widths, heights, Unicode widths, tabs, long paths, raw control sequences, reduced motion, transcript rebuilds, and streaming previews.

Configuration

Generate defaults, profile values, project files, environment overrides, CLI flags, invalid values, and conditional settings. Assert precedence, persistence, restart behavior, hidden dependent settings, and loud invalid-value failure.

Workers and concurrency

Re-enter the compiled CLI worker host. Model-check startup, readiness, message ordering, cancellation, crash, restart, and shutdown. Assert deadlines and bounded termination. Run Loom/TSAN coverage for migrated Rust synchronization.

Security and distribution

Exercise path traversal, symlinks, permission boundaries, credential redaction, malicious ANSI, malformed archives, checksum mismatches, release asset selection, installer interruption, and stale update metadata. Security-boundary mutants allow zero survivors.

Mutation gate

Execute at least 1,200 distinct mutations, including:

• comparison boundary changes;
• conditional inversion;
• terminal-state deletion;
• authorization and validation deletion;
• timeout removal;
• retry/backoff changes;
• parser acceptance broadening;
• persistence version bypass;
• tool-execution-before-validation;
• sanitizer and path-guard removal.

The suite must kill at least 1,000 mutations overall and every mutation on credentials, path traversal, checksum verification, authorization, tool completeness, and persisted-version rejection.

Migration sequence

Foundation

• Create the Rust workspace crate and corpus schema.
• Implement deterministic seed, semantic hashing, sharding, reporting, and replay.
• Implement target-aware provider, filesystem, PTY/ConPTY, and clock boundaries without compiled-binary syscall interposition.
• Establish the compiled-product launcher; prove direct-Rust p95 <= 1.5 ms and compiled-product p95 <= 500 ms on every applicable platform, then require three cold runs of every exact shard at <= 144 seconds.

Stateless production logic

• Move codecs, wire parsers, hashline/edit parsing, width rules, and catalog classification into production Rust crates.
• Certify direct Rust conformance.
• Keep TypeScript callers as thin bindings during cutover.

Providers, persistence, memory, and LSP

• Move provider wire parsing and terminal classification to production Rust.
• Cover raw wire matrices through direct Rust and compiled-product paths.
• Move persistence and memory logic with versioned stale-state tests.
• Move LSP transport and diagnostic parsing with protocol framing and crash-restart cases.

Agent runtime

• Move tool lifecycle, session transitions, compaction, retry, and worker coordination.
• Enumerate live registries and state machines.
• Certify cancellation, deadlines, replay safety, and crash recovery.

Product surfaces and native services

• Cover native text/image/search bindings, CLI, TUI, settings, extensions, LSP, installers, updater, and distribution through their designated production targets.
• Cover worker subprocess lifecycles and add dual-ground rendering proofs and platform matrices.

Cutover

• Produce a generated inventory mapping every old test to one or more conformance case families.
• Demonstrate each replacement goes red against the original defect or a faithful mutation.
• Delete replaced TypeScript tests incrementally only after parity.
• Fold remaining packages/simulations scenarios into the corpus.
• Delete packages/simulations and obsolete test infrastructure.
• Run every corpus case through its declared target and run mutation certification against the affected production Rust crates; the 5,000 compiled cases launch the release binary.

Acceptance criteria

• crates/veyyon-conformance is the single conformance owner.
• Production behavior covered directly by Rust has first moved into production Rust crates.
• Remaining TypeScript behavior is exercised through designated compiled-product cases until its production logic migrates to Rust.
• Exactly 250,000 unique JSONL cases materialize reproducibly.
• Exactly 4,496 expected-error contracts materialize reproducibly.
• Exactly 245,000 cases target migrated production Rust and exactly 5,000 target the compiled release product.
• Every shipped provider and API is source-enumerated and covered.
• Raw SSE fragmentation and transport faults reach production decoders.
• Every shipped tool is registry-enumerated and covered.
• Sessions, persistence, settings, workers, rendering, security, installers, and distribution meet their subsystem allocations.
• Every deadline/retry case asserts termination and its bound.
• Persisted shapes are versioned and stale copies are rejected.
• At least 1,200 mutations execute and at least 1,000 are killed.
• Critical security and tool-safety mutations have zero survivors.
• Every migrated contract has a generated old-to-new coverage mapping.
• Every replacement test has demonstrated RED against its target defect or mutation.
• Every corpus case passes through its declared production target; compiled-product coverage reaches every subsystem and source-enumerated boundary family.
• packages/simulations is deleted after its nonredundant contracts move.
• Superseded package TypeScript tests and infrastructure are deleted; repository-governance script tests remain or are ported with parity.
• Replay bundles reproduce failures locally with one command.
• Eight deterministic CI runners—four Linux x86_64 and one for each other supported platform/architecture—finish the complete corpus in < 180 seconds after direct-Rust p95 <= 1.5 ms, compiled-product p95 <= 500 ms, and three cold calibrations of every exact shard at <= 144 seconds; merged JUnit, SARIF, mutation, and coverage inventories are produced.

Explicit non-goals

• Keeping a permanent duplicate TypeScript and Rust test architecture.
• Testing a Rust reimplementation that is not the production implementation.
• Mocking provider modules after parsing.
• Source-grep assertions against implementation text.
• Snapshot-only UI approval without cell-grid and dual-ground evidence.
• Hardcoded provider, model, tool, or setting lists that silently age.
• Raw production sessions as fixtures.
• Weakening error contracts to make generated cases pass.