Skip to content

feat: add lexical-mutation corpus generator - #14

Merged
3leapsdave merged 1 commit into
mainfrom
feat/lexical-mutation-matrix-v1
Aug 3, 2026
Merged

feat: add lexical-mutation corpus generator#14
3leapsdave merged 1 commit into
mainfrom
feat/lexical-mutation-matrix-v1

Conversation

@3leapsdave

Copy link
Copy Markdown
Member

Adds a lexical lane alongside the existing crypto fixture lanes: a deterministic generator for synthetic term-matching benchmark corpora.

Grammar

Every term is an immutable zzlx anchor followed by a seed-derived base32 body. Transforms only ever touch body scalars, so a mutated variant is still greppable as synthetic. Two shapes are deliberate exceptions, both documented: common-word decoys, where an ordinary word is the whole point of the control, and below-policy-length terms, which are too short to hold a full anchor.

The matrix

Seven surfaces with per-surface transform classes and severity mixes — 49 required cells, six of which carry a pinned severity rather than the surface rotation. Unicode classes, vowel-drop, and truncation outside snake_case are implemented but sit outside the required matrix behind --include-extensions.

Population floors are enforced inside generation: a run that cannot meet them fails rather than emitting a thin corpus. A default run produces 721 cases — 588 positives, 49 negative controls, and 84 below-policy-length cases.

Output planes

Generation writes a sterile plane (opaque identifiers, answer-key coordinates, digests) and a protected plane (term values and the artifacts a detector scans). The fixture set references the protected manifest by SHA-256 alone. docs/lexical-matrix.md states plainly that this is a handling boundary rather than a confidentiality control: generation is deterministic from a seed the sterile plane carries, so the protected plane is reconstructible by anyone with this source.

Correctness guards

  • A positive case whose rendered variant is byte-identical to the unmutated term is rejected at generation time. Guarding the rendered form rather than the token slice is deliberate — a transform can change tokens and still render identically, which is how that class of defect hides.
  • Term values are checked for collisions against each other and against a common-word list, covering rendered variants as well as base terms.
  • Any 32-bit seed produces a corpus; transform applicability is established after token splitting, since splitting decides which scalars end up adjacent.

Ownership markers

Each generator now owns a distinct output-root marker, so --force can only replace a root its own lane created and a corpus holding no key material is no longer labelled with the marker that locates key-bearing roots. The marker kind is the authorization attribute; the tool field is provenance only. A corpus is built alongside its target and swapped in once complete, so an interrupted run leaves the previous one intact.

Verification

make check-all less the decernor contract lane, which needs a binary at the pinned commit on the running host. Package coverage 87.0%, CLI 80.9%. Determinism verified across separate processes: identical seed produces byte-identical output trees.

🤖 Generated with Claude Code

Adds a lexical lane alongside the crypto fixture lanes: a deterministic
generator for synthetic term-matching benchmark corpora, built from an
immutable anchor plus a seed-derived body so every generated string stays
visibly synthetic.

Changes:
- Add internal/lexmatrix: grammar, transform library covering fourteen
  mutation classes, surface rendering with byte-span answer keys,
  deterministic seeded generation, and fail-closed cell accounting
- Split output into a sterile plane (opaque identifiers, coordinates and
  digests) and a protected plane (term values and source artifacts)
- Refuse to record a positive case whose rendered variant is identical to the
  unmutated term, and reject a term value that collides with another
- Build a corpus alongside its target and swap it in once complete, so an
  interrupted run leaves the previous corpus intact
- Add cmd/synthcorpus-lexgen and wire it into make build
- Give each generator its own output-root ownership marker, so --force can
  only replace a root its own lane created; document the marker kind as the
  authorization attribute and the tool field as provenance only
- Add docs/lexical-matrix.md covering grammar, matrix, floors and planes
- Gate both shipped binaries' version identity in the release checklist

Generated by Claude Opus 5 via Claude Code under supervision of @3leapsdave

Co-Authored-By: Claude Opus 5 <noreply@3leaps.net>
Role: devlead
Committer-of-Record: @3leapsdave
@3leapsdave
3leapsdave merged commit 0586115 into main Aug 3, 2026
3 checks passed
@3leapsdave
3leapsdave deleted the feat/lexical-mutation-matrix-v1 branch August 3, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant