Eval-first expansion: 435 deterministic cases, 23 gates, teach/mimic voice system, measured model tiering by theclaymethod · Pull Request #2 · theclaymethod/unslop

theclaymethod · 2026-07-06T22:41:12Z

Ten adversarially-verified work packages that grow the eval suite from 201 to 435 deterministic cases and from 12 to 23 machine-readable gates, add six product capabilities, and replace assumptions about model tiering with measured results. Zero baseline rows dropped; the single intentional xfail (FP-06) is unchanged.

Scorecard

	main	this branch
Deterministic cases	201	435 (434 pass + FP-06 xfail)
Behavioral cases	33 (5 backstopped)	33 (all deterministically backstopped)
Gates (`--list-gates`)	12	23
Holdout lift (objective)	—	+4.2 pts (0.917 vs 0.875, 12 held-out cases)

What's in it

Eval infrastructure

Minimal-pair paragraph fixtures (16 pairs, target-isolated both directions) and seeded documents with finding manifests, plus hygiene gates (check_pairs, check_seeded_docs)
Self-enforcing pattern addition: every scanner pattern must be exercised by a row and every category FP-protected (check_pattern_coverage, 76/76 and 314/314 and 52/52), with a rehearsal kata that proves each safety net fires (kata_add_pattern)

Detection

Discourse-level silhouette scanner: five research-validated idea-arrangement metrics with a drift-pinned human reference; separates 12/12 AI docs from 8/8 human docs on the committed corpus
New pattern families landed eval-first, including several caught live by benching real model output ("research indicates", subject-gated "tells a story", scan-before-decline English gating)

Voice (teach and mimic)

Deterministic stylometric voice scorer with a General Impostors rank as the gaming-resistant core; three synthetic author corpora as committed measurement fixtures
teach: layered voice cards (core card + per-situation sheets, no claim without sample evidence) with coverage-driven interactive sample gathering
mimic --refine: hill-climb loop with held-out acceptance, divergence guard against reward hacking, and a permanent regression fixture where a gate-passing stuffed candidate must lose to honest prose (0.733 vs 0.143 under the GI composite; the pre-fix raw distance would have accepted the attacker)
harvest: bootstrap the teach corpus from chat transcripts; assistant-authored text provably never reaches candidates (contamination gate, mutation-verified)
calibrate: A/B preference game over deterministic single-dimension transforms, Wilson-bound aggregation, voice-overrides-defaults conflict surfacing

Co-writer and contribution

LSP-style suggestions with contract gates (span-minimal, replacements scan clean, accept-all safe)
contribute: a wild-caught AI-ism becomes an eval row from the exact specimen plus a structured PR, fully offline, with redaction safety and red-first verification
Model-parity harness with dry-run rows; references/pipeline.md now carries the measured tiering table (2026-07-06 live benches: span replacement is safe on the cheapest tier, register-sensitive full rewrites need frontier models, macro structure defeated all six models tested and stays machine-gated)

Docs

docs/PRODUCT.md doctrine (passes all three of the repo's own scanners), updated CLAUDE/AGENTS/maintenance/CHECKS, measured claims only

How it was built and verified

Every package was implemented by one model and adversarially reviewed by a different one (Codex, Opus, and Sonnet in alternation), with mutation testing as the standard of proof: delete the pattern, the suite must fail; fake the fixture, the gate must catch it. Review findings were fixed eval-first (rows red before code) or explicitly documented as accepted queues in the relevant commit messages (WP8 fixture word floors and length balance; WP10a impostor-pool calibration items). A final nemesis pass against a written completion spec found two regressions that all 23 gates had missed (English-gate friendly fire on low-function-word slop; a context-free ban); both are fixed and pinned in this branch.

Verification

python3 evals/run_adversarial.py          # 434 PASS / 1 XFAIL (FP-06) / 0 FAIL
python3 evals/run_adversarial.py --list-gates   # 23 gates, all green, CHECKS.md pinned

Behavioral: runs/ artifacts are not committed; holdout was scored post-assembly (numbers above), holdback remains sealed. The without-skill baseline is contaminated by the globally installed skill, so per-case deltas matter more than aggregate lift (see evals/BEHAVIORAL-EVALS.md).

🤖 Generated with Claude Code

https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

…earsal kata - evals/check_pattern_coverage.py: every structural pattern must match >=1 eval-row input and every banned phrase must appear in the row corpus (no grandfathering — 28 DET rows, 28 REC coverage packs, 31 FP protection rows added to reach 75/75 patterns, 313/313 phrases); every category must be claimed by an FP row's protects field (52/52, backfilled) - evals/kata_add_pattern.py: rehearses the maintenance procedure in a temp copy and asserts each safety net fires in order (coverage -> parity -> green -> red-on-removal); deterministic transcript - Coverage gate immediately caught a dead pattern: filler_opener matched literal 'So' against lowercased text; resurrected comma-gated and soft with FP-83 protecting ordinary sentence-initial 'So' - DOC-09/DOC-10 gates wired into --list-gates and CHECKS.md; maintenance.md documents the conventions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add evals/run_model_parity.py: measures whether cheap models match strong models on the two model-dependent product surfaces — Tier-1 pack DETECTION (Task A) and Tier-2 REPLACEMENT generation (Task B) — over a fixed seeded corpus, graded deterministically. - Config-driven matrix {name, kind, model_id}: kind claude-cli (`claude -p --model <id>`, Anthropic spectrum) or openrouter (GPT spectrum, key from macOS keychain service OPENROUTER_API_KEY). - Task A: 6 seeded fixtures (phrases-core/voice/register-guards; 2 are register-guard cases), pack + chunk per the Tier-1 contract, recall + false findings vs a frozen manifest. - Task B: 6 seeded findings; span-minimal replacement graded by the co-writer contract — both scanners (no tell added, target removed), constraints preserved, span-minimal. - Outputs per-model/per-task JSON table + markdown summary. --dry-run grades canned response fixtures with no network; --no-network proves graceful degradation; --check-determinism proves stable grading. Eval rows (dry-run, network-free): PARITY-01 grading correct on canned good/bad responses (both tasks), PARITY-02 determinism, PARITY-03 missing key/network never breaks the suite. Docs: references/pipeline.md gains a Model Parity section, binds any change to co-writer/mimic/pack features to running the harness across both spectrums, and ties the detection/replacement tiering rows to its data. CLAUDE.md + AGENTS.md get one identical required-checks line for those features. Gates green: run_adversarial (203 pass, 1 xfail), build_shared_benchmark --check, check_taboo_parity, check_gates_doc, check_skill_examples, check_packs, skill-benchmark validate --strict-leakage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

Add agent-invoked co-writer mode and a graceful non-English decline, both under the existing scanner constitution (eval-first; all gates green). scripts/suggest.py Emit LSP-style structured suggestions {span, severity, category, rationale, suggested_replacement, phrased_as_question} from banned_phrase_scan + structure_scan. Detection is deterministic; replacement generation is DELEGATED (suggested_replacement null) with a --apply-replacements FILE mode that merges externally-produced replacements and light-validates them. Soft findings are phrased as questions. Deterministic order, non-overlapping spans. scripts/check_suggestions.py Blocking contract gates, each a named failure: span-minimality (edit touches only its span; whole-sentence rewrites fail), replacement-scanner (each replacement passes both scanners in isolation and in context), accept-all (applying every suggestion yields a doc passing both scanners with validate_preservation exit 0), span-overlap. English-only decline Cheap function-word heuristic (english_function_share / is_probably_english) added identically to banned_phrase_scan.py and structure_scan.py. Below the conservative threshold -> {"non_english": true, "violations": []}, exit 0 with a stderr note. Threshold tuned so ESL English still scans. Eval rows (RED before these scripts existed): SUGG-01..05 (suggestion emission, soft-as-question, oversized-replacement rejection, accept-all safety, overlap rejection) and LANG-01a/01b + LANG-02a/02b (Spanish declined by both scanners, ESL English scanned by both). Fixtures under evals/fixtures/suggest/. SKILL.md gains a compact Co-writer Mode section (agent-invoked; cheap detection, delegated replacements, blocking contract gates, suggestions surfaced never silently applied). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

# Conflicts: # evals/adversarial-evals.json

Ship scripts/silhouette_scan.py: a macro-structure scanner one level above structure_scan that scores idea arrangement, not surface cadence. Five validated one-sided tells (scaffold_opener_share, callback_content, role_entropy_bits, preview_fulfillment, heading_preview) reused verbatim from the research prototype, composited as a weighted relu-of-z penalty against a committed human reference; flags at silhouette_penalty >= 1.0. Scale deviation (documented in the scanner and reference): these tells are degenerate at zero across the human corpus, so a literal sample-IQR denominator with a 0.05 floor over-fires (2/8 human false positives). The scorer scales each metric by max(sample_iqr, human_fence) instead, which keeps the exact weighted-relu shape, reproduces the research struct01/03/09/11 signature, and holds 0/8 human false positives while catching 12/12 ai (>= the 8/12 target). - evals/check_silhouette.py: --reference drift gate (regenerates human_reference.json from named in-repo sources and asserts equality) and --separation gate (>= 8/12 ai, 0/8 human on the copied-in corpus). - evals/fixtures/silhouette/: committed human_reference.json, self-contained ai/human corpus, and SIL fixtures. - Eval rows SIL-01..08 (RED-first) plus SIL-05a/b proving the silhouette + structure_scan anti-gaming pairing against a cue-deletion attack, and DOC-07 for check_silhouette. Two gates wired into list_gates(); CHECKS.md regenerated. - Docs: silhouette subsection in references/taboo-phrases.md, detector pointers in pack-structure.md, and silhouette_scan added to the SKILL.md validation list as blocking with a genre carve-out. Voice-fingerprint (per-author median/scale) integration is noted for the teach/mimic branch and deliberately left unimplemented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

WP13's harvest rows reference a fixture directory; the WP9 coverage gate assumed file paths. Walk directories, skip non-text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

# Conflicts: # evals/adversarial-evals.json

…mpostor rank scripts/voice_profile.py + voice_score.py: char-3gram cosine, cosine delta, sentence EMD, punctuation/contraction/MTLD/word-length distances, z-normalized vs an impostor pool, General Impostors rank as the dominant gaming-resistant term; copy gate (4-gram overlap + LCS); seeded determinism. Three synthetic author corpora (amara/boris/celia) as measurement fixtures; VOICE-01..11 rows (3x3 separation, GI sanity, gaming guard, copy gate, determinism, drift-pinned profiles via check_voice). Adversarially verified: four fresh mechanical attacks lose by >=0.85 composite; separation diagonal wins at 5 seeds. Known follow-ups (approved-with-fixes, queued): replace the mislabeled stuffed fixture with a true mechanical attack, same-genre impostor pool, real --background calibration for delta z-scores, feature-space GI subsets, full length-floor nulling, function-word dedupe, IDF-weighted char-3grams. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ructured PRs scripts/contribute.py (precheck/scaffold/verify/report, fully offline): specimen fidelity byte-for-byte, redaction safety (tell-overlap exit 4), red-first verification with TODO-refusal, golden-tested PR body template. references/contribute.md: agent procedure with two user-confirmation gates (redaction, publication), explicit bundle-to-suite bridge, fork path. CONTRIB-01..10 rows + contribute gate. Review fixes: frozen --date injection (fixtures no longer date-fragile), casefolded phrase assertions (capitalized tells mergeable), structure_scan flags honored in precheck (both-scanners contract), conditional REC row in the template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

# Conflicts: # evals/adversarial-evals.json

… docs-genre callback carve-out - docs/PRODUCT.md: the grill-session decisions as durable doctrine; passes all three scanners (and dogfooding it surfaced the carve-out below) - references/pipeline.md: Model Parity results table populated from the live 2026-07-06 benches; tiering guidance updated to measured conclusions - Bare 'research indicates/shows/suggests' joins vague_attribution (FN-31/FP-84/REC-49, eval-first) — caught live when a model paraphrased around 'studies show' in the replacement bench - silhouette_scan --genre docs suppresses callback_content only: doctrine/spec register reprises opening themes by convention (SIL-9 pins the carve-out; essay recap codas still flag by default) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

# Conflicts: # evals/adversarial-evals.json

- 14 phrase-family pairs + 2 macro pairs (with/without twins, single controlled difference): with-twin flags exactly the target family, without-twin clean on both scanners; verified target isolation on every pair - 3 realistic seeded documents with finding manifests + clean twins; check_seeded_docs enforces recall and clean-twin precision - check_pairs.py pair-hygiene gate (existence, word-delta, clean-without, with-target); DOC-07/08 gates wired - Maintenance docs prefer pairs for contextual patterns Adversarially verified (approve-with-fixes); queued follow-ups from review: raise fixtures to spec word floors and enforce in check_pairs, corpus-level length-balance check (with-twins currently skew +2.4w), structural manifest entries + exact-multiset dupe/extra charging in check_seeded_docs, vague_attribution pair single-difference rework. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

# Conflicts: # evals/CHECKS.md # evals/adversarial-evals.json # evals/run_adversarial.py # references/maintenance.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Adds scripts/calibrate_pairs.py (deterministic dimension-controlled minimal pairs for contractions, em_dash, sentence_length, connectives, staccato -- constraint-preserving, exits 3 when a dimension isn't expressible in the given passage) and scripts/calibrate_score.py (Wilson-lower-bound aggregation over a preferences JSONL, insufficient-below-k gating, --next dimension ordering, and --profile conflict detection against a measured stylometric profile). references/calibrate.md documents the agent-hosted game flow and provenance discipline (stated-preference vs measured-from-samples); SKILL.md gets one pointer line. Eval rows CAL-01..08 cover per-dimension transform correctness, the inexpressible-dimension exit code, seed determinism, canned-stream aggregation, below-k insufficiency, conflict detection, --next tiebreaking, and the missing-file exit code. Wired a calibrate-suite gate (--only CAL) into list_gates() and regenerated the CHECKS.md matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

Implements WP10b on top of the WP10a voice scorer. Teach (scripts/voice_card.py): distills a profile + samples into a LAYERED voice card — a <=300-word always-loaded card.md core with a "when writing X, read card/X.md" index table, plus one card/<situation>.md sheet per COVERED situation from a 10-dimension taxonomy. Uncovered dimensions get no sheet and are named in card.md, so the card never fabricates a voice without sample evidence. --coverage emits a deterministic lexical coverage matrix that drives teach prompts (misclassification only adds/drops sheets, never card claims). --provenance writes an auditable sha256 manifest. Byte-identical outputs. Mimic --refine (evals/run_mimic_refine.py): iterative hill-climb with seeded A/DEV document splits, hard removal gates per candidate (banned-phrase, structure, draft->candidate preservation, copy-gate vs A, 150-word floor), DEV-only acceptance, and a divergence guard that halts on the reward-hacking signature (A improves while DEV worsens two rounds) with reward_hacking_warning. Fully deterministic in --candidates-dir dry-run mode. Derives ranked directives with paired card-amendment lines; writes report.json, final.md, and the refined card. Stats (evals/mimic_stats.py): per-item paired deltas, BCa bootstrap, and sign-flip permutation; improved iff CI_low>0 AND p<0.05. Eval-first: MIMIC-01..09 and CARD-01..07 rows added to adversarial-evals.json first (RED before check_mimic.py existed), driven by evals/check_mimic.py over committed dry-run fixtures. New "mimic-logic" gate wired into list_gates and CHECKS.md regenerated. references/mimic.md replaces the onslaught stub; SKILL.md gains a Teach & Mimic subsection within the DOC-05 budget; .unslop/ gitignored. Stdlib-only, no new xfail, all gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

Eval-first per CLAUDE.md: extended/added CAL-01, CAL-04, CAL-09..11 to pin each behavior change before touching product code. D1: em_dash restricted to the paired-dash<->paired-comma path only. A lone joiner dash has no comma-pair equivalent that preserves sentence count (the old lone-dash->period path silently split one sentence into two); lone-dash-only passages now decline with exit 3. base_em_dash.txt fixture rewritten to use paired dashes on both sentences. CAL-01 now also asserts sentence count is unchanged between A and B for em_dash. D2: contraction rewrites now skip matches sitting inside a capitalized multi-word span (e.g. "Venue Can't Stop"), so a proper noun that happens to reuse a contractable word is left untouched instead of becoming "Venue Cannot Stop". Separately, _verify_constraints_preserved now does a whole-occurrence (token-boundary) comparison instead of substring containment, since "Venue Can" (a proper noun clipped at an apostrophe by extract_constraints) is a substring of the corrupted "Venue Cannot" too. New CAL-09 + evals/fixtures/calibrate/proper_noun_contraction.txt. D3: generate_pair now runs banned_phrase_scan on both variants and annotates the output with a_flags/b_flags (category lists, empty when clean) instead of declining flagged variants. references/calibrate.md gets a "Voice overrides defaults" section: a user's consistent preference for a flagged pole is recorded with its flags, surfaced once, and marked "user- preference overrides register guard" on the card. New CAL-10 + staccato_flagged.txt / contractions_clean.txt fixtures. D4: connectives plain pole now emits "But " (no comma), avoiding "So,"/ "Also," forms that read as filler_opener. D5: calibrate_score.py reports preferred: null (status "tied") when a dimension's top two tallies are exactly equal, instead of silently picking whichever label sorts last. D6: aggregate() dedups preferences by pair_id, keeping the latest row by ts, so a replayed round counts once and the user's latest choice wins. New dedup_by_pair_id() + CAL-11. D7: references/calibrate.md instructs randomizing A/B display order per round (seeded, reproducible) and recording the mapping so choices resolve to the correct pole. preferences_canned.jsonl extended with a staccato tie case and a connectives pair_id replay case; CAL-04 assertions extended accordingly. CAL-07's tiebreak premise (connectives lowest-n) still holds since connectives ends at n=1 post-dedup, still the fixture's minimum. Full adversarial suite: 334 pass, 1 documented xfail, 0 regressions. All 16 blocking gates verified green; behavioral-tune/rubric-judge (non- blocking) skipped -- calibrate has no shared-benchmark skill rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

# Conflicts: # evals/CHECKS.md # evals/adversarial-evals.json # evals/run_adversarial.py

…ard guard F1: run_mimic_refine's acceptance/divergence scoring used a raw weighted distance with no General Impostors term, so a marker-stuffed candidate that cleared every hard gate could be accepted over honest prose. Score DEV and A with the full voice_score composite (0.5*(1-GI) + 0.5* clipped weighted impostor-z) against a seeded impostor pool (--impostors, default the committed pool). New MIMIC-10 pins the regression: a punctuation/repetition-stuffed candidate (cand02) and honest prose (cand01) both pass the hard gates, but under the GI composite honest wins (0.143 vs 0.733). Divergence fixtures re-derived to keep the genuine two-style A-down/DEV-up dynamics under the new composite. F2: voice_card now recomputes the profile from --samples and exits 2 with a named field mismatch when the supplied --profile does not describe them, so a stale profile cannot silently drive a card (CARD-08). F3: implement the LIVE generation path (--generate-cmd): per iteration assemble B prompts (draft + A-split card + k=2 nearest-A samples + directives), invoke the generator per beam over stdin, then run the same gate/score/accept pipeline. --baseline selects prompt samples through the same path. mock_generator.py makes it deterministically testable (MIMIC-11). F4: document --name, the .txt/.md requirement (+ zero-doc diagnostics in voice_profile/voice_card), the coverage->prompt templates per taxonomy dimension, low_confidence surfacing, and rewrite mimic.md Scoring/refine to describe the GI-bearing composite and the now-live generation path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

# Conflicts: # evals/CHECKS.md # evals/adversarial-evals.json # evals/run_adversarial.py

… false agency - Both scanners now decline as non-English only when the function-word heuristic fails AND zero patterns fired: imperative headline stacks and buzzword noun-lists are English slop with few function words, and the old order silently defeated the shipped headline-cadence detection (LANG-3a/3b pin both genres; Spanish still declines) - 'tells a story' flat hard ban replaced with an inanimate-subject structural pattern (data/numbers/charts/... tells a story): grandmothers and clinical case notes stay clean (FP-85), the genuine false-agency form still flags (REC-50) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… faster gates Applies a reviewed cleanup list without changing any script's CLI flags, JSON keys, exit codes, or stderr messages (all pinned by eval rows). Shared modules (dedup): - scripts/_lang.py: ENGLISH_FUNCTION_WORDS/english_function_share/is_probably_english, previously byte-identical copies in banned_phrase_scan.py and structure_scan.py. - structure_scan.py now imports split_sentences from readability_metrics instead of keeping its own copy. - silhouette_scan.py's stopword set is a verified pure superset of structure_scan's; SILHOUETTE_STOPWORDS = structure's set | the extras. - harvest_classify.py imports recency_value/DATE_FLOOR from harvest_samples. - evals/_check_support.py: ROOT, run(cmd, timeout=60), load_evals() -- the timeout=60 safety net that only check_contrib.py had now covers check_pairs, check_seeded_docs, check_mimic, check_contrib, check_voice, and check_pattern_coverage too. Small cleanups: contribute.py's row_fn drops its unused category param; run_model_parity.py's resolve_models param renamed responses->payload; harvest_classify.py's heuristic() attaches suspect_ai/dictated so rank_enriched needs no reconstruction; check_pattern_coverage.py's paired flag formulas become a plain "if neither: both = True"; check_contrib.py drops the __import__("scripts.contribute", ...) spelling for a normal import; run_mimic_refine.py computes docs_a/matrix_a once and passes it to both make_live_source and write_outputs; calibrate_pairs.py factors its four near-identical contraction-replacement closures into one _contraction_repl helper used by both directions. Altitude items: GENRE_SUPPRESSIONS lookup tables replace the inline `genre != "..."` conditionals in structure_scan.py and silhouette_scan.py; check_gates_doc.py additively verifies every *.py token in a gate command exists under ROOT (behavioral-tune and rubric-judge are exempt -- neither command has a .py token). Efficiency (Phase 2), each verified against the same eval rows / diffed outputs before landing: - run_model_parity.py replaces its subprocess-per-scanner-call helpers with in-process imports of banned_phrase_scan/structure_scan/validate_preservation (mirrors run_mimic_refine's import pattern). PARITY slice: 14.2s -> 0.33s. - voice_score.py's lcs_len (O(n*m) DP) is replaced by has_common_substring_over(), an O(n+m) rolling-hash check for "any shared substring longer than the 120-char threshold" (hash matches are verified against the source text, so no false positives). Nothing pins the exact longest_common_substring value (checked); it now reports the matched threshold window length on a hit, 0 otherwise -- documented in the docstring. The violation boolean is unchanged. - gi_score() precomputes per-key distances once per candidate/impostor instead of recomputing distances() from scratch every trial; trials do a subset-weighted sum over the precomputed values. Arithmetically exact (same RNG draw order, same float sums) -- verified the VOICE-08 determinism value and the full check_voice --separation/--gi/--gaming output are byte-identical before/after. - check_voice.py and check_pairs.py convert their subprocess-per-cell/row scanner calls to in-process imports (voice_score/voice_profile, banned_phrase_scan/structure_scan), mirroring the CLI's own decline/exit logic so output stays byte-compatible. Deferred (out of scope for a contract-safe pass): a protects-grain redesign, giving silhouette_scan.py its own English-decline gate, and decomposing run_mimic_refine.py's build_report(). python3 evals/run_adversarial.py: 434 PASS / 1 XFAIL / 0 FAIL, unchanged throughout; wall time 54.8s -> 25.3s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

'Teach my voice' now carries the complete flow in SKILL.md and mimic.md: the agent harvests samples when none are offered, runs the approval loop, creates all directories, builds profile/card/provenance, surfaces low_confidence and coverage gaps, and closes with a scored demo mimic. The user supplies only approvals and answers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Split the monolithic SKILL.md into a shared-doctrine router plus one self-contained flow file per command under references/commands/. Bare `/unslop <text>` defaults to rewrite; invoking a subcommand now requires reading references/commands/<command>.md first. Top-level verb surface (per product decision): teach, cleanup, rewrite, mimic. contribute stays reachable as a maintenance path, not a verb. cleanup folds the old audit report-only mode; mimic folds the refine hill-climb; teach folds harvest (step 1) and offers calibrate when coverage is thin. Flow text moved out of the deep references (mimic/harvest/calibrate/contribute), which now hold internals plus a pointer, so no content lives in two places. SKILL.md keeps the constitution: register guards, validation gates and blocking semantics, output formats, script and reference tables. 214 lines, within the DOC-05 budget; Quick Examples still pass DOC-02. Eval-first parity gate: evals/check_commands.py pins the four-verb surface and keeps argument-hint, the routing table, and the command files in exact agreement (files exist both ways, <= 90 lines each, first line `# /unslop <cmd>`). Added as DOC-13 (RED with the gate absent, GREEN once implemented) and wired into list_gates() as command-router-parity; CHECKS.md regenerated from --list-gates (24 gates, DOC-03 pins it). Full deterministic battery green: adversarial 435 + 1 XFAIL, build --check, taboo parity, pattern coverage, kata 4/4, pairs, seeded docs, silhouette 12/0, voice all, skill examples, packs, gates doc, HARV / CONTRIB / CAL / MIMIC / CARD slices, strict-leakage. All new and moved command files pass banned_phrase_scan with zero hard violations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

Sub-flows stay reachable by natural name without being top-level verbs: a Routing-by-phrase table maps audit, harvest, calibrate, refine, voice-check, and found-a-new-tell phrasings to their command-file sections, pinned by the router parity gate (required synonyms + real targets). mimic.md gains the voice check: score a draft against a taught profile, report in plain words, change nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Codex session JSONL (~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl) has a different envelope shape than Claude Code JSONL: user/agent messages nest under event_msg.payload.type, and response_item rows carry a role that can be user, assistant, or developer. Codex also injects non-user content into user-role turns (AGENTS.md dumps, <environment_context> banners, and similar wrapper tags), which is a distinct contamination vector from plain assistant authorship. Detect claude-jsonl vs codex-jsonl per file by content shape, reuse the existing filter/tripwire/dedup pipeline unchanged, and drop injected wrapper content under a named instruction-injection drop reason instead of silently discarding it. HARV-12..15 pin: assistant/base_instructions/ injection markers never reach candidates, substantive codex user prose is harvested, injection wrappers are dropped with a named reason, and a mixed claude+codex+text-folder run produces correctly attributed candidates from all three sources. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

Rebuild the README around the current four-verb product (teach, cleanup, rewrite, mimic) instead of the pre-restructure audit/rewrite framing. - Lead with what unslop is and the four commands. - Document the three detection layers (phrase, structure, silhouette) with live counts: 313 banned phrases, 77 structural patterns, and the committed silhouette corpus separation (12/12 AI, 0/8 human). - Add a "gamut" tour of removed pattern families with quoted examples so the README passes its own phrase scanner. - Cover do-no-harm protections, teach/mimic voice building with the contamination guarantee and gaming-resistant scoring, the cleanup co-writer contract gates, contribute flywheel, and the eval suite (439 cases, 24 gates, 33-case behavioral split). - Record the measured 2026-07-06 model-tiering results and refresh the scripts tour and project tree to current reality. README passes banned_phrase_scan (0 hard) and structure_scan --genre docs clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5

… artifacts - evals/TUNE-RESULTS.md gains the 2026-07-06 holdout run (objective +4.2, judge-blended flat, contaminated-baseline caveat) so public claims trace to a repo record - README lift interpretation cites the recorded number - .codex/ nemesis review specs untracked and gitignored: process artifacts, not product Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

theclaymethod and others added 29 commits July 5, 2026 21:41

Add harvest sample bootstrap lane

14437fa

Fix harvest merge and ranking regressions

d052617

Merge branch 'wp11-cowriter' into eval-forms-integrate

42ecf29

# Conflicts: # evals/adversarial-evals.json

Merge branch 'wp12-parity' into eval-forms-integrate

1aa840f

# Conflicts: # evals/adversarial-evals.json

Merge branch 'wp13-harvest' into eval-forms-integrate

36806a1

# Conflicts: # evals/adversarial-evals.json

Handle directory fixture references in pattern-coverage corpus

042a282

WP13's harvest rows reference a fixture directory; the WP9 coverage gate assumed file paths. Walk directories, skip non-text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge branch 'wp14-silhouette' into eval-forms-integrate

78de6a2

# Conflicts: # evals/adversarial-evals.json

Merge branch 'wp16-contribute' into eval-forms-integrate

5769bf0

# Conflicts: # evals/adversarial-evals.json

Merge branch 'wp-docs-pattern' into eval-forms-integrate

e0b87bc

# Conflicts: # evals/adversarial-evals.json

Merge branch 'wp8-pairs' into eval-forms-integrate

1d643be

# Conflicts: # evals/CHECKS.md # evals/adversarial-evals.json # evals/run_adversarial.py # references/maintenance.md

Align protects claim with merged hedge_stack category name

1871daf

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge branch 'wp15-calibrate' into eval-forms-integrate

9134504

# Conflicts: # evals/CHECKS.md # evals/adversarial-evals.json # evals/run_adversarial.py

Merge branch 'wp10a-voice-committed' into eval-forms-integrate

a5718ab

# Conflicts: # evals/CHECKS.md # evals/adversarial-evals.json # evals/run_adversarial.py

Drop em-dash pairs from SKILL.md teach and contract bullets

b78f366

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

theclaymethod force-pushed the eval-forms-integrate branch from 4e9ecbc to b78f366 Compare July 6, 2026 23:45

theclaymethod and others added 5 commits July 6, 2026 18:50

theclaymethod merged commit 0f735c9 into main Jul 7, 2026
0 of 2 checks passed

theclaymethod deleted the eval-forms-integrate branch July 7, 2026 14:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eval-first expansion: 435 deterministic cases, 23 gates, teach/mimic voice system, measured model tiering#2

Eval-first expansion: 435 deterministic cases, 23 gates, teach/mimic voice system, measured model tiering#2
theclaymethod merged 34 commits into
mainfrom
eval-forms-integrate

theclaymethod commented Jul 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

theclaymethod commented Jul 6, 2026

Scorecard

What's in it

How it was built and verified

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant