Skip to content

Validation

Jump to bottom
Nick Hamze edited this page Jun 3, 2026 · 12 revisions

Validation

Youish's main quality check is deterministic and model-free. The release suite should pass before publishing changes.

The current quality hierarchy is:

  1. Better writing: clearer point, stronger structure, real reader action.
  2. Concision: no padded rewrites, no voice-marker hoarding.
  3. Voice: preserve the best source markers inside that standard.

Core Checks

python3 scripts/validate_skill.py
python3 scripts/public_copy_check.py --description "Paste messy notes. Get sharp rewrites that sound like you on a good day."
python3 scripts/check_marketplace.py --version 0.3.13
python3 scripts/regression_100.py
python3 -m py_compile scripts/*.py

Expected results include:

Skill repo validation passed.
Public copy check passed.
Marketplace check passed.
VALIDATOR SELF-TESTS: PASS
TOTAL: 100/100 passed

public_copy_check.py is repo-only release hygiene. It protects the README and repo description from stiff marketing language, missing in-process examples, and public-copy phrases we have already rejected. It is intentionally not part of the installed skill package.

Lab Tool Smoke Checks

python3 scripts/audit.py --source "I think notice may be due in 10 days." --rewrite "I think notice may be due in 10 days." --preserve-uncertainty --protected "10 days"
python3 scripts/rewrite_report.py --source "I think notice may be due in 10 days." --rewrite "I think notice may be due in 10 days." --protected "10 days" --preserve-uncertainty
python3 scripts/case_lab.py --case-id sample_case_01 --source "rough but redacted source" --rewrite "clean but still redacted rewrite" --must "redacted"
python3 scripts/voice_probe.py --sample "This draft is not bad. It just apologizes for existing."

Voice Profile Contract Check

profile="$(mktemp)"
python3 scripts/voice_profile.py compile \
  --sample "This draft is not bad. It just walks into the room and apologizes for existing. I want the useful bit without the meeting oatmeal." \
  --out "$profile"
python3 scripts/voice_profile.py validate "$profile"

Privacy Fixture Checks

python3 scripts/redact_case.py \
  --case-id redacted_case_01 \
  --source "Person A has nick@example.com and sk-abc123456789012345678." \
  --rewrite "Person A shipped it." \
  --expected "Person A still needs the note fixed." \
  --json

python3 scripts/failure_fixture.py \
  --case-id fixture_case_01 \
  --source "[[keep: 10 days]] [[voice: haunted changelog]] I think notice may be due in 10 days." \
  --failed-rewrite "Notice is due in 10 days." \
  --desired-rewrite "I think notice may be due in 10 days, haunted changelog and all." \
  --preserve-uncertainty \
  --allow-inline-private \
  --json

Package And Install Checks

PLUGIN_VERSION=0.3.13
python3 scripts/build_release_assets.py --version "$PLUGIN_VERSION"
python3 scripts/check_release_assets.py "dist/release-v$PLUGIN_VERSION" --version "$PLUGIN_VERSION"

tmpdir="$(mktemp -d)"
mkdir -p "$tmpdir/codex-skills"
ln -s "$(pwd)" "$tmpdir/codex-skills/youish"
python3 scripts/check_install.py --install-dir "$tmpdir/codex-skills/youish"
python3 scripts/install.py --install-dir "$tmpdir/codex-skills/youish-installed"
python3 scripts/install.py --copy --install-dir "$tmpdir/codex-skills/youish-installed-copy"
python3 scripts/check_install.py --install-dir "$tmpdir/codex-skills/youish-installed-copy"
bash -n install.sh
archive="$tmpdir/youish.tar.gz"
tar -czf "$archive" --exclude .git --exclude dist -C "$(dirname "$PWD")" "$(basename "$PWD")"
YOUISH_ARCHIVE_URL="file://$archive" bash install.sh --install-dir "$tmpdir/codex-skills/youish-one-line"
python3 scripts/check_install.py --install-dir "$tmpdir/codex-skills/youish-one-line"

What The Suite Covers

The 100 primary fixtures cover corporate filler, blunt Slack, legal precision, apologies, concision, odd voice, technical notes, unsupported claims, sensitive writing, messy thought dumps, reusable profile boundaries, format preservation, diagnosis-only requests, exact word counts, no-dash constraints, word caps, best-marker selection, and voice-hoarding failures. Contract tests also cover anti-no-op rewrites: raw passthrough, near-copy edits, padded edits, timid edits, voice hoarding, and dropped best markers.

Lightweight Taste Loop

Before a taste-sensitive release, run 5-10 ugly public-safe prompts through Youish. Judge each output by the simple standard: better, tighter, still mine.

Use synthetic prompts first: buried ask, timid argument, repetitive rant, bloated announcement, weak ending, polite boundary, generic cleanup, and docs/strategy argument. If an output feels too timid, too long, too generic, or too close to the source, turn the miss into a redacted fixture or contract test before release.

Proof Matrix

This does not prove literary taste. It proves Youish keeps the constraints it claims to protect.

Promise How It Is Tested
Improve the writing Editorial-lift, thought-dump, reader-action, and source-only artifact contracts.
Keep rewrites concise Concision fixtures, word caps, max-ratio checks, and padded-output mutations.
Keep the user's voice Voice marker fixtures, profile contracts, and mutation tests that remove keeper phrases.
Preserve facts and claims Protected fact checks, required claims, forbidden assertions, numeric drift checks.
Preserve uncertainty Legal and technical cases that fail if maybe becomes false certainty.
Preserve exact quotes and identity markers Quote and identity contracts for punctuation, diacritics, pronouns, protected names, and source-specific phrasing.
Refuse invented details Unsupported detail markers, invented numbers, and unsupported entity detection.
Avoid generic AI polish Buzzword checks plus pattern tests for shiny but empty phrasing.
Avoid under-editing Editorial-lift contracts fail near-copy rewrites when the draft needs a clearer thesis, stronger structure, or sharper stakes.
Avoid timid rewrites Timidity contracts catch unsupported hedges, buried theses, and missing strongest source-supported claims.
Keep rewrites forceful and tight Combined contracts fail rewrites that are too close to the source, too long, too bland, or too politely defanged.
Keep the best voice marker Preferred-marker checks fail rewrites that keep a weaker colorful phrase while dropping the line that actually carried the voice.
Avoid voice hoarding Voice-budget contracts require texture-rich rewrites to choose the best source markers instead of keeping every funny line.
Respect constraints Exact word count, no-dash, format, wrapper, and paragraph-shape checks.
Keep requested output shape Format contracts for options, option diversity, greetings, signoffs, diagnosis-only responses, code fences, and line prefixes.
Turn notes into usable text Source-only thought-dump cases with artifact cleanup and reader-action checks.
Keep scorecards honest Transcript integrity checks recompute from raw output, reject unproven passing records, and distinguish partial suites from complete public scores.

The scorecard is intentionally boring: full-suite gates, stable failure codes, hashes, package checks, marketplace checks, and public-safe reporting. Taste up front. Receipts in the back.

Clone this wiki locally