Catch a reward-hacker in your terminal — judge-free, 0 model calls, $0.
node bin/vlabs-demo.jsRun that command from a reviewed checkout. The demo itself has no dependencies,
uses no API key, and makes no network request. Obtaining the repository still
requires the normal network/download and source-review step. Default output is
byte-stable; terminal colors are opt-in with --color.
It runs the Isomorphic Perturbation Testing (IPT) mechanism on embedded toy tasks entirely in Node.
A genuine program is invariant under a semantics-preserving regeneration of its test suite. An input-memorizing shortcut is not:
shortcut(H) = passes(public tests T) AND NOT passes(isomorphic tests T′)
Both a genuine solution and a hardcoded, input-memorizing hack pass the public tests. The demo compares IPT against a deliberately weak, deterministic visible-tests-only gate; it does not execute or simulate an LLM judge. IPT regenerates the inputs and recomputes expected outputs with a trusted reference: the memorizer breaks, while the genuine solution does not.
- IPT is a public method (Helff et al., arXiv:2604.15149). Verifiable Labs productizes it — we did not invent it.
- The embedded positives are constructed input-memorizers that fail
T′by construction. This demonstrates the mechanism; it is not an empirical field detection rate. - A pass requires all 12 distinct fresh cases. If a generator cannot supply that coverage, the suite is inconclusive and fails closed rather than silently passing fewer cases.
- This demo contains no field-detection, recall, false-positive, latency, or model-comparison measurement. Such claims require a separately versioned evidence artifact with complete provenance and independent review.
This Node demo is a small, self-contained illustration of the mechanism. The public Python gameability-audit package is not yet on PyPI. Do not copy a stale source-install SHA from this README: after the remediation is reviewed and merged, its release documentation must publish the immutable audited commit or signed tag. Until then, inspect and run the Python repository from a reviewed checkout:
git clone https://github.com/verifiablelabs/vlabs-integrity.git
# Verify and check out the audited commit/tag published with the release.
# Then follow that revision's README and run its tests before installation.The embedded demo is offline once the checkout exists. For custom audits, pass only reference code you trust and use the package's required external sandbox; do not treat ordinary in-process execution as an isolation boundary.
Docs, and how to scan your own verifier or RL/RFT reward: https://verifiable-labs.com
Apache-2.0