A validator is a regex plus rules that Paramify runs against every artifact
uploaded to an evidence set. This guide writes one by hand in the validators/
registry, proves it can fail, publishes it, and checks its verdict in Paramify.
The running example is aws_kms_key_rotation. To have your AI agent write and
prove validators for you, use /suggest-validator
instead.
You need: the repo installed ·
Node.js (Paramify runs validator regexes as JavaScript, and so does check) ·
a real run of the fetcher ·
a Paramify API key for step 6
A validator names the evidence sets it applies to by reference_id
(how upload picks a set). Then find the keys in your
newest run that prove the claim. Here the claim is "every key we can read
rotates", so the regex reads readable_keys and rotated_keys:
Copy the commands
paramify describe aws_kms_key_rotation --json | jq -r .evidence_set.reference_id
jq .payload.results.summary evidence/<run>/aws_kms_key_rotation_<target>.jsonCopy the template to
validators/aws/kms_keys_all_rotated.yaml. The key must match the filename,
and the name must be unique in your workspace. Write the regex as JavaScript,
with named groups. A first draft states the claim in one rule:
key: kms_keys_all_rotated
name: All Readable KMS Keys Rotated
type: AUTOMATED
role: configuration
statement: Every KMS key whose rotation status could be read has automatic rotation enabled.
regex: '"readable_keys":\s*(?<readable_keys>\d+)[\s\S]*?"rotated_keys":\s*(?<rotated_keys>\d+)'
rules_summary: MATCH_GROUP[1] EQUALS MATCH_GROUP[2]
validation_rules:
- regexOperation: { type: MATCH_GROUP, groupNumber: 1 } # readable_keys
criteria: EQUALS
value: { type: MATCH_GROUP, groupNumber: 2 } # rotated_keys
evidence_sets:
- EVD-KMS-ROTThe validator schema describes every field.
Pin three verdicts in validators/_cases/kms_keys_all_rotated.yaml: compliant,
non-compliant, and the keys renamed upstream
(case format). Keep the artifacts minimal
and synthetic. Never paste real evidence into a case.
validator: kms_keys_all_rotated
cases:
- name: all_rotated
expect: PASS
artifact: '{"metadata":{"exit_code":0},"payload":{"results":{"summary":{"readable_keys":3,"rotated_keys":3}}}}'
- name: one_not_rotated
expect: FAIL
artifact: '{"metadata":{"exit_code":0},"payload":{"results":{"summary":{"readable_keys":3,"rotated_keys":2}}}}'
- name: summary_keys_renamed
expect: FAIL
artifact: '{"metadata":{"exit_code":0},"payload":{"results":{"summary":{"readableKeys":3,"rotatedKeys":2}}}}'check runs the cases with the semantics Paramify uses, in JavaScript. The
draft fails it: with the keys renamed, its rule compares nothing to nothing
and passes.
Guard the read. Put a presence rule first, so a regex that matched nothing fails:
rules_summary: MATCH_COUNT NOT_EQUALS 0; MATCH_GROUP[1] EQUALS MATCH_GROUP[2]
validation_rules:
- regexOperation: { type: MATCH_COUNT } # the summary was read
criteria: NOT_EQUALS
value: { type: CUSTOM_TEXT, customText: "0" }
- regexOperation: { type: MATCH_GROUP, groupNumber: 1 } # readable_keys
criteria: EQUALS
value: { type: MATCH_GROUP, groupNumber: 2 } # rotated_keysRun it again. Every case now gets the verdict it expects:
Copy the command
paramify validators check --select kms_keys_all_rotatedA validator that counts violations (zero matches means pass) can't guard
itself this way. It needs an integrity partner and a set-level case
(example).
Every set also gets the shared collection_succeeded validator, which fails
an artifact from a run that didn't finish
(why it's shared).
Add EVD-KMS-ROT to its evidence_sets rather than writing your own. Then
review the change:
Copy the commands
git status --short validators/
git diff validators/common/Commit all three in your private copy of the repo.
Upload a run before you sync. The sync attaches a validator to its sets only when it creates it, so the set must already exist (how sync works). Preview the sync, scoped to your manifest:
With your key set, the preview reads the workspace, so a validator that's
already there shows would_skip_exists. Then upload and sync for real
(paramify upload --with-validators does both in one command):
Copy the commands
paramify validators sync -m manifest.yaml --dry-run
paramify upload
paramify validators sync -m manifest.yamlWatch for skipped_exists on the collection check. If your workspace
already has it, the sync leaves it alone and doesn't attach it to the new set.
Attach it by hand.
Go to Implementation → Evidence Sets and open KMS Key Rotation.
Validators tab: both validators are listed.
Artifacts tab: upload a fresh run, because artifacts uploaded before the sync get no result. With one key in three not rotated, its Validation Result is Partial. Click it to see each validator's verdict.
Using Validators in Paramify explains the rules and results views.
| Symptom | Fix |
|---|---|
check shows COMPILE_ERROR on every case |
The regex uses Python's (?P<name>…). Write (?<name>…). |
check stops with must equal the filename or schema validation failed |
Make key match the filename, and fix the field the error names (schema). |
check prints SKIP … names nothing in the registry |
The case file's validator: doesn't match any registry key. Check the spelling. |
check --select lists other validators under "no cases" |
It counts only the files the filter picked. Run paramify validators check with no filter for the real list. |
A rule's disposition has no effect |
The API discards it, so every rule must hold for a pass. |
HTTP 400, set_not_found, or app verdicts that differ from the YAML |
See the agent guide's troubleshooting. |
More detail: registry README







