A comparator reads other fetchers' evidence from the same run, matches records
across them, and writes one evidence file that a validator can check. This walks
through demo_policy_check, which ships with the repo and reads demo_hello.
You copy the template, name the inputs, write the comparison, then run it after
its inputs and preview the upload.
You need: the repo installed · the fetchers whose evidence you'll compare, each writing JSON evidence
Comparators live under comparators/, not fetchers/, so it's clear which code
calls an API (why).
Copy the commands
mkdir -p comparators/<category>
cp -r comparators/_template comparators/<category>/<short_name>
ls comparators/<category>/<short_name>Replace every placeholder. depends_on is what makes it a comparator. Leave
secrets empty and add no targets: the runner gives a comparator no
credentials (fetcher.yaml rules).
name: demo_policy_check
version: 0.1.0
description: A verdict per policy in demo_hello's evidence from the same run.
category: demo
depends_on:
- demo_hello
runtime:
type: python
entry: fetcher.py
output:
type: json
path: demo_policy_check.json
secrets: []
evidence_set:
reference_id: EVD-DEMO-POLICY-CHECK
name: Demo — Synthetic Policy CheckCopy the command
paramify describe <category>_<short_name>Write only check: it gets each input's payloads and returns verdicts, not
copies of the input records. The shared run() reads the inputs, applies the
input policy, and always
writes the file:
def check(inputs):
results = [
{"policy": p.get("policy"),
"verdict": "enforced" if p.get("enabled") and p.get("applies_to") == "all-users" else "not_enforced"}
for payload in inputs["demo_hello"].payloads
for p in payload.get("results", [])
]
enforced = sum(1 for r in results if r["verdict"] == "enforced")
summary = {
"policies": len(results),
"policies_enforced": enforced,
"policies_not_enforced": len(results) - enforced,
"percent_enforced": enforced * 100 // len(results) if results else 0,
}
return results, summary
if __name__ == "__main__":
raise SystemExit(run(check, output="demo_policy_check.json", summary_keys=SUMMARY_KEYS))Adding the comparator adds its inputs too (manifest commands).
Copy the commands
paramify manifest init
paramify manifest add <category>_<short_name>
paramify validate manifest.yamlEvery fetcher runs first, then every comparator.
Copy the command
paramify run manifest.yamlThe runner adds metadata.inputs: each input file it handed over. Your payload
holds one verdict per record, and summary carries inputs_complete
(output).
Copy the command
paramify evidence evidence/run-<timestamp>/<category>_<short_name>.jsonEach file goes to its own evidence set, so the comparator's verdicts sit beside its inputs. Then upload and check Paramify.
Copy the command
paramify upload --dry-runNext: write validators for it
| Symptom | Fix |
|---|---|
depends_on names an unknown fetcher |
A copied template still lists <input_fetcher_a>. Name real fetchers in depends_on. |
which writes no JSON evidence |
An input is an issue report, a discovery fetcher, or writes CSV or HTML. Compare a JSON evidence fetcher instead. |
depends_on <input>, which can't run (…) |
That input is a comparator that can't run. Fix the cause in the parentheses. |
comparators (depends_on) belong under comparators/ in the contract tests |
Move the directory to comparators/<category>/<short_name>/, or delete depends_on if it collects from a source. |
The comparator shows [FAIL] with input_failed |
Open the evidence of the input its error names; that file holds the cause. |
Every summary value is null |
An input is missing: not in the manifest, not run, or no complete file. metadata.error says which. |
COMPARATOR_INPUTS is not set |
You ran fetcher.py by hand. Run it with paramify run. |
More detail: comparator reference · design and decisions · the create-fetcher skill





