Each audit folder's audit.yaml references a dataset path plus target/protected-attribute columns, but nothing in CI verifies the referenced CSV actually exists and contains those exact columns before faircode benchmark runs against it - .github/workflows/audits.yml's own comment references a past real incident ("a broken dataset path or script (like the AI Fair Recruitment FileNotFoundError)") that this would have caught earlier.
Request: a small manifest-validation script (or a parametrized test in tests/) that loads every */audit.yaml via faircode.manifest.load_manifest and asserts the dataset path resolves and the declared columns are present, run in CI alongside the existing per-audit smoke test.
Each audit folder's
audit.yamlreferences a dataset path plus target/protected-attribute columns, but nothing in CI verifies the referenced CSV actually exists and contains those exact columns beforefaircode benchmarkruns against it -.github/workflows/audits.yml's own comment references a past real incident ("a broken dataset path or script (like the AI Fair Recruitment FileNotFoundError)") that this would have caught earlier.Request: a small manifest-validation script (or a parametrized test in
tests/) that loads every*/audit.yamlviafaircode.manifest.load_manifestand asserts the dataset path resolves and the declared columns are present, run in CI alongside the existing per-audit smoke test.