Contract test: all 9 basic-case examples fail schema validation - #1
Conversation
…ples Adds tests/test_examples.py which validates every examples/basic-case/*.json against its phase schema, plus a guard that fails on unmapped examples. Currently fails 9/9, exposing the examples<->schema drift.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce547117a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| details = "\n".join( | ||
| f" - {'/'.join(map(str, e.path)) or '(root)'}: {e.message}" for e in errors | ||
| ) | ||
| pytest.fail(f"{example_name} fails {schema_name} ({len(errors)} errors):\n{details}") |
There was a problem hiding this comment.
Every mapped example currently sets meta.validated_by_human to true, while all nine mapped schemas require it to be false, so every parametrized case reaches this unconditional failure. The checked GitHub Actions workflow runs the complete pytest suite, meaning this commit makes CI fail on every push and pull request until the fixtures or schemas are corrected alongside the test.
Useful? React with 👍 / 👎.
| assert example_path.exists(), f"missing example: {example_path}" | ||
| assert schema_path.exists(), f"missing schema: {schema_path}" | ||
|
|
||
| validator = Draft202012Validator(_load(schema_path)) |
There was a problem hiding this comment.
When an example contains an invalid generated_at or refusal date, constructing Draft202012Validator without a FormatChecker ignores the schemas' date-time formats, so this contract test can pass an example that runtime validation rejects. src/specula_agent/schemas.py::validate_artifact explicitly supplies format_checker=FormatChecker(); this test should use the same behavior to guard the shipped examples accurately.
Useful? React with 👍 / 👎.
Regenerates all 9 examples/basic-case/*.json to conform to schemas/: - meta.validated_by_human -> false (draft state) - canonical meta.mode per phase (convergence, brand_archaeology, refusal_register, narrative_synthesis, community_cocreation, ...) - payload restructured to schema shape (brand_dna, competitive_map/ white_spaces, prototypes, refusals, narrative_system, co_creation, guardian_report), removing fields barred by additionalProperties:false All 9 now pass tests/test_examples.py; full suite 30 passed.
1 participant
ISSUE-esempi-non-validano.md (già pronto nella cartella verifica-framework