Repository navigation
fix!: enforce the OSCAL datatype constraints in the models - #79
Merged
Merged
Conversation
The post-processor removed `pattern=` from 22 `str` fields when a type name such as `int` occurred in the 10 lines before the pattern (the word "integrity" is enough). The models also accepted values that OSCAL rejects and changed values that OSCAL accepts. - Place each pattern by the base type of its `Annotated`, with an AST pass: a `str` pattern stays, a number pattern goes, a pattern on any other base fails the generation and names it. - Translate each pattern from ECMA-262 to the Rust `regex` dialect (`scripts/regex_translation.py`), so `.` and `\s` match what the schema means (PRS-007). - Add `v1/_datatypes.py` with `OscalUri`, `OscalUriReference`, `OscalEmailAddress`, and `OscalDateTimeWithTimezone`. The hoist names the four schema definitions with `customTypePath`. Each check runs in pydantic-core. A uri, uri-reference, or email value keeps the input text. - Depend on `pydantic`, not `pydantic[email]`. - Archive the change and add the `datatype-constraints` main spec. `build-resource/invalid-document-id` now passes. BREAKING CHANGE: the models reject values that they accepted before: values that a pattern, the RFC 3986 `URI` and `URI-reference` grammars, the RFC 5321 `Mailbox` grammar, or the date-time pattern of the schema rejects. A uri, uri-reference, or email-address field holds a `str`, not `AnyUrl` or `EmailStr`, and a serialization writes the input text. The runtime dependency is `pydantic` without the `email` extra: a caller that uses `email-validator` through this package must install it. The module-level names `URIDatatype`, `EmailAddress`, `AnyUrl`, `AwareDatetime`, and `EmailStr` leave `oscal_bindings.v1.models`. Closes #77
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #77.
fix_non_string_patternsremovedpattern=fromstrfields when a type name such asintoccurred in the 10 lines before it ("integrity" is enough). 22 fields lost their pattern, among themDocumentId.identifierandBackMatterResource.uuid. An AST pass now places each pattern by the base type of itsAnnotated:strkeeps it, a number loses it, any other base fails the generation and names the site.regexdialect (scripts/regex_translation.py, PRS-007)..and\sdiffer between the dialects, so^\S(.*\S)?$accepteda<CR>b. The expected results for the 10 schema patterns come from Node.js (uflag).v1/_datatypes.py:OscalUri,OscalUriReference,OscalEmailAddress(each astrthat keeps the input text) andOscalDateTimeWithTimezone. The hoist names the four schema definitions withcustomTypePath. Each check runs in pydantic-core. They replaceAnyUrl(changedhttp://atohttp://a/),EmailStr(changed case, rejecteda@b), and the uncheckedAwareDatetimeandstr.pydantic, notpydantic[email].datatype-constraintsmain spec is added.build-resource/invalid-document-idnow passes (entry removed from the expected failures).Breaking
str, notAnyUrlorEmailStr.email-validatorthrough this package must install it.URIDatatype,EmailAddress,AnyUrl,AwareDatetime, andEmailStrleaveoscal_bindings.v1.models.Performance
Python 3.14.8, pydantic 2.14.2, pydantic-core 2.50.2, NIST SP 800-53 rev5 catalog (10.4 MB). Median of 7 per run, then the median of 3 runs; before and after alternate on one machine.
parse_oscal(bytes)serialize_oscal(doc, indent=None)serialize_oscal(doc)doc.model_dump_json()pydantic_core.to_json(doc)The after column is 2 to 5% lower in every row, also in rows that this change does not touch: read it as no slowdown. Import of
oscal_bindings: 159.5 ms before, 150.0 ms after. Peak memory ofparse_oscalfor the catalog: 36.5 MB before, 35.3 MB after.PERFORMANCE.mdhas the same numbers.Test plan
hatch run releasepasses (ruff, mypy, bandit, pip-audit,hatch test --all --cover, docs). 3,625 tests pass on Python 3.11 to 3.14, with 16 strict expected failures.hatch run generateleavesmodels.pyunchanged.tests/test_datatype_constraints.py), the translator against the 10 schema patterns, the four datatypes in JSON and Python input, a drift test of the pattern literals against the pinned schema, and a completeness test that every string site of the schema reaches a model.Notes for review
last_modified" does not match the behavior: a construction by name maps the name to its alias first, so the error location islast-modified. The test asserts the alias. The spec text may need a fix.translatealso passes\xHH,\uHHHH, and\u{…}through, so its output is its own fixed point and the post-processor can run on its own output. This goes beyond the list in design D2;DEVELOPING.mddocuments it.strsites that lost a pattern, where the proposal says 23.tests/test_hoist_names.pyrestores the original four definitions in the copy it validates, because the hoisted copy now differs from the artifact schema on purpose.