Skip to content

fix!: enforce the OSCAL datatype constraints in the models - #79

Merged
dfkunstler merged 1 commit into
mainfrom
fix/77-enforce-oscal-datatype-constraints
Nov 17, 2026
Merged

dfkunstler merged 1 commit into
mainfrom
fix/77-enforce-oscal-datatype-constraints

Conversation

@dfkunstler

Copy link
Copy Markdown
Member

Summary

Closes #77.

  1. fix_non_string_patterns removed pattern= from str fields when a type name such as int occurred in the 10 lines before it ("integrity" is enough). 22 fields lost their pattern, among them DocumentId.identifier and BackMatterResource.uuid. An AST pass now places each pattern by the base type of its Annotated: str keeps it, a number loses it, any other base fails the generation and names the site.
  2. Patterns are translated from ECMA-262 to the Rust regex dialect (scripts/regex_translation.py, PRS-007). . and \s differ between the dialects, so ^\S(.*\S)?$ accepted a<CR>b. The expected results for the 10 schema patterns come from Node.js (u flag).
  3. New v1/_datatypes.py: OscalUri, OscalUriReference, OscalEmailAddress (each a str that keeps the input text) and OscalDateTimeWithTimezone. The hoist names the four schema definitions with customTypePath. Each check runs in pydantic-core. They replace AnyUrl (changed http://a to http://a/), EmailStr (changed case, rejected a@b), and the unchecked AwareDatetime and str.
  4. The runtime dependency is pydantic, not pydantic[email].
  5. The OpenSpec change is archived and the datatype-constraints main spec is added.

build-resource/invalid-document-id now passes (entry removed from the expected failures).

Breaking

  1. The models reject values that they accepted before.
  2. A uri, uri-reference, or email-address field holds a str, not AnyUrl or EmailStr.
  3. A caller that uses email-validator through this package must install it.
  4. URIDatatype, EmailAddress, AnyUrl, AwareDatetime, and EmailStr leave oscal_bindings.v1.models.

Performance

Python 3.14.8, pydantic 2.14.2, pydantic-core 2.50.2, NIST SP 800-53 rev5 catalog (10.4 MB). Median of 7 per run, then the median of 3 runs; before and after alternate on one machine.

Path Before, GC on ms After, GC on ms Before, GC off ms After, GC off ms
parse_oscal(bytes) 64.0 61.1 47.2 45.6
serialize_oscal(doc, indent=None) 135.6 132.7 128.8 125.7
serialize_oscal(doc) 144.0 140.4 137.6 134.1
doc.model_dump_json() 142.0 138.8 135.3 132.4
pydantic_core.to_json(doc) 143.0 138.6 136.0 132.7

The after column is 2 to 5% lower in every row, also in rows that this change does not touch: read it as no slowdown. Import of oscal_bindings: 159.5 ms before, 150.0 ms after. Peak memory of parse_oscal for the catalog: 36.5 MB before, 35.3 MB after. PERFORMANCE.md has the same numbers.

Test plan

  1. hatch run release passes (ruff, mypy, bandit, pip-audit, hatch test --all --cover, docs). 3,625 tests pass on Python 3.11 to 3.14, with 16 strict expected failures.
  2. A second hatch run generate leaves models.py unchanged.
  3. The NIST and AWS corpora parse and validate against the JSON Schema under the stricter rules.
  4. New tests: one for each scenario of the spec (tests/test_datatype_constraints.py), the translator against the 10 schema patterns, the four datatypes in JSON and Python input, a drift test of the pattern literals against the pinned schema, and a completeness test that every string site of the schema reaches a model.

Notes for review

  1. The spec scenario "construction reports one error at last_modified" does not match the behavior: a construction by name maps the name to its alias first, so the error location is last-modified. The test asserts the alias. The spec text may need a fix.
  2. translate also passes \xHH, \uHHHH, and \u{…} through, so its output is its own fixed point and the post-processor can run on its own output. This goes beyond the list in design D2; DEVELOPING.md documents it.
  3. The completeness test finds 22 str sites that lost a pattern, where the proposal says 23.
  4. tests/test_hoist_names.py restores the original four definitions in the copy it validates, because the hoisted copy now differs from the artifact schema on purpose.

The post-processor removed `pattern=` from 22 `str` fields when a type name
such as `int` occurred in the 10 lines before the pattern (the word
"integrity" is enough). The models also accepted values that OSCAL rejects
and changed values that OSCAL accepts.

- Place each pattern by the base type of its `Annotated`, with an AST pass: a
  `str` pattern stays, a number pattern goes, a pattern on any other base
  fails the generation and names it.
- Translate each pattern from ECMA-262 to the Rust `regex` dialect
  (`scripts/regex_translation.py`), so `.` and `\s` match what the schema
  means (PRS-007).
- Add `v1/_datatypes.py` with `OscalUri`, `OscalUriReference`,
  `OscalEmailAddress`, and `OscalDateTimeWithTimezone`. The hoist names the
  four schema definitions with `customTypePath`. Each check runs in
  pydantic-core. A uri, uri-reference, or email value keeps the input text.
- Depend on `pydantic`, not `pydantic[email]`.
- Archive the change and add the `datatype-constraints` main spec.

`build-resource/invalid-document-id` now passes.

BREAKING CHANGE: the models reject values that they accepted before: values
that a pattern, the RFC 3986 `URI` and `URI-reference` grammars, the RFC 5321
`Mailbox` grammar, or the date-time pattern of the schema rejects. A uri,
uri-reference, or email-address field holds a `str`, not `AnyUrl` or
`EmailStr`, and a serialization writes the input text. The runtime dependency
is `pydantic` without the `email` extra: a caller that uses `email-validator`
through this package must install it. The module-level names `URIDatatype`,
`EmailAddress`, `AnyUrl`, `AwareDatetime`, and `EmailStr` leave
`oscal_bindings.v1.models`.

Closes #77
@dfkunstler
dfkunstler merged commit 89d824c into main Nov 17, 2026
12 checks passed
@dfkunstler
dfkunstler deleted the fix/77-enforce-oscal-datatype-constraints branch November 17, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Post-processor removes pattern= from string fields whose context has the text 'int'

1 participant