Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions .agents/summary/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,19 @@ graph LR
| `OscalBaseModel.__init__` | Maps field-name kwargs to aliases; leaves the name if the alias is also passed so `extra='forbid'` rejects it. Marked `__pydantic_base_init__ = True` so `model_validate*` doesn't route through it |
| `_kwarg_aliases(cls)` | `functools.cache`'d `{name: alias}` for alias-differing fields, excluding names that are also some field's alias |

### 1b. `v1/_datatypes.py` — the four datatypes that a pattern does not define (hand-written)

**Responsibility:** The types that the models use for the OSCAL datatypes `uri`, `uri-reference`, `email-address`, and `date-time-with-timezone`. The hoist names each definition with `customTypePath`, so codegen emits these types at each site. Imports only `pydantic`, `pydantic_core`, and stdlib.

| Symbol | Value type | Checks |
|--------|-----------|--------|
| `OscalUri` | `str` (the input text) | The OSCAL pattern, then the RFC 3986 `URI` grammar. Each scheme is valid |
| `OscalUriReference` | `str` (the input text) | The RFC 3986 `URI-reference` grammar |
| `OscalEmailAddress` | `str` (the input text) | The two OSCAL patterns, then the RFC 5321 `Mailbox` grammar with RFC 6531 |
| `OscalDateTimeWithTimezone` | `AwareDatetime` | The OSCAL pattern on the input string; in Python input also a `datetime` whose serialization matches it. One `oscal_date_time_with_timezone` error at the field |

Each check runs in pydantic-core with the Rust `regex` engine. The pattern literals are the output of `translate` for the schema patterns; a test fails when one drifts.

## 2. `v1/parser.py` — Runtime parse/serialize/validate

**Responsibility:** Convert JSON ⇄ models and validate.
Expand Down Expand Up @@ -126,8 +139,9 @@ Not part of the installed package. `scripts/` is not a package: a script imports
| `spec_artifact.py` | The one pin of the spec artifact (`0.8.0+oscal.1.2.3` + sha256), shared by `generate` and the tests; cache `tests/.corpus_cache/`. The artifact supplies the complete OSCAL schema (`artifact_paths(...).schema`), `names.json`, and the corpus, and the pin is the only selector of the OSCAL release | `SPEC_ARTIFACT_VERSION`, `SPEC_ARTIFACT_SHA256`, `SPEC_ARTIFACT`, `OSCAL_RELEASE`, `ensure_spec_artifact`, `artifact_paths`, `ArtifactPaths` |
| `pinned_archive.py` | Stdlib-only, cross-process-locked download (HTTPS only, HTTPS-only redirects, 64 MiB body cap), sha256 check before extraction, tar-slip-safe extraction with the `data` filter (Python 3.11.4+), atomic publish | `PinnedArchive`, `ensure_pinned_archive`, `http_get`, `build_https_opener`, `CorpusError`, `ResponseTooLargeError`, `FileLock` |
| `manifest.py` | Validates `names.json`: JSON Schema (`jsonschema-rs`), format/release, schema sha256 against the schema bytes, unique names, resolvable pointers, type references | `validate_manifest`, `load_validated_manifest(paths, *, schema_bytes=None, schema_path=None)` (no bytes and no path means `paths.schema`), `read_schema_bytes`, `ManifestError` |
| `hoist_names.py` | Reads the artifact schema from `paths.schema` one time, and writes `build/codegen/oscal_complete_schema.json`, in which each manifest site is `definitions/<Name>` with a `$ref` at the site | `hoist`, `HoistError`, `OUTPUT` |
| `postprocess_models.py` | Phase 1 structure fixes, adds union aliases / `MappingCollectionMappings` / `OscalDocument`, names scalar aliases, applies three guards against the manifest | `postprocess`, `guard_failures`, `PostprocessError`, `add_union_aliases`, `add_single_or_many`, `rename_document_union`, `scalar_renames`, `collapse_scalar_root_models`, `fix_non_string_patterns`, `scalar_names`, `import_bindings`, `artifact_schema_path`, `load_manifest(schema_bytes)`, `write_atomically`, `parse_args`/`main` (`--models`/`--schema`, each with its own default, `--schema` resolved in `main` to the artifact schema; `--output`, defaulting to `--models`) |
| `hoist_names.py` | Reads the artifact schema from `paths.schema` one time, and writes `build/codegen/oscal_complete_schema.json`, in which each manifest site is `definitions/<Name>` with a `$ref` at the site, and the four hand-written datatypes carry `customTypePath` | `hoist`, `HoistError`, `OUTPUT` |
| `postprocess_models.py` | Phase 1 structure fixes and pattern placement (`place_patterns`), adds union aliases / `MappingCollectionMappings` / `OscalDocument`, names scalar aliases, applies three guards against the manifest | `postprocess`, `guard_failures`, `PostprocessError`, `add_union_aliases`, `add_single_or_many`, `rename_document_union`, `scalar_renames`, `collapse_scalar_root_models`, `place_patterns`, `scalar_names`, `import_bindings`, `artifact_schema_path`, `load_manifest(schema_bytes)`, `write_atomically`, `parse_args`/`main` (`--models`/`--schema`, each with its own default, `--schema` resolved in `main` to the artifact schema; `--output`, defaulting to `--models`) |
| `regex_translation.py` | `translate`: maps an ECMA-262 pattern of the schema to the Rust `regex` dialect (`.` and `\s`); errors on lookaround, back-references, `\d`, `\w`, `\b`, an inline flag | `translate`, `RegexTranslationError` |
| `bench_serialization.py` | Parse and serialize time benchmark (numbers in `PERFORMANCE.md`) | — |
| `bench_memory.py` | Peak-memory benchmark of a parse (numbers in `PERFORMANCE.md`) | — |

Expand Down
7 changes: 5 additions & 2 deletions .agents/summary/data_models.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,11 @@ These recur across document types:
| By-name input | Constructor kwargs accept names or aliases (`BackMatterResourceRlink(media_type=...)`). `model_validate({...})` stays alias-only (`extra_forbidden` at any depth); JSON input rejects a name key the same way (needs pydantic ≥2.14.2; earlier 2.13/2.14 dropped it, [#6](https://github.com/cairn-proofs/oscal-bindings-python/issues/6)). No `validate_by_name` in config |
| Constraints | `Annotated[T, Field(...)]` with `pattern`, `min_length`, etc. (produced by `--field-constraints`) |
| Scalars | Collapsed to `TypeAlias` (not `RootModel`) — e.g. a `uuid` field is a constrained `str`, accessed directly |
| Datetimes | `AwareDatetime`; `pattern=` constraints stripped by the post-processor since they're invalid on non-str types |
| Emails | `EmailStr`; redundant `pattern=` stripped |
| Patterns | A `pattern=` on a `str` stays, translated from ECMA-262 to the Rust `regex` dialect (`scripts/regex_translation.py`, PRS-007); a pattern on a number is removed |
| Datetimes | `OscalDateTimeWithTimezone` (`v1/_datatypes.py`): an `AwareDatetime` whose input string must match the OSCAL pattern; in Python input a `datetime` must serialize to text that matches it. The value is a `datetime` (a seventh fraction digit is lost, [#19](https://github.com/cairn-proofs/oscal-bindings-python/issues/19)) |
| URIs | `OscalUri`: a `str` that matches the OSCAL pattern and the RFC 3986 `URI` grammar; each scheme is valid; the value is the input text, so a serialization writes it as given |
| URI references | `OscalUriReference`: a `str` that matches the RFC 3986 `URI-reference` grammar; the value is the input text |
| Emails | `OscalEmailAddress`: a `str` that matches both OSCAL patterns and the RFC 5321 `Mailbox` grammar with the RFC 6531 extension (non-ASCII, one-label domains); the value is the input text, so case and form are kept |

## Naming

Expand Down
4 changes: 2 additions & 2 deletions .agents/summary/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ Declared in `pyproject.toml` `[project.dependencies]`.

| Dependency | Constraint | Why it's needed |
|------------|-----------|-----------------|
| `pydantic[email]` | `>=2.14.2,<3` | The generated models are Pydantic v2 models (via `OscalBaseModel`). The `email` extra provides `EmailStr` validation used by contact fields. The 2.14.2 floor is the first release whose JSON validation rejects a Python-field-name key as `extra_forbidden` instead of dropping it ([#6](https://github.com/cairn-proofs/oscal-bindings-python/issues/6), GHSA-qpmf-fgh3-ff25); `ConfigDict(serialize_by_alias=True)` in `v1/_base.py` needs 2.11+. The runtime also uses v2-only APIs (`model_validate_json`, `model_dump_json`, `RootModel`, `model_serializer`). `tests/test_pydantic_version.py` guards the range. |
| `pydantic` | `>=2.14.2,<3` | The generated models are Pydantic v2 models (via `OscalBaseModel`). The hand-written datatypes of `v1/_datatypes.py` validate with pydantic-core's Rust `regex` engine, so the package needs no `email` extra and no `email-validator`. The 2.14.2 floor is the first release whose JSON validation rejects a Python-field-name key as `extra_forbidden` instead of dropping it ([#6](https://github.com/cairn-proofs/oscal-bindings-python/issues/6), GHSA-qpmf-fgh3-ff25); `ConfigDict(serialize_by_alias=True)` in `v1/_base.py` needs 2.11+. The runtime also uses v2-only APIs (`model_validate_json`, `model_dump_json`, `RootModel`, `model_serializer`). `tests/test_pydantic_version.py` guards the range. |

That is the **only** runtime dependency. The library is otherwise pure standard library (`pathlib`, `uuid`, `datetime`, `re`, `dataclasses`, `enum`, `typing`).

```mermaid
graph TD
APP["Downstream application"] --> OB["oscal_bindings"]
OB --> PYD["pydantic[email] >= 2.14.2, < 3"]
OB --> PYD["pydantic >= 2.14.2, < 3"]
OB --> STD["Python stdlib<br/>(pathlib, uuid, datetime, re, dataclasses)"]
```

Expand Down
7 changes: 5 additions & 2 deletions .agents/summary/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ flowchart TD
must be valid for the base datatype. An open-enum site replaces only the `enum` branch of its `anyOf`.
6. Property `title`/`description` stay beside the `$ref`, and a union branch without a `description` takes that of its
union, so fields and variant classes keep their docs.
7. The definitions of the four datatypes `URIDatatype`, `URIReferenceDatatype`, `EmailAddressDatatype`, and
`DateTimeWithTimezoneDatatype` become `{"type": "string", "customTypePath": "oscal_bindings.v1._datatypes.<Name>"}`
with their `description`. Codegen emits the hand-written type at each site, with no `pattern` beside it.

The output is deterministic: two runs give byte-identical copies.

Expand All @@ -48,8 +51,8 @@ The output is deterministic: two runs give byte-identical copies.
1. **`add_single_or_many`** — `MappingCollectionMappings` as `RootModel[Mapping | list[Mapping]]`, replacing the inline
union of the field.
2. **Phase 1** — `collapse_scalar_root_models` (scalar `RootModel[...]` → `Annotated` alias, manifest root models
kept), `ensure_annotated_import`, `fix_email_str_patterns` / `fix_non_string_patterns` (strip `pattern=` from
`EmailStr`, `AwareDatetime`, `AnyUrl`, and numeric fields).
kept), `ensure_annotated_import`, `place_patterns` (reads the base type of each `Annotated`: a `str` pattern stays,
translated by `regex_translation.translate`; a number pattern goes; a pattern on another base is a failure).
3. **`scalar_renames`** — each scalar alias is named from its definition key without the `oscal-complete-oscal-<module>:`
prefix (`LocationUuid`). A target name that is not PascalCase (`[A-Z][A-Za-z0-9]*`), or that a module-level import
binds, is a failure.
Expand Down
Loading
Loading