diff --git a/.agents/summary/architecture.md b/.agents/summary/architecture.md index c7c6abc..787dc57 100644 --- a/.agents/summary/architecture.md +++ b/.agents/summary/architecture.md @@ -11,7 +11,7 @@ There is no network server, no persistence layer, and no long-running process. T ```mermaid graph TB subgraph L0["Layer 0 — Schema (input)"] - S["oscal_complete_schema.json + names.json
(pinned spec artifact 0.7.0+oscal.1.2.3)"] + S["oscal_complete_schema.json + names.json
(pinned spec artifact 0.8.0+oscal.1.2.3)"] end subgraph L1["Layer 1 — Generated models"] BASE["v1/_base.py
OscalBaseModel (hand-written)"] @@ -81,8 +81,8 @@ The top-level submodule paths get real files — `oscal_bindings/models.py`, `os ```mermaid flowchart LR - N["spec artifact 0.7.0+oscal.1.2.3
names.json (sha256-pinned)"] -->|"manifest.py
validate"| H - A["spec artifact 0.7.0+oscal.1.2.3
oscal_complete_schema.json"] --> H["hoist_names.py"] + N["spec artifact 0.8.0+oscal.1.2.3
names.json (sha256-pinned)"] -->|"manifest.py
validate"| H + A["spec artifact 0.8.0+oscal.1.2.3
oscal_complete_schema.json"] --> H["hoist_names.py"] A --> C H --> HS["build/codegen/oscal_complete_schema.json
(each manifest site = definitions/<Name>)"] HS -->|"datamodel-codegen"| B["build/codegen/models.py
(raw: manifest-named classes,
RootModel scalars)"] diff --git a/.agents/summary/components.md b/.agents/summary/components.md index f9857cc..ff8fa94 100644 --- a/.agents/summary/components.md +++ b/.agents/summary/components.md @@ -119,7 +119,7 @@ Not part of the installed package. `scripts/` is not a package: a script imports | File | Responsibility | Notable symbols | |------|----------------|-----------------| -| `spec_artifact.py` | The one pin of the spec artifact (`0.7.0+oscal.1.2.3` + sha256), shared by `generate` and the tests; cache `tests/.corpus_cache/`. The artifact supplies the complete OSCAL schema (`artifact_paths(...).schema`), `names.json`, and the corpus, and the pin is the only selector of the OSCAL release | `SPEC_ARTIFACT_VERSION`, `SPEC_ARTIFACT_SHA256`, `SPEC_ARTIFACT`, `OSCAL_RELEASE`, `ensure_spec_artifact`, `artifact_paths`, `ArtifactPaths` | +| `spec_artifact.py` | The one pin of the spec artifact (`0.8.0+oscal.1.2.3` + sha256), shared by `generate` and the tests; cache `tests/.corpus_cache/`. The artifact supplies the complete OSCAL schema (`artifact_paths(...).schema`), `names.json`, and the corpus, and the pin is the only selector of the OSCAL release | `SPEC_ARTIFACT_VERSION`, `SPEC_ARTIFACT_SHA256`, `SPEC_ARTIFACT`, `OSCAL_RELEASE`, `ensure_spec_artifact`, `artifact_paths`, `ArtifactPaths` | | `pinned_archive.py` | Stdlib-only, cross-process-locked download (HTTPS only, HTTPS-only redirects, 64 MiB body cap), sha256 check before extraction, tar-slip-safe extraction with the `data` filter (Python 3.11.4+), atomic publish | `PinnedArchive`, `ensure_pinned_archive`, `http_get`, `build_https_opener`, `CorpusError`, `ResponseTooLargeError`, `FileLock` | | `manifest.py` | Validates `names.json`: JSON Schema (`jsonschema-rs`), format/release, schema sha256 against the schema bytes, unique names, resolvable pointers, type references | `validate_manifest`, `load_validated_manifest(paths, *, schema_bytes=None, schema_path=None)` (no bytes and no path means `paths.schema`), `read_schema_bytes`, `ManifestError` | | `hoist_names.py` | Reads the artifact schema from `paths.schema` one time, and writes `build/codegen/oscal_complete_schema.json`, in which each manifest site is `definitions/` with a `$ref` at the site | `hoist`, `HoistError`, `OUTPUT` | diff --git a/.agents/summary/dependencies.md b/.agents/summary/dependencies.md index 8ec80f3..43ff5a3 100644 --- a/.agents/summary/dependencies.md +++ b/.agents/summary/dependencies.md @@ -64,7 +64,7 @@ The pin is the only selector of the schema and its release. Codegen reads the ho ## Spec Artifact (not a package dependency) -`scripts/spec_artifact.py` pins the spec artifact of [cairn-proofs/oscal-bindings-specs](https://github.com/cairn-proofs/oscal-bindings-specs) (`0.7.0+oscal.1.2.3`) by sha256. It holds the complete OSCAL JSON Schema, the naming manifest (`names.json` and its JSON Schema), and the conformance corpus. `hatch run generate` and `hatch test` share the pin and the cache `tests/.corpus_cache/`, so a cold cache needs network access once. An sdist checkout needs network access too, to run `generate`. +`scripts/spec_artifact.py` pins the spec artifact of [cairn-proofs/oscal-bindings-specs](https://github.com/cairn-proofs/oscal-bindings-specs) (`0.8.0+oscal.1.2.3`) by sha256. It holds the complete OSCAL JSON Schema, the naming manifest (`names.json` and its JSON Schema), and the conformance corpus. `hatch run generate` and `hatch test` share the pin and the cache `tests/.corpus_cache/`, so a cold cache needs network access once. An sdist checkout needs network access too, to run `generate`. ## Dependency Risk Notes diff --git a/.agents/summary/index.md b/.agents/summary/index.md index 6f99ba0..abdbcb9 100644 --- a/.agents/summary/index.md +++ b/.agents/summary/index.md @@ -46,14 +46,14 @@ The implementation lives in the major-version package **`oscal_bindings.v1`**; t - **8 OSCAL document types:** Catalog, Profile, Component Definition, System Security Plan, Assessment Plan, Assessment Results, Plan of Action and Milestones, Mapping Collection. - **Layout:** implementation under `src/oscal_bindings/v1/`; top-level package re-exports it through explicit alias modules at `oscal_bindings/models.py`, `oscal_bindings/parser.py`, `oscal_bindings/extensions/__init__.py`. -- **Active schema:** `oscal_complete_schema.json` of the spec artifact pinned in `scripts/spec_artifact.py` (`0.7.0+oscal.1.2.3`, by sha256). The repo holds no copy of the schema. The pin is the only selector of the schema and its release. +- **Active schema:** `oscal_complete_schema.json` of the spec artifact pinned in `scripts/spec_artifact.py` (`0.8.0+oscal.1.2.3`, by sha256). The repo holds no copy of the schema. The pin is the only selector of the schema and its release. - **`__oscal_schema_version__`** (`"1.2.3"`) on `oscal_bindings.v1`, re-exported at the top level, records the exact release the models came from. - **Entry points:** `parse_oscal`, `parse_oscal_file`, 8 typed `parse_*` functions, `serialize_oscal`, `validate_oscal`, `OscalDoc`, `make_hash`/`make_rlink`/`make_resource`, `validate_element`/`get_supported_element_types`. - **Parsers accept `str | bytes`;** bytes skips a UTF-8 decode (useful for HTTP/S3/file byte-sources). - **`v1/models.py` is GENERATED** by `scripts/hoist_names.py` + `datamodel-codegen` + `scripts/postprocess_models.py`. Change the hoist/post-processor (or the store manifest for names) and run `hatch run generate` — never hand-edit. A cold cache needs network access. (`oscal_bindings/models.py` at the top level is a hand-written alias module, not generated.) - **`v1/__init__.py` only re-exports** — the entire public API is visible there; the top level derives its `__all__` from it. - **Extensions pattern:** new helpers go in `src/oscal_bindings/v1/extensions/`, re-exported up through `v1/extensions/__init__.py` then `v1/__init__.py`. -- **Names come only from `names.json`** of the spec artifact pinned in `scripts/spec_artifact.py` (`0.7.0+oscal.1.2.3`). The hoist makes each manifest site a named definition; post-processor guards fail on any name outside the manifest. Never rename in the post-processor. +- **Names come only from `names.json`** of the spec artifact pinned in `scripts/spec_artifact.py` (`0.8.0+oscal.1.2.3`). The hoist makes each manifest site a named definition; post-processor guards fail on any name outside the manifest. Never rename in the post-processor. - **Namespaces are major-scoped** (`v1`); minor-version namespacing was considered and rejected on compatibility-boundary grounds. OSCAL 2.0 is the trigger for a second version package (see `DEVELOPING.md` Key Design Decisions). - **Exceptions:** `OscalParseError` (has `.errors`), `OscalAccessError`. diff --git a/.agents/summary/review_notes.md b/.agents/summary/review_notes.md index 355bfd2..ac5b117 100644 --- a/.agents/summary/review_notes.md +++ b/.agents/summary/review_notes.md @@ -33,7 +33,7 @@ No contradictions found across the document set. | `OscalDoc` facade incl. `assessment_period()` | Complete | Including nested-task walking and AP/AR-only constraint | | Builders | Complete | Including `document_ids` validation at construction and the `media-type` alias | | Element validation | Complete | `_ELEMENT_MAP` derivation explained | -| Code generation + post-processing | Complete | Manifest validation, the hoist and its guards, the post-processor steps and its three guards against the manifest, and the CLI contracts. The pipeline takes every name from the naming manifest (spec artifact `0.7.0+oscal.1.2.3`, package version 0.4.0); these docs describe that pipeline only | +| Code generation + post-processing | Complete | Manifest validation, the hoist and its guards, the post-processor steps and its three guards against the manifest, and the CLI contracts. The pipeline takes every name from the naming manifest (spec artifact `0.8.0+oscal.1.2.3`, package version 0.4.0); these docs describe that pipeline only | | Version layout + import paths | Complete | Version package, top-level re-exports, alias modules, and the import-path guidance are covered in architecture.md, components.md, and interfaces.md | | Schema refresh procedure | Complete | workflows.md §1c documents how to move the spec artifact pin and what a reviewer checks | | Dependencies / toolchain / Python versions | Complete | Runtime, codegen, dev, build tooling all covered | diff --git a/.agents/summary/workflows.md b/.agents/summary/workflows.md index 890c9e3..a00f992 100644 --- a/.agents/summary/workflows.md +++ b/.agents/summary/workflows.md @@ -7,7 +7,7 @@ the naming manifest (`names.json`) of the pinned spec artifact. The binding hold ```mermaid flowchart TD - A["hatch run generate"] --> P["scripts/spec_artifact.py pin
0.7.0+oscal.1.2.3 (sha256)
cache: tests/.corpus_cache/"] + A["hatch run generate"] --> P["scripts/spec_artifact.py pin
0.8.0+oscal.1.2.3 (sha256)
cache: tests/.corpus_cache/"] P --> M["scripts/manifest.py
validate names.json"] M -->|"invalid"| X["non-zero exit,
nothing written"] M --> H["python scripts/hoist_names.py
artifact oscal_complete_schema.json
→ build/codegen/oscal_complete_schema.json"] diff --git a/AGENTS.md b/AGENTS.md index 5a8a972..ee3f865 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ Neither `v1/__init__.py` nor the top-level `__init__.py` defines symbols — bot Names come from the store's naming manifest. The binding holds no naming rule for a manifest type. `hatch run generate` runs three steps: -1. **Hoist.** `python scripts/hoist_names.py` fetches the spec artifact pinned in `scripts/spec_artifact.py` (`0.7.0+oscal.1.2.3`, sha256-checked, cached in `tests/.corpus_cache/`), validates its `names.json` with `scripts/manifest.py` (manifest JSON Schema via `jsonschema-rs`, format and release, artifact-schema sha256, unique names, resolvable pointers), reads the complete OSCAL schema of the same artifact one time (the manifest check hashes the bytes that the hoist parses), and writes `build/codegen/oscal_complete_schema.json` (git-ignored). In that copy each manifest site is `definitions/` with a `$ref` at the site. A merge with different bodies, or a closed-enum value that its base datatype rejects, fails the run. +1. **Hoist.** `python scripts/hoist_names.py` fetches the spec artifact pinned in `scripts/spec_artifact.py` (`0.8.0+oscal.1.2.3`, sha256-checked, cached in `tests/.corpus_cache/`), validates its `names.json` with `scripts/manifest.py` (manifest JSON Schema via `jsonschema-rs`, format and release, artifact-schema sha256, unique names, resolvable pointers), reads the complete OSCAL schema of the same artifact one time (the manifest check hashes the bytes that the hoist parses), and writes `build/codegen/oscal_complete_schema.json` (git-ignored). In that copy each manifest site is `definitions/` with a `$ref` at the site. A merge with different bodies, or a closed-enum value that its base datatype rejects, fails the run. 2. **Codegen.** `datamodel-codegen --input ./build/codegen/oscal_complete_schema.json --output ./build/codegen/models.py --base-class oscal_bindings.v1._base.OscalBaseModel --collapse-root-models ... --formatters black isort` (see DEVELOPING "Names reach codegen through the hoisted schema" for the flag choices). Each `object`, `document`, and `enum` class takes its name from its definition key. `--base-class` makes all 185 object models subclass `OscalBaseModel`; the two `RootModel`s (`OscalDocument`, `MappingCollectionMappings`) don't. 3. **Post-process.** `python scripts/postprocess_models.py --models ./build/codegen/models.py --output src/oscal_bindings/v1/models.py`. Phase 1 collapses scalar `RootModel`s to `Annotated` aliases and strips `pattern=` from non-string fields (`AwareDatetime`/`AnyUrl`/numeric/`EmailStr`). Then it adds the 13 union `TypeAlias`es (`CatalogGroup`, plain unions, so `isinstance` works) and writes each field that uses one as `Annotated[, Field(union_mode='left_to_right')]` (the variant properties are exclusive, so this picks the same variant as smart mode, faster, #43), the `MappingCollectionMappings` root model, and the `OscalDocument` root model. It discriminates `OscalDocument.root` on the root key of the input: each member is `Tag`-ged with its class name, the root key of each member is the manifest `body-field`, and `root_key_discriminator` (`v1/_base.py`) selects the one member to validate (an input with no known root key is one `oscal_document_type` error). It names each scalar alias from its definition key without the `oscal-complete-oscal-:` prefix. Three guards compare the result with the manifest and the artifact schema and report every offender: each manifest name is defined once in the form of its kind, each other name is a scalar alias named for a scalar definition of the schema, no name is defined twice, and no manifest name or scalar alias is bound by a module-level import. The manifest must record the sha256 of the `--schema` bytes. The completed module goes to `--output` through a temp file + `os.replace`; on a failure nothing is written to `--output`, so a failed `generate` leaves the committed `models.py` untouched. `generate` passes no `--schema`: an omitted `--schema` resolves at call time to `artifact_paths(ensure_spec_artifact()).schema`, and an explicit one (a candidate schema, or a small test schema) must still match `schema.sha256` of the manifest. `--models` falls back to its own default, `--output` defaults to `--models`; a missing path is a non-zero exit. diff --git a/DEVELOPING.md b/DEVELOPING.md index b61a484..d50c311 100644 --- a/DEVELOPING.md +++ b/DEVELOPING.md @@ -105,7 +105,7 @@ Every type name comes from the naming manifest, `names.json`, of the pinned spec rule for a manifest type. `hatch run generate` runs three steps: 1. `scripts/hoist_names.py` writes `build/codegen/oscal_complete_schema.json` (git-ignored). - 1. It fetches the spec artifact that `scripts/spec_artifact.py` pins (`0.7.0+oscal.1.2.3`, by sha256) through + 1. It fetches the spec artifact that `scripts/spec_artifact.py` pins (`0.8.0+oscal.1.2.3`, by sha256) through `scripts/pinned_archive.py`. 2. `scripts/manifest.py` validates `names.json`: its JSON Schema from the same artifact (with `jsonschema-rs`), the format version and OSCAL release, the sha256 of the artifact schema, unique names, and pointers that resolve. diff --git a/scripts/spec_artifact.py b/scripts/spec_artifact.py index 4807d34..1b671b9 100644 --- a/scripts/spec_artifact.py +++ b/scripts/spec_artifact.py @@ -21,9 +21,9 @@ if TYPE_CHECKING: from collections.abc import Callable -SPEC_ARTIFACT_VERSION = "0.7.0+oscal.1.2.3" +SPEC_ARTIFACT_VERSION = "0.8.0+oscal.1.2.3" SPEC_ARTIFACT_SHA256 = ( - "39f61f5c40200882106d9df6367d5aa5ccc6d604288741a8a70e93e119fa933e" + "3da20cc6127e033d82e47046435caefe2b3a7ad7554908da264e5f4fe7f48dda" ) # The `+` of the version is written as %2B in the URL path. _URL_VERSION = SPEC_ARTIFACT_VERSION.replace("+", "%2B") diff --git a/tests/support/expected_failures.py b/tests/support/expected_failures.py index a3ae0f0..aa3e362 100644 --- a/tests/support/expected_failures.py +++ b/tests/support/expected_failures.py @@ -18,4 +18,44 @@ # A round trip keeps six datetime fraction digits and drops the seventh. "round-trip/seven-fraction-digits": f"{ISSUES}/19", "yaml-round-trip/string-scalars": f"{ISSUES}/19", + # PRS-014: only failures of the selected nested-union branch. + "parse/closed-enum-out-of-set": f"{ISSUES}/69", + # ERR-012: read-error category for a file path that cannot be read. + "parse-file/directory": f"{ISSUES}/61", + # FAC-007: datetime bounds from assessment_period(). + "assessment-period/mixed-offsets": f"{ISSUES}/58", + "assessment-period/equal-instant": f"{ISSUES}/58", + "assessment-period/seventh-fraction-digit": f"{ISSUES}/58", + # ELV-002: one element-type name for each manifest type. + "element-types/all": f"{ISSUES}/59", + # BMB-004, ERR-013: resource identifier from a key, builder-error category. + "build-resource/key": f"{ISSUES}/62", + "build-resource/key-with-title": f"{ISSUES}/62", + "build-resource/non-ascii-key": f"{ISSUES}/62", + "build-resource/key-and-identifier": f"{ISSUES}/62", + "build-resource/empty-key": f"{ISSUES}/62", + # FAC-001 to FAC-003, BMB-001 to BMB-003: the runner lacks these operations. + **dict.fromkeys( + ( + "facade/catalog", + "facade/profile", + "facade/component-definition", + "facade/system-security-plan", + "facade/assessment-plan", + "facade/assessment-results", + "facade/plan-of-action-and-milestones", + "facade/mapping-collection", + "facade/missing-file", + "build-hash/default-algorithm", + "build-hash/algorithm-string", + "build-hash/off-list-algorithm", + "build-hash/invalid-algorithm", + "build-rlink/reference-only", + "build-rlink/media-type-and-hashes", + "build-resource/generated-identifier", + "build-resource/omitted-inputs", + "build-resource/invalid-document-id", + ), + f"{ISSUES}/63", + ), } diff --git a/tests/test_conformance.py b/tests/test_conformance.py index 472e0f8..41924a4 100644 --- a/tests/test_conformance.py +++ b/tests/test_conformance.py @@ -211,6 +211,11 @@ def test_every_expected_failure_names_a_corpus_case() -> None: assert stale == [], f"expected failures not in the pinned corpus: {stale}" +@pytest.mark.xfail( + strict=True, + reason="read-error and builder-error are not mapped yet: " + "https://github.com/cairn-proofs/oscal-bindings-python/issues/61, /62", +) def test_every_category_is_mapped() -> None: """Each category in the corpus has a branch in ``category``.""" index = json.loads((corpus_root() / "cases.json").read_text("utf-8")) diff --git a/tests/test_spec_artifact_script.py b/tests/test_spec_artifact_script.py index db60529..e3c826b 100644 --- a/tests/test_spec_artifact_script.py +++ b/tests/test_spec_artifact_script.py @@ -28,9 +28,9 @@ def test_script_and_test_support_share_one_pin_and_one_cache() -> None: assert spec_artifact.SPEC_ARTIFACT is pin.SPEC_ARTIFACT - assert pin.SPEC_ARTIFACT_VERSION == "0.7.0+oscal.1.2.3" + assert pin.SPEC_ARTIFACT_VERSION == "0.8.0+oscal.1.2.3" assert pin.SPEC_ARTIFACT_SHA256 == ( - "39f61f5c40200882106d9df6367d5aa5ccc6d604288741a8a70e93e119fa933e" + "3da20cc6127e033d82e47046435caefe2b3a7ad7554908da264e5f4fe7f48dda" ) assert pin.CACHE_ROOT.resolve() == corpus.CACHE_ROOT.resolve() assert REPO_ROOT.resolve() / "tests" / ".corpus_cache" == pin.CACHE_ROOT