Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .agents/summary/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ There is no network server, no persistence layer, and no long-running process. T
```mermaid
graph TB
subgraph L0["Layer 0 — Schema (input)"]
S["oscal_complete_schema.json + names.json<br/>(pinned spec artifact 0.7.0+oscal.1.2.3)"]
S["oscal_complete_schema.json + names.json<br/>(pinned spec artifact 0.8.0+oscal.1.2.3)"]
end
subgraph L1["Layer 1 — Generated models"]
BASE["v1/_base.py<br/>OscalBaseModel (hand-written)"]
Expand Down Expand Up @@ -81,8 +81,8 @@ The top-level submodule paths get real files — `oscal_bindings/models.py`, `os

```mermaid
flowchart LR
N["spec artifact 0.7.0+oscal.1.2.3<br/>names.json (sha256-pinned)"] -->|"manifest.py<br/>validate"| H
A["spec artifact 0.7.0+oscal.1.2.3<br/>oscal_complete_schema.json"] --> H["hoist_names.py"]
N["spec artifact 0.8.0+oscal.1.2.3<br/>names.json (sha256-pinned)"] -->|"manifest.py<br/>validate"| H
A["spec artifact 0.8.0+oscal.1.2.3<br/>oscal_complete_schema.json"] --> H["hoist_names.py"]
A --> C
H --> HS["build/codegen/oscal_complete_schema.json<br/>(each manifest site = definitions/&lt;Name&gt;)"]
HS -->|"datamodel-codegen"| B["build/codegen/models.py<br/>(raw: manifest-named classes,<br/>RootModel scalars)"]
Expand Down
2 changes: 1 addition & 1 deletion .agents/summary/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Not part of the installed package. `scripts/` is not a package: a script imports

| File | Responsibility | Notable symbols |
|------|----------------|-----------------|
| `spec_artifact.py` | The one pin of the spec artifact (`0.7.0+oscal.1.2.3` + sha256), shared by `generate` and the tests; cache `tests/.corpus_cache/`. The artifact supplies the complete OSCAL schema (`artifact_paths(...).schema`), `names.json`, and the corpus, and the pin is the only selector of the OSCAL release | `SPEC_ARTIFACT_VERSION`, `SPEC_ARTIFACT_SHA256`, `SPEC_ARTIFACT`, `OSCAL_RELEASE`, `ensure_spec_artifact`, `artifact_paths`, `ArtifactPaths` |
| `spec_artifact.py` | The one pin of the spec artifact (`0.8.0+oscal.1.2.3` + sha256), shared by `generate` and the tests; cache `tests/.corpus_cache/`. The artifact supplies the complete OSCAL schema (`artifact_paths(...).schema`), `names.json`, and the corpus, and the pin is the only selector of the OSCAL release | `SPEC_ARTIFACT_VERSION`, `SPEC_ARTIFACT_SHA256`, `SPEC_ARTIFACT`, `OSCAL_RELEASE`, `ensure_spec_artifact`, `artifact_paths`, `ArtifactPaths` |
| `pinned_archive.py` | Stdlib-only, cross-process-locked download (HTTPS only, HTTPS-only redirects, 64 MiB body cap), sha256 check before extraction, tar-slip-safe extraction with the `data` filter (Python 3.11.4+), atomic publish | `PinnedArchive`, `ensure_pinned_archive`, `http_get`, `build_https_opener`, `CorpusError`, `ResponseTooLargeError`, `FileLock` |
| `manifest.py` | Validates `names.json`: JSON Schema (`jsonschema-rs`), format/release, schema sha256 against the schema bytes, unique names, resolvable pointers, type references | `validate_manifest`, `load_validated_manifest(paths, *, schema_bytes=None, schema_path=None)` (no bytes and no path means `paths.schema`), `read_schema_bytes`, `ManifestError` |
| `hoist_names.py` | Reads the artifact schema from `paths.schema` one time, and writes `build/codegen/oscal_complete_schema.json`, in which each manifest site is `definitions/<Name>` with a `$ref` at the site | `hoist`, `HoistError`, `OUTPUT` |
Expand Down
2 changes: 1 addition & 1 deletion .agents/summary/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The pin is the only selector of the schema and its release. Codegen reads the ho

## Spec Artifact (not a package dependency)

`scripts/spec_artifact.py` pins the spec artifact of [cairn-proofs/oscal-bindings-specs](https://github.com/cairn-proofs/oscal-bindings-specs) (`0.7.0+oscal.1.2.3`) by sha256. It holds the complete OSCAL JSON Schema, the naming manifest (`names.json` and its JSON Schema), and the conformance corpus. `hatch run generate` and `hatch test` share the pin and the cache `tests/.corpus_cache/`, so a cold cache needs network access once. An sdist checkout needs network access too, to run `generate`.
`scripts/spec_artifact.py` pins the spec artifact of [cairn-proofs/oscal-bindings-specs](https://github.com/cairn-proofs/oscal-bindings-specs) (`0.8.0+oscal.1.2.3`) by sha256. It holds the complete OSCAL JSON Schema, the naming manifest (`names.json` and its JSON Schema), and the conformance corpus. `hatch run generate` and `hatch test` share the pin and the cache `tests/.corpus_cache/`, so a cold cache needs network access once. An sdist checkout needs network access too, to run `generate`.

## Dependency Risk Notes

Expand Down
4 changes: 2 additions & 2 deletions .agents/summary/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,14 @@ The implementation lives in the major-version package **`oscal_bindings.v1`**; t

- **8 OSCAL document types:** Catalog, Profile, Component Definition, System Security Plan, Assessment Plan, Assessment Results, Plan of Action and Milestones, Mapping Collection.
- **Layout:** implementation under `src/oscal_bindings/v1/`; top-level package re-exports it through explicit alias modules at `oscal_bindings/models.py`, `oscal_bindings/parser.py`, `oscal_bindings/extensions/__init__.py`.
- **Active schema:** `oscal_complete_schema.json` of the spec artifact pinned in `scripts/spec_artifact.py` (`0.7.0+oscal.1.2.3`, by sha256). The repo holds no copy of the schema. The pin is the only selector of the schema and its release.
- **Active schema:** `oscal_complete_schema.json` of the spec artifact pinned in `scripts/spec_artifact.py` (`0.8.0+oscal.1.2.3`, by sha256). The repo holds no copy of the schema. The pin is the only selector of the schema and its release.
- **`__oscal_schema_version__`** (`"1.2.3"`) on `oscal_bindings.v1`, re-exported at the top level, records the exact release the models came from.
- **Entry points:** `parse_oscal`, `parse_oscal_file`, 8 typed `parse_*` functions, `serialize_oscal`, `validate_oscal`, `OscalDoc`, `make_hash`/`make_rlink`/`make_resource`, `validate_element`/`get_supported_element_types`.
- **Parsers accept `str | bytes`;** bytes skips a UTF-8 decode (useful for HTTP/S3/file byte-sources).
- **`v1/models.py` is GENERATED** by `scripts/hoist_names.py` + `datamodel-codegen` + `scripts/postprocess_models.py`. Change the hoist/post-processor (or the store manifest for names) and run `hatch run generate` — never hand-edit. A cold cache needs network access. (`oscal_bindings/models.py` at the top level is a hand-written alias module, not generated.)
- **`v1/__init__.py` only re-exports** — the entire public API is visible there; the top level derives its `__all__` from it.
- **Extensions pattern:** new helpers go in `src/oscal_bindings/v1/extensions/`, re-exported up through `v1/extensions/__init__.py` then `v1/__init__.py`.
- **Names come only from `names.json`** of the spec artifact pinned in `scripts/spec_artifact.py` (`0.7.0+oscal.1.2.3`). The hoist makes each manifest site a named definition; post-processor guards fail on any name outside the manifest. Never rename in the post-processor.
- **Names come only from `names.json`** of the spec artifact pinned in `scripts/spec_artifact.py` (`0.8.0+oscal.1.2.3`). The hoist makes each manifest site a named definition; post-processor guards fail on any name outside the manifest. Never rename in the post-processor.
- **Namespaces are major-scoped** (`v1`); minor-version namespacing was considered and rejected on compatibility-boundary grounds. OSCAL 2.0 is the trigger for a second version package (see `DEVELOPING.md` Key Design Decisions).
- **Exceptions:** `OscalParseError` (has `.errors`), `OscalAccessError`.

Expand Down
2 changes: 1 addition & 1 deletion .agents/summary/review_notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ No contradictions found across the document set.
| `OscalDoc` facade incl. `assessment_period()` | Complete | Including nested-task walking and AP/AR-only constraint |
| Builders | Complete | Including `document_ids` validation at construction and the `media-type` alias |
| Element validation | Complete | `_ELEMENT_MAP` derivation explained |
| Code generation + post-processing | Complete | Manifest validation, the hoist and its guards, the post-processor steps and its three guards against the manifest, and the CLI contracts. The pipeline takes every name from the naming manifest (spec artifact `0.7.0+oscal.1.2.3`, package version 0.4.0); these docs describe that pipeline only |
| Code generation + post-processing | Complete | Manifest validation, the hoist and its guards, the post-processor steps and its three guards against the manifest, and the CLI contracts. The pipeline takes every name from the naming manifest (spec artifact `0.8.0+oscal.1.2.3`, package version 0.4.0); these docs describe that pipeline only |
| Version layout + import paths | Complete | Version package, top-level re-exports, alias modules, and the import-path guidance are covered in architecture.md, components.md, and interfaces.md |
| Schema refresh procedure | Complete | workflows.md §1c documents how to move the spec artifact pin and what a reviewer checks |
| Dependencies / toolchain / Python versions | Complete | Runtime, codegen, dev, build tooling all covered |
Expand Down
2 changes: 1 addition & 1 deletion .agents/summary/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ the naming manifest (`names.json`) of the pinned spec artifact. The binding hold

```mermaid
flowchart TD
A["hatch run generate"] --> P["scripts/spec_artifact.py pin<br/>0.7.0+oscal.1.2.3 (sha256)<br/>cache: tests/.corpus_cache/"]
A["hatch run generate"] --> P["scripts/spec_artifact.py pin<br/>0.8.0+oscal.1.2.3 (sha256)<br/>cache: tests/.corpus_cache/"]
P --> M["scripts/manifest.py<br/>validate names.json"]
M -->|"invalid"| X["non-zero exit,<br/>nothing written"]
M --> H["python scripts/hoist_names.py<br/>artifact oscal_complete_schema.json<br/>→ build/codegen/oscal_complete_schema.json"]
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Neither `v1/__init__.py` nor the top-level `__init__.py` defines symbols — bot

Names come from the store's naming manifest. The binding holds no naming rule for a manifest type. `hatch run generate` runs three steps:

1. **Hoist.** `python scripts/hoist_names.py` fetches the spec artifact pinned in `scripts/spec_artifact.py` (`0.7.0+oscal.1.2.3`, sha256-checked, cached in `tests/.corpus_cache/`), validates its `names.json` with `scripts/manifest.py` (manifest JSON Schema via `jsonschema-rs`, format and release, artifact-schema sha256, unique names, resolvable pointers), reads the complete OSCAL schema of the same artifact one time (the manifest check hashes the bytes that the hoist parses), and writes `build/codegen/oscal_complete_schema.json` (git-ignored). In that copy each manifest site is `definitions/<CanonicalName>` with a `$ref` at the site. A merge with different bodies, or a closed-enum value that its base datatype rejects, fails the run.
1. **Hoist.** `python scripts/hoist_names.py` fetches the spec artifact pinned in `scripts/spec_artifact.py` (`0.8.0+oscal.1.2.3`, sha256-checked, cached in `tests/.corpus_cache/`), validates its `names.json` with `scripts/manifest.py` (manifest JSON Schema via `jsonschema-rs`, format and release, artifact-schema sha256, unique names, resolvable pointers), reads the complete OSCAL schema of the same artifact one time (the manifest check hashes the bytes that the hoist parses), and writes `build/codegen/oscal_complete_schema.json` (git-ignored). In that copy each manifest site is `definitions/<CanonicalName>` with a `$ref` at the site. A merge with different bodies, or a closed-enum value that its base datatype rejects, fails the run.
2. **Codegen.** `datamodel-codegen --input ./build/codegen/oscal_complete_schema.json --output ./build/codegen/models.py --base-class oscal_bindings.v1._base.OscalBaseModel --collapse-root-models ... --formatters black isort` (see DEVELOPING "Names reach codegen through the hoisted schema" for the flag choices). Each `object`, `document`, and `enum` class takes its name from its definition key. `--base-class` makes all 185 object models subclass `OscalBaseModel`; the two `RootModel`s (`OscalDocument`, `MappingCollectionMappings`) don't.
3. **Post-process.** `python scripts/postprocess_models.py --models ./build/codegen/models.py --output src/oscal_bindings/v1/models.py`. Phase 1 collapses scalar `RootModel`s to `Annotated` aliases and strips `pattern=` from non-string fields (`AwareDatetime`/`AnyUrl`/numeric/`EmailStr`). Then it adds the 13 union `TypeAlias`es (`CatalogGroup`, plain unions, so `isinstance` works) and writes each field that uses one as `Annotated[<alias>, Field(union_mode='left_to_right')]` (the variant properties are exclusive, so this picks the same variant as smart mode, faster, #43), the `MappingCollectionMappings` root model, and the `OscalDocument` root model. It discriminates `OscalDocument.root` on the root key of the input: each member is `Tag`-ged with its class name, the root key of each member is the manifest `body-field`, and `root_key_discriminator` (`v1/_base.py`) selects the one member to validate (an input with no known root key is one `oscal_document_type` error). It names each scalar alias from its definition key without the `oscal-complete-oscal-<module>:` prefix. Three guards compare the result with the manifest and the artifact schema and report every offender: each manifest name is defined once in the form of its kind, each other name is a scalar alias named for a scalar definition of the schema, no name is defined twice, and no manifest name or scalar alias is bound by a module-level import. The manifest must record the sha256 of the `--schema` bytes. The completed module goes to `--output` through a temp file + `os.replace`; on a failure nothing is written to `--output`, so a failed `generate` leaves the committed `models.py` untouched. `generate` passes no `--schema`: an omitted `--schema` resolves at call time to `artifact_paths(ensure_spec_artifact()).schema`, and an explicit one (a candidate schema, or a small test schema) must still match `schema.sha256` of the manifest. `--models` falls back to its own default, `--output` defaults to `--models`; a missing path is a non-zero exit.

Expand Down
2 changes: 1 addition & 1 deletion DEVELOPING.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ Every type name comes from the naming manifest, `names.json`, of the pinned spec
rule for a manifest type. `hatch run generate` runs three steps:

1. `scripts/hoist_names.py` writes `build/codegen/oscal_complete_schema.json` (git-ignored).
1. It fetches the spec artifact that `scripts/spec_artifact.py` pins (`0.7.0+oscal.1.2.3`, by sha256) through
1. It fetches the spec artifact that `scripts/spec_artifact.py` pins (`0.8.0+oscal.1.2.3`, by sha256) through
`scripts/pinned_archive.py`.
2. `scripts/manifest.py` validates `names.json`: its JSON Schema from the same artifact (with `jsonschema-rs`), the
format version and OSCAL release, the sha256 of the artifact schema, unique names, and pointers that resolve.
Expand Down
4 changes: 2 additions & 2 deletions scripts/spec_artifact.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@
if TYPE_CHECKING:
from collections.abc import Callable

SPEC_ARTIFACT_VERSION = "0.7.0+oscal.1.2.3"
SPEC_ARTIFACT_VERSION = "0.8.0+oscal.1.2.3"
SPEC_ARTIFACT_SHA256 = (
"39f61f5c40200882106d9df6367d5aa5ccc6d604288741a8a70e93e119fa933e"
"3da20cc6127e033d82e47046435caefe2b3a7ad7554908da264e5f4fe7f48dda"
)
# The `+` of the version is written as %2B in the URL path.
_URL_VERSION = SPEC_ARTIFACT_VERSION.replace("+", "%2B")
Expand Down
40 changes: 40 additions & 0 deletions tests/support/expected_failures.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,44 @@
# A round trip keeps six datetime fraction digits and drops the seventh.
"round-trip/seven-fraction-digits": f"{ISSUES}/19",
"yaml-round-trip/string-scalars": f"{ISSUES}/19",
# PRS-014: only failures of the selected nested-union branch.
"parse/closed-enum-out-of-set": f"{ISSUES}/69",
# ERR-012: read-error category for a file path that cannot be read.
"parse-file/directory": f"{ISSUES}/61",
# FAC-007: datetime bounds from assessment_period().
"assessment-period/mixed-offsets": f"{ISSUES}/58",
"assessment-period/equal-instant": f"{ISSUES}/58",
"assessment-period/seventh-fraction-digit": f"{ISSUES}/58",
# ELV-002: one element-type name for each manifest type.
"element-types/all": f"{ISSUES}/59",
# BMB-004, ERR-013: resource identifier from a key, builder-error category.
"build-resource/key": f"{ISSUES}/62",
"build-resource/key-with-title": f"{ISSUES}/62",
"build-resource/non-ascii-key": f"{ISSUES}/62",
"build-resource/key-and-identifier": f"{ISSUES}/62",
"build-resource/empty-key": f"{ISSUES}/62",
# FAC-001 to FAC-003, BMB-001 to BMB-003: the runner lacks these operations.
**dict.fromkeys(
(
"facade/catalog",
"facade/profile",
"facade/component-definition",
"facade/system-security-plan",
"facade/assessment-plan",
"facade/assessment-results",
"facade/plan-of-action-and-milestones",
"facade/mapping-collection",
"facade/missing-file",
"build-hash/default-algorithm",
"build-hash/algorithm-string",
"build-hash/off-list-algorithm",
"build-hash/invalid-algorithm",
"build-rlink/reference-only",
"build-rlink/media-type-and-hashes",
"build-resource/generated-identifier",
"build-resource/omitted-inputs",
"build-resource/invalid-document-id",
),
f"{ISSUES}/63",
),
}
5 changes: 5 additions & 0 deletions tests/test_conformance.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,11 @@ def test_every_expected_failure_names_a_corpus_case() -> None:
assert stale == [], f"expected failures not in the pinned corpus: {stale}"


@pytest.mark.xfail(
strict=True,
reason="read-error and builder-error are not mapped yet: "
"https://github.com/cairn-proofs/oscal-bindings-python/issues/61, /62",
)
def test_every_category_is_mapped() -> None:
"""Each category in the corpus has a branch in ``category``."""
index = json.loads((corpus_root() / "cases.json").read_text("utf-8"))
Expand Down
4 changes: 2 additions & 2 deletions tests/test_spec_artifact_script.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@

def test_script_and_test_support_share_one_pin_and_one_cache() -> None:
assert spec_artifact.SPEC_ARTIFACT is pin.SPEC_ARTIFACT
assert pin.SPEC_ARTIFACT_VERSION == "0.7.0+oscal.1.2.3"
assert pin.SPEC_ARTIFACT_VERSION == "0.8.0+oscal.1.2.3"
assert pin.SPEC_ARTIFACT_SHA256 == (
"39f61f5c40200882106d9df6367d5aa5ccc6d604288741a8a70e93e119fa933e"
"3da20cc6127e033d82e47046435caefe2b3a7ad7554908da264e5f4fe7f48dda"
)
assert pin.CACHE_ROOT.resolve() == corpus.CACHE_ROOT.resolve()
assert REPO_ROOT.resolve() / "tests" / ".corpus_cache" == pin.CACHE_ROOT
Expand Down
Loading