Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,14 @@ jobs:
# and nothing about the code changes at merge time. Post-merge validation is handled by
# the support-data job below, which re-runs the unit tests *after* refreshing the support
# database -- the one thing that is new on main.
#
# They are also SKIPPED on a pull_request whose head is this repo's own develop branch
# (i.e. the develop -> main PR): every push to develop already runs them on the exact
# same commit, and those results show on the PR. PRs from feature branches and forks
# (which have no push trigger here) still run them.
lint:
name: Ruff lint (Python ${{ matrix.python-version }})
if: ${{ github.event_name != 'push' || github.ref != 'refs/heads/main' }}
if: ${{ (github.event_name == 'push' && github.ref != 'refs/heads/main') || (github.event_name == 'pull_request' && !(github.head_ref == 'develop' && github.event.pull_request.head.repo.full_name == github.repository)) }}
runs-on: ${{ matrix.os || 'ubuntu-latest' }}
strategy:
fail-fast: false
Expand Down Expand Up @@ -49,7 +54,7 @@ jobs:

test:
name: Tests (Python ${{ matrix.python-version }})
if: ${{ github.event_name != 'push' || github.ref != 'refs/heads/main' }}
if: ${{ (github.event_name == 'push' && github.ref != 'refs/heads/main') || (github.event_name == 'pull_request' && !(github.head_ref == 'develop' && github.event.pull_request.head.repo.full_name == github.repository)) }}
runs-on: ${{ matrix.os || 'ubuntu-latest' }}
strategy:
fail-fast: false
Expand Down Expand Up @@ -83,7 +88,7 @@ jobs:
# depend on the support database, so there is nothing new to re-check at merge time.
sbom-smoke:
name: SBOM generation (smoke test)
if: ${{ github.event_name != 'push' || github.ref != 'refs/heads/main' }}
if: ${{ (github.event_name == 'push' && github.ref != 'refs/heads/main') || (github.event_name == 'pull_request' && !(github.head_ref == 'develop' && github.event.pull_request.head.repo.full_name == github.repository)) }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -148,7 +153,9 @@ jobs:
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add oscal/data/oscal_support.zip
git commit -m "chore: refresh OSCAL support database [skip ci]"
# No [skip ci]: GITHUB_TOKEN pushes never trigger workflows, and [skip ci] here would
# block the tag-triggered Release workflow when a release tag lands on this commit.
git commit -m "chore: refresh OSCAL support database"
git pull --rebase origin main
git push origin HEAD:main
fi
4 changes: 3 additions & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ jobs:
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add docs/api-llm.html docs/api.html
git commit -m "chore: refresh API docs [skip ci]"
# No [skip ci]: GITHUB_TOKEN pushes never trigger workflows, and [skip ci] here would
# block the tag-triggered Release workflow when a release tag lands on this commit.
git commit -m "chore: refresh API docs"
git pull --rebase origin main
git push origin HEAD:main
fi
Expand Down
25 changes: 22 additions & 3 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,33 @@ name: Release
# Release flow for a maintainer: bump oscal/_version.py, merge to main, let CI refresh and
# commit the support database, then: git tag vX.Y.Z && git push origin vX.Y.Z
# (The GitHub Release is created by this workflow; do not create it by hand.)
#
# Fallback: if the tag push does not auto-trigger this workflow, run it manually from the
# Actions tab (or `gh workflow run publish.yml -f tag=vX.Y.Z`) against an EXISTING tag. The
# build runs against that tag's commit, identically to the push path.
on:
push:
tags: ["v*"]
workflow_dispatch:
inputs:
tag:
description: "Existing version tag to build the Release for (e.g. v4.2.3). The tag must already exist on the remote."
required: true
type: string

# The tag this run operates on: the pushed tag (push event) or the dispatch input.
# On a push event `inputs.tag` is null, so this resolves to github.ref_name (the tag).
env:
RELEASE_TAG: ${{ inputs.tag || github.ref_name }}

jobs:
build:
name: Build distribution
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.tag || github.ref_name }}

- name: Set up Python
uses: actions/setup-python@v5
Expand All @@ -31,10 +48,10 @@ jobs:
- name: Verify tag matches package version
run: |
PKG=$(python -c "import re,pathlib; print(re.search(r'__version__\s*=\s*\"([^\"]+)\"', pathlib.Path('oscal/_version.py').read_text()).group(1))")
TAG="${GITHUB_REF_NAME#v}"
TAG="${RELEASE_TAG#v}"
echo "release tag = $TAG package version = $PKG"
if [ "$TAG" != "$PKG" ]; then
echo "::error::Release tag ($GITHUB_REF_NAME) does not match oscal/_version.py ($PKG). Bump oscal/_version.py or fix the tag."
echo "::error::Release tag ($RELEASE_TAG) does not match oscal/_version.py ($PKG). Bump oscal/_version.py or fix the tag."
exit 1
fi

Expand All @@ -60,6 +77,8 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.tag || github.ref_name }}

- name: Set up Python
uses: actions/setup-python@v5
Expand Down Expand Up @@ -102,7 +121,7 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
run: |
TAG="${GITHUB_REF_NAME}"
TAG="${RELEASE_TAG}"
# Draft (mutable) -> upload assets -> publish (seals immutably WITH the assets).
# The wheel + sdist are attached so pypi.yml can publish the exact same artifacts.
gh release create "$TAG" --repo "$GITHUB_REPOSITORY" --draft \
Expand Down
129 changes: 129 additions & 0 deletions docs/OSCAL_VERSION_MONITORING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# OSCAL Version Monitoring and Support-Data Delivery

**Status:** idea / design notes. Not started — set aside to ship the current release.
**Captured:** 2026-11-13.

**Goal:** the published `oscal` package should include the latest NIST OSCAL release in its
support database as soon as possible after NIST publishes it, while staying transparent to
users (`pip install oscal` is all they do) and working fully offline once installed.

## Background: how the support database works today

- `oscal/data/oscal_support.zip` ships **inside** the `oscal` wheel.
- `python -m oscal.update_support` builds it from the NIST OSCAL GitHub releases:
- `--all` clears and re-fetches every release (about 15 today and growing; dozens of files each).
- `--new` (the default) fetches only releases the database doesn't have yet — usually a
fast "nothing new" check.
- CI (`ci.yml`, `support-data` job) currently runs `--all` on every merge to main, re-runs the
unit tests, and commits the zip to main if it changed.
- Consequence: new OSCAL data only reaches users when a new **library** release is cut.

## Relationship to the SBOM

- Each release's SBOM is generated from that release's built wheel and attached to that
(immutable) GitHub Release. Later database changes on main never invalidate an existing SBOM;
they only affect what the *next* release contains.
- Ordering rule: any database update must happen **before** the build, and the SBOM is
generated from the built wheel: `DB update → test → build → SBOM → release`.
- The current SBOM (`cyclonedx-py environment`) lists installed packages, not data files, so it
does not record which OSCAL versions the database contains. Option 4 below fixes that.

## Options considered

### 1. Scheduled database update committed to main

A scheduled workflow checks NIST for new releases, runs `update_support --new`, runs the unit
tests, and commits the zip to main if they pass. The next library release picks it up.

- Cheap and safe; data reaches main quickly.
- Data still reaches **users** only when a library release is cut.

### 2. Scheduled update + automatic library release + automatic PyPI publish — rejected

Extends option 1 to bump the patch version and run the release workflow automatically.

**Rejected (for now)** because the release would build from the current state of main, not the
last release. Anything merged to main but not yet meant for release would ship with the data
update. It also creates version collisions with develop (the bot and a human both claiming the
next patch number) and requires removing the human PyPI approval gate.

If ever revisited, required guardrails:
- main must always be releasable, *or* the job builds from the last release tag plus new data;
- the bot owns the patch number (manual releases bump at least the minor version);
- the job refuses to release if anything other than the zip and `_version.py` changed;
- auto-release only patch-level OSCAL releases (e.g. 1.1.3 → 1.1.4); minor/major OSCAL releases
(e.g. 1.2.0 added the Control Mapping model) may need code support, so open an issue/PR
for human review instead;
- open a GitHub issue when the update or tests fail;
- share the `main-asset-commit` concurrency group.

### 3. Database regeneration once per PR

To have the regenerated database reviewed in the PR without rebuilding it on every PR update:

- run the regen job on `pull_request: types: [ready_for_review]` (iterate as a draft PR; mark
ready once), committing the zip to the PR branch;
- use `--new` normally; use `--all` only when the PR changes `update_support.py` or
`oscal_support.py`;
- the job must run the unit tests itself (commits pushed with `GITHUB_TOKEN` don't trigger CI);
- pull the PR branch afterwards to avoid binary conflicts on the zip.

### 4. Separate data package — recommended direction

Publish the database as its own PyPI project (e.g. `oscal-support-data`) that `oscal` depends on
(`dependencies = ["oscal-support-data>=2026.11"]`). This is how `tzdata` and `certifi` work.

- **Transparent:** `pip install oscal` installs it automatically; works offline afterwards.
- **Independent releases:** data can be released on its own schedule, including fully
automatically, without ever building or releasing library code from main. This removes the
blocker in option 2.
- **User-updatable:** `pip install -U oscal-support-data` gets new data without a library update.
- **Versioning:** date-based (e.g. `2026.11.13`); never collides with library versions.
- **SBOM:** appears as a dependency with its own version, so each library SBOM records which data
release is in use.

Costs / contract:
- a format-version check at load time (clear error on mismatch) and/or a bounded dependency
range (e.g. `>=2026.11,<2027`);
- the library must ignore OSCAL versions in the data it does not yet support, so data releases
can include new OSCAL versions before code support lands;
- bump the library's lower bound when it starts relying on newer data;
- a second PyPI project, release workflow, and version stream.

Options also ruled out: download-on-first-use (breaks offline), install-time hooks (wheels can't
run code at install), an optional extra like `pip install oscal[data]` (a plain install wouldn't
include the data). A later add-on could let users refresh data into a local cache
(`update_support --new`) that the library prefers when present.

## Two PyPI projects from this one repository

Supported by both PyPI and GitHub (common in large projects).

```
class/
pyproject.toml # oscal (library)
oscal/
packages/oscal-support-data/
pyproject.toml # oscal-support-data
oscal_support_data/
__init__.py # path helper only
oscal_support.zip
```

- The library build must exclude `packages/`; the data package must **not** depend on `oscal`.
- The generator (`update_support`) stays in the repo as tooling, not in the data package.
- Separate workflows: `publish.yml` (library) and `publish-data.yml` (data; scheduled + manual).
- Distinct tag prefixes: `v4.2.5` (library) vs `data-v2026.11.13` (data).
- PyPI trusted publishers are per project (repo + workflow file + environment). Create the new
project with a "pending publisher"; its first upload creates it.
- GitHub Releases share one list; create data releases with `--latest=false`.
- CI and local dev install both editable (`pip install -e . -e packages/oscal-support-data`) so
library tests use the repo's data, not PyPI's. Run library tests on data changes too.

## Suggested sequence when picked up

1. Settle the data format contract (format version + how unsupported OSCAL versions are ignored).
2. Split the data into `packages/oscal-support-data` and make `oscal` depend on it.
3. Add `publish-data.yml` (manual first), register the PyPI pending publisher, release once.
4. Add the scheduled check (`update_support --new` → tests → data release), with failure issues.
5. Remove the `support-data` regen from the library's CI.
14 changes: 13 additions & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

The sequencing of releases may change in response to client needs and other external factors.

## CURRENT: Version 4.2.2
## CURRENT: Version 4.2.4

- Profile import-selection authoring API — safer, granular control over an import's `include-controls` / `exclude-controls` and the `with-child-controls` flag:
- `set_import_with_children()` toggles `with-child-controls` on the existing include/exclude selection independently, without rebuilding it.
Expand Down Expand Up @@ -89,6 +89,18 @@ below (POA&M / AP / AR).

## Future:

### OSCAL Version Monitoring and Support-Data Delivery

- [ ] Get new NIST OSCAL releases into the published support database as soon as possible,
transparently to users (`pip install oscal` only) and fully offline once installed.
- Recommended direction: move the support database into a separate PyPI package
(e.g. `oscal-support-data`, like `tzdata`/`certifi`) published from this repo, which
`oscal` depends on, with a scheduled check that releases new data automatically —
without releasing library code from main.
- Rejected for now: automatic *library* releases on new OSCAL data (would ship whatever
is on main).
- Details, options, and suggested sequence: [OSCAL_VERSION_MONITORING.md](OSCAL_VERSION_MONITORING.md).

### SBOM Packaging — PEP 770 Alignment

- [ ] Embed a CycloneDX SBOM inside the published wheel/sdist per **PEP 770**
Expand Down
4 changes: 3 additions & 1 deletion docs/VERSION_HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,11 @@
- All import-selection setters remain staged, metaschema-validated, and atomic (a rejected edit rolls back and returns `None`).
- CI: pinned the Python 3.9 test/lint matrix leg to `ubuntu-22.04` (`actions/setup-python` has no 3.9 build for Ubuntu ≥24.04, now the `ubuntu-latest` image). Python 3.9 support is unchanged this release; see `docs/5.0.0_PLANNING.md` for the planned drop of 3.9/3.10 and addition of 3.14.

## Version 4.2.2
## Version 4.2.4

- additional methods for managing profile import control inclusion and exclusion directives
- original method is still available, but only provided coarse-grain control
- new methods allow targeted adjustments
- new release script


2 changes: 1 addition & 1 deletion oscal/_version.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@
so the build backend can read the literal without importing ``oscal`` (and its
dependencies).
"""
__version__ = "4.2.2"
__version__ = "4.2.4"
Loading
Loading