Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/verify-2026-08-hardening-guidance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ jobs:
# A generated file that has been hand-edited is the one failure this
# repository cannot detect by reading. Regenerate and diff.
python tools/source_inputs.py --check
python tools/executable_first_corpus.py
python tools/extract.py
python tools/pattern_examples.py
python tools/copy_examples.py
Expand All @@ -122,8 +123,9 @@ jobs:
python tools/approach_pages.py
python tools/scenario_page.py
python tools/bundle.py
outputs=(data assets/diagrams assets/bundle.js
assessment-first.html catalog-first.html component-first.html scenario.html)
outputs=(data assets/diagrams assets/bundle.js examples/executable-first/oscal
assessment-first.html catalog-first.html component-first.html
executable-first.html scenario.html)
# git diff alone misses newly generated, untracked files.
changes=$(git status --porcelain --untracked-files=all -- "${outputs[@]}")
if [[ -n "$changes" ]]; then
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ field names.
```
## Correction <n>

- approach: catalog-first | assessment-first | component-first
- approach: catalog-first | assessment-first | component-first | executable-first
- received: 2026-09-01
- corrected: what they said was wrong, in their words, unedited
- applied: what changed on the site, and where
Expand All @@ -49,6 +49,21 @@ field names.
your own work is information.
```

## The fourth approach

Executable-first was added from a concept note rather than from a published body of
content, so its proponent is the note's author and the request below goes to
them in the same words. The corpus its extracts come from is the site's own,
written by `tools/executable_first_corpus.py` from the CIS Benchmark and the DISA
STIG for Ubuntu 24.04 in the shape the note proposes, so the three judgements
the request names apply with one difference: the extracts were chosen from
files the site wrote, and the judgement worth the most is whether the shape
those files give the note's mechanism, the parts, the props, the links and the
parameters, is the shape the note intends. Two publishers' guidance was
transcribed into that corpus, and `examples/executable-first/README.md` says what
of it is theirs and what is the generator's; a correction from either of them
about the transcription is recorded here in the same form.

## The request that has not been sent

Recorded here so that when it goes out, what was asked is on the record too. The
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ Assessments.

Three organizations have published OSCAL content addressing the question:
**where does technical hardening guidance live in
OSCAL, and how does an automated check attach to it?** The analysis compares
three approaches against six questions, encodes
the same two rules in all three model structures to isolate modelling
OSCAL, and how does an automated check attach to it?** A concept note proposes
a fourth way. The analysis compares the four approaches against six questions,
encodes the same two rules in all four model structures to isolate modelling
differences, and identifies the join key for each approach.

**The JSON encodings on the question pages were authored for the analysis.**
Expand All @@ -21,6 +21,25 @@ inventories the published content separately, one row per document, with
contents counted from each file. Extracts supporting claims retain declared
pointers into published files and are re-derived on every build.

**The fourth approach's corpus is the site's.** Executable-first comes from a
concept note, *Executable Assessment Methods*, held at
`examples/executable-first/executable-assessment-methods.md` and linked from the
artifacts page, and its proponent has published no OSCAL. So the site wrote a
corpus in the note's shape from two pieces of guidance it already held: the CIS
Benchmark for Ubuntu 24.04 becomes a catalog whose every recommendation carries
an objective and an executable method, with the benchmark's four profiles and
its own SP 800-53 references as a mapping collection; the DISA STIG for the same
system becomes a profile that imports the NIST catalog and adds an objective and
a method to each control a rule serves. `tools/executable_first_corpus.py` writes
the eight files under `examples/executable-first/oscal/`, seven in the
assessment-method shape and one, the same benchmark as a validation component
carrying an `automation-scripts` assembly, in the Rules shape, which is a
proposed assembly and fails validation as it must. `--corpus` recomputes what
they hold, and the status annotation on every card, the inventory page and
`examples/executable-first/README.md` say whose they are. The column on the six
questions page draws its extracts from those files, and the namespace on its
props is the note's own placeholder, carried as the note wrote it.

The analysis uses published hardening guidance from CIS, DISA, CISA, and AWS.
The introduction lists one row per benchmark or guide in a collapsed disclosure,
with links to source material. File icons download the 16 documents recorded
Expand Down Expand Up @@ -51,14 +70,15 @@ implied recommendation. Costs are stated for readers to evaluate.

## Status

Active analysis with eight built pages: `index.html`,
`six-questions.html`, the three approach pages, `scenario.html`, `questions.html`
Active analysis with nine built pages: `index.html`,
`six-questions.html`, the four approach pages, `scenario.html`, `questions.html`
and `oscal-artifacts.html`. Strict verification is not fully passing. Remaining
review and verification items:

- **Proponent review has not happened.** No correction requests have been sent
to the publishers; no corrections have been received, and no publisher has
declined review. The site states the review status.
to the publishers or to the author of the concept note; no corrections have
been received, and no publisher has declined review. The site states the
review status.
- Schema, external-link and browser checks need their dependencies and network
access; offline skips are not strict passes.
- The examples are curated: four published assessment-plan schema defects were
Expand Down Expand Up @@ -285,29 +305,31 @@ Every figure is recomputed from the corpora.

```
python tools/source_inputs.py --check # require prepared inputs; never fetch here
python tools/executable_first_corpus.py # write the executable-first corpus from sources/
python tools/extract.py # rebuild data/snippets and data/provenance.json
python tools/pattern_examples.py # write the two rules in all three shapes
python tools/pattern_examples.py # write the two rules in all four shapes
python tools/copy_examples.py # rebuild data/examples.json from verified inputs
python tools/oscal_artifacts.py # inventory what the three groups have published
python tools/oscal_artifacts.py # inventory what the four approaches have published
python tools/sources_files.py # rebuild data/source-files.json from sources/
python tools/diagrams.py # rebuild every published SVG in assets/diagrams
python tools/approach_pages.py # rebuild the three approach pages
python tools/approach_pages.py # rebuild the four approach pages
python tools/scenario_page.py # rebuild the worked scenario page
python tools/bundle.py # rebuild the offline fallback
python tools/verify.py --all --offline # verify offline; run strict separately
```

The workflow runs generators in the listed dependency order. The bundle runs
last to mirror all preceding output. Five generators accept `--check` to compare
last to mirror all preceding output. Six generators accept `--check` to compare
without writing and exit non-zero for stale committed files:
`pattern_examples.py`, `copy_examples.py`, `oscal_artifacts.py`, `sources_files.py`,
and `bundle.py`.
`executable_first_corpus.py`, `pattern_examples.py`, `copy_examples.py`,
`oscal_artifacts.py`, `sources_files.py`, and `bundle.py`.
`approach_pages.py` records generated-page hashes and refuses to overwrite
hand-edited pages. Editing a generated page instead of source data stops the
build rather than losing the edit.

`tools/manifest.yaml` declares 32 snippets, all JSON. Each entry names a published
OSCAL JSON source file, an RFC 6901 pointer, the illustrated question, and any
`tools/manifest.yaml` declares 43 snippets, all JSON. Each entry names an
OSCAL JSON source file, resolved through the source lock or, prefixed `site:`,
held in this repository, an RFC 6901 pointer, the illustrated question, and any
trimming applied. Trimming is declared and marked in rendered
output; there is no silent truncation.
`data/provenance.json` records the SHA-256 of both the source file and the
Expand All @@ -334,33 +356,45 @@ longer machine-checked by that phase.

```
data/
snippets/ 32 JSON files, one per extract, never edited by hand
snippets/ 43 JSON files, one per extract, never edited by hand
schema-evidence/ 8 verbatim OSCAL 1.2.1 schema fragments with their constraints
six-questions.json the six questions and the answer matrix, the central claim set
pattern-examples.json the two rules, written in all three shapes. Ours
oscal-artifacts.json what each of the three groups has published, counted
pattern-examples.json the two rules, written in all four shapes. Ours
oscal-artifacts.json what each approach has published, counted, and a note
for the one whose corpus the site generated
examples.json committed local fingerprints and pinned public links
criteria.json the 15 evaluation criteria, maintained as analysis content
sources.json the guidance read as input, one row per benchmark
source-files.json every file under sources/, generated, with size and type
views.json the three views of what a rule is
views.json the three views of what a rule is, and which approaches
hold each; two hold the first
check-carrier.json the check axis: what each approach's OSCAL carries about
the check, the same rule from two corpora, and figures
the harness recomputes
decisions.json the construct decision, Rules against assessment-method,
the placement decision, and the grid that crosses them
glossary.json the vocabulary, including the terms the group has not
settled. Rendered as term cards in place; there is no
glossary page
questions.json editorial open questions and their reasoning, in four sections
corpus-stats.json every figure cited, each with its derivation
provenance.json generated
examples/
executable-first/ the concept note, and under oscal/ the seven files
tools/executable_first_corpus.py writes in its shape
tools/
source-lock.json tracked local inventories and pinned public archive metadata
source_inputs.py explicit public fetch and offline input integrity checks
manifest.yaml the declarative extract manifest
manifest.yaml the declarative extract manifest; a source prefixed site:
is read from this repository rather than the source lock
executable_first_corpus.py builds examples/executable-first/oscal from sources/
extract.py builds data/snippets and data/provenance.json
pattern_examples.py builds data/pattern-examples.json, the site's own encodings
copy_examples.py builds data/examples.json from verified source inputs
oscal_artifacts.py builds data/oscal-artifacts.json from the three corpora
oscal_artifacts.py builds data/oscal-artifacts.json from the four corpora
sources_files.py builds data/source-files.json from sources/
diagrams.py builds the published SVGs in assets/diagrams from data/
approach_pages.py builds the three approach pages from one template
approach_pages.py builds the four approach pages from one template
scenario_page.py builds scenario.html, the one page that carries figures
bundle.py builds assets/bundle.js, the offline fallback
svgrender.py rasterises the diagrams for the grayscale contact sheet
Expand All @@ -369,17 +403,17 @@ tools/
axe_run.mjs serves the site and runs axe-core over every page
```

Nine tools run in the regeneration pipeline. Do not hand-edit generated files:
`data/snippets/*.json` and `data/provenance.json`,
`data/pattern-examples.json`, `data/examples.json`, `data/oscal-artifacts.json`,
`data/source-files.json`, `assets/diagrams/*.svg`, the
three approach pages, `scenario.html`, and `assets/bundle.js`. The workflow regenerates the files
and fails on differences from committed versions.
Ten tools run in the regeneration pipeline. Do not hand-edit generated files:
`examples/executable-first/oscal/*.json`, `data/snippets/*.json` and
`data/provenance.json`, `data/pattern-examples.json`, `data/examples.json`,
`data/oscal-artifacts.json`, `data/source-files.json`, `assets/diagrams/*.svg`,
the four approach pages, `scenario.html`, and `assets/bundle.js`. The workflow
regenerates the files and fails on differences from committed versions.
The local files under `examples/` are authoritative inputs, not disposable
generator output; their committed fingerprints must already pass source checks
before regeneration.

`tools/diagrams.py` builds 19 diagrams and writes 9. The remaining 10 appeared
`tools/diagrams.py` builds 18 diagrams and writes 10. The remaining 8 appeared
only in a removed component gallery, including the base layer map and three
stakeholder variants. The builders remain together in the drawing code.
`PUBLISHED`, at the foot of the file, lists diagrams written to
Expand All @@ -401,11 +435,11 @@ seven rules. The rules also appeared on the removed methodology page. Key rules:
prose per page, exempting file paths as provenance.
Guidance publishers and source paths remain identified for provenance;
the artifact inventory describes published content, not proponent positions.
- **Equal budget is enforced by construction.** One generator writes the three
- **Equal budget is enforced by construction.** One generator writes the four
approach pages and rejects word-count differences above ten per cent.
- **Non-verbal encoding counts as editorializing.** Badges, hollow cells, and
hatch fills convey arguments outside word counts. Every annotation type
applies to all three approaches or none.
applies to all four approaches or none.
- **Consequences, not verdicts.** State the cost; let the reader price it.
- **Every figure carries its denominator**, and every figure is recomputed from
the corpora on every build.
Expand Down Expand Up @@ -442,15 +476,29 @@ the manifest; run `python tools/verify.py --example` to check encodings.
Submit changes to the 15 evaluation criteria or open questions for analysis
review, and update their structural checks when the agreed structure changes.

## Adding a fourth approach
## Adding an approach

Adding an approach requires data changes and a page, not a site rewrite.
Adding an approach requires data changes and a page, not a site rewrite. The
fourth approach was added this way, from a concept note rather than a corpus,
and the places that had to learn a fourth are recorded in `BUILD-LOG.md` under
that session: one entry in every per-approach data file, a fourth column in the
encoding generator, a fourth hue at the same lightness, a fourth marker shape,
and the checks that counted to three.

1. **Define the source inputs.** Extend [tools/source-lock.json](tools/source-lock.json)
and [tools/source_inputs.py](tools/source_inputs.py) together, with tests,
to authorize the new local inventory or pinned public archive. Record
fingerprints and publisher terms; never rely on a personal folder or upload
a private corpus. Prepare public inputs explicitly and check them offline.
An approach with no published corpus, one that exists as a proposal, can
have one written in its shape from the guidance under `sources/`, as
executable-first has: the generator goes under `tools/`, its output under
`examples/<approach>/oscal/`, the approach's key into `GENERATED` in
`tools/verify.py` and `SITE_CORPORA` in `tools/oscal_artifacts.py`, its
status annotation says the corpus is generated, and manifest sources
prefixed `site:` read from it. The one document it does have goes under
`examples/<approach>/` and is named in `UNPUBLISHED` in
`tools/oscal_artifacts.py` so the inventory page links it.
2. **Declare the extracts.** Add manifest entries with a source file, an RFC 6901
pointer, and the illustrated question. Run `python tools/extract.py`.
3. **Add the approach to `data/six-questions.json`.** Add an `approaches` entry
Expand All @@ -459,7 +507,7 @@ Adding an approach requires data changes and a page, not a site rewrite.
`matrix` cell for each of the seven question rows, with a state, tooltip note,
and supporting extracts. Classify unanswered cells using one of the three
unanswered states.
4. **Encode the two rules in the approach's structure.** Add a fourth column to
4. **Encode the two rules in the approach's structure.** Add a column to
`tools/pattern_examples.py`: the same two rules, in the same order, for every
answered question row. Each block names the containing OSCAL construct and
uses the approach's namespace on properties requiring a namespace.
Expand All @@ -482,6 +530,12 @@ corpus directory and option letter. The inventory derives the remaining content
from files rather than hand-authored descriptions. Keep the OSCAL corpus outside
`sources/`, which lists input hardening guidance, not derived OSCAL content.

The checks that count approaches read the count from `data/six-questions.json`,
with four exceptions that name them: the option-letter order in
`tools/verify.py`, `tools/pagecheck.js` and `assets/site.js`, the approach list
in `tools/diagrams.py` with its marker shape, and the approach colour tokens in
`assets/site.css`, which `--a11y` holds to one lightness.

## License

CC BY 4.0 applies to analysis-authored text, diagrams, and encodings. See
Expand Down
Loading