From 2fe2bfd65bc68f5e6eec689e706be5b42cae555d Mon Sep 17 00:00:00 2001 From: Pirooz Javan Date: Wed, 9 Sep 2026 11:44:30 -0400 Subject: [PATCH 1/7] Refactor analysis documentation and improve clarity - Updated the question in the analysis JSON for better specificity regarding OSCAL models and automated checks. - Revised meta descriptions and content in index.html files for analyses, patterns, and recommendations to enhance clarity and consistency. - Improved the wording in the lede paragraphs across various HTML files to better convey the purpose of each section. - Added a new JavaScript file for regression checks related to analysis-scoped progress notices, ensuring proper functionality and coverage. - Enhanced error messages and handling in the site.js file for better user experience during loading failures. --- README.md | 54 +- .../README.md | 439 +++++---- .../assessment-first.html | 97 +- .../assets/bundle.js | 8 +- .../diagrams/710-scenario-assessment.svg | 18 +- .../assets/diagrams/710-scenario-catalog.svg | 20 +- .../diagrams/710-scenario-component.svg | 20 +- .../assets/shell.html | 9 +- .../assets/site.css | 20 + .../assets/site.js | 36 + .../catalog-first.html | 91 +- .../component-first.html | 101 ++- .../data/generated-pages.json | 6 +- .../data/glossary.json | 56 +- .../data/joins.json | 62 +- .../data/scenario.json | 66 +- .../data/six-questions.json | 118 +-- .../data/stakeholders.json | 52 +- .../data/tradeoffs.json | 96 +- .../data/views.json | 22 +- .../index.html | 93 +- .../oscal-artifacts.html | 33 +- .../questions.html | 15 +- .../scenario.html | 45 +- .../six-questions.html | 40 +- .../tools/approach_pages.py | 60 +- .../tools/bannercheck.js | 205 +++++ .../tools/bundle.py | 12 +- .../tools/pagecheck.js | 19 +- .../tools/pattern_examples.py | 2 +- .../tools/scenario_page.py | 36 +- .../tools/verify.py | 846 ++++++++---------- docs/analysis/analyses.json | 2 +- docs/analysis/index.html | 22 +- docs/assets/site.js | 2 +- docs/index.html | 34 +- docs/patterns/index.html | 33 +- docs/recommendations/index.html | 30 +- 38 files changed, 1554 insertions(+), 1366 deletions(-) create mode 100644 docs/analysis/2026-08-automating-technical-hardening-guidance/tools/bannercheck.js diff --git a/README.md b/README.md index 453919b..59aa99b 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,28 @@ # OSCAL Foundation — Pattern Library -A curated collection of high-quality, realistic [OSCAL](https://pages.nist.gov/OSCAL/) example artifacts published by the **OSCAL Foundation** to serve as patterns and practices for the community. +A collection of [OSCAL](https://pages.nist.gov/OSCAL/) example artifacts published by the **OSCAL Foundation** as community patterns and practices. ## About OSCAL Foundation -The Open Security Controls Assessment Language (OSCAL) is a machine-readable language that simplifies and standardizes information system security assessments through the exchange of information via automation. +The Open Security Controls Assessment Language (OSCAL) is a machine-readable language for simplifying and standardizing information system security assessments through automated information exchange. -Originally developed by the National Institute of Standards and Technology (NIST) in collaboration with FedRAMP and industry, OSCAL aims to improve the efficiency, timeliness, accuracy, and consistency of system security assessments. +Developed by the National Institute of Standards and Technology (NIST) with FedRAMP and industry, OSCAL aims to improve the efficiency, timeliness, accuracy, and consistency of system security assessments. -The **OSCAL Foundation** is dedicated to furthering the development and adoption of the OSCAL standards. The Foundation is a nonprofit organization seeking 501(c)(3) tax-exempt status recognition. +The **OSCAL Foundation** supports the development and adoption of OSCAL standards. The Foundation is a nonprofit organization seeking recognition of 501(c)(3) tax-exempt status. ## Purpose -There are few high-quality, representative examples of what an actual compliance package in OSCAL looks like, and few places where the arguments about how to build one are written down and kept. This repository holds both. +The repository collects OSCAL compliance-package examples and design analyses. -It is published as a website: **** +The repository is published as a website: **** -Three areas, each with its own lifecycle. +Three areas have separate lifecycles. -| Area | What it holds | +| Area | Contents | |---|---| -| [**Patterns**](docs/patterns/summit/) | Model office examples covering the seven OSCAL models, published as files a tool can read | -| [**Analyses**](docs/analysis/) | Efforts that debate a question about OSCAL, one area per effort, retained after the effort ends | -| [**Recommendations**](docs/recommendations/) | What the Foundation recommends, each one citing the analysis it came from | +| [**Patterns**](docs/patterns/summit/) | Model office examples covering the seven OSCAL models, published as machine-readable files | +| [**Analyses**](docs/analysis/) | Comparisons of OSCAL approaches, retained in separate areas after conclusion | +| [**Recommendations**](docs/recommendations/) | Foundation recommendations citing the supporting analyses | ## Examples @@ -32,17 +32,17 @@ Three areas, each with its own lifecycle. ## Analyses -Each effort gets its own dated area and keeps it. An area is never renamed, never moved and never deleted, and a concluded one is not edited into agreement with a later view: when an effort is superseded the new one gets its own area and the old one is marked, so the record shows the change rather than replacing it. +Each analysis has a dated area, never renamed, moved, or deleted. Concluded analyses are not rewritten to match later views. A replacement analysis receives a new area; the earlier analysis is marked as superseded to preserve the record. | Opened | Analysis | Status | |---|---|---| | 2026-08 | [Automating Technical Hardening Guidance with OSCAL](docs/analysis/2026-08-automating-technical-hardening-guidance/) | active | -An analysis carries an `analysis.json` beside its `index.html`, and an entry in [docs/analysis/analyses.json](docs/analysis/analyses.json) that the index page renders from. Adding an effort means adding an area and appending to that file; no page is edited. +Each analysis has an `analysis.json` beside `index.html` and an entry in [docs/analysis/analyses.json](docs/analysis/analyses.json), the registry rendered by the analysis index. Adding an analysis requires a new area and registry entry, not an index-page edit. ## OSCAL Models Covered -Each example in this library aims to include artifacts for all seven OSCAL models: +Each library example aims to include artifacts for all seven OSCAL models: 1. **Catalog** — Security control definitions 2. **Profile** — Baseline selection and tailoring @@ -54,7 +54,7 @@ Each example in this library aims to include artifacts for all seven OSCAL model ## Repository Structure -`docs/` is the site, verbatim. There is no build step and nothing is generated at deploy time: what is in the folder is what is published. +`docs/` is published unchanged, with no build or generation step at deployment. ``` Pattern-Library/ @@ -85,34 +85,40 @@ Pattern-Library/ └── recommendations.json ``` -The artifacts sit inside `patterns/` rather than at the repository root so that the pages linking them resolve the same way locally and published. Nothing has to be copied or rewritten between the two. +Artifacts reside in `patterns/` rather than the repository root so links resolve identically in local and published pages. No copying or rewriting is required. ## Running the site -Press **F5**. The `Pattern Library` configuration starts a static server on 4173 and opens the landing page with the debugger attached, so breakpoints in `docs/assets/site.js` bind to the served file. Edge is the default because Chrome is not installed everywhere; the Chrome configuration is there if you have it. +Press **F5**. The `Pattern Library` configuration starts a static server on port 4173 and opens the landing page with the debugger attached. Breakpoints in `docs/assets/site.js` bind to the served file. Edge is the default; a Chrome configuration is available for installations with Chrome. -Nothing is installed and nothing in this repository serves the site — the task is a stock `python3 -m http.server` over `docs/`. Any equivalent works: +The task serves `docs/` with `python3 -m http.server`, without installing dependencies or using repository server code. An equivalent static server also works: ``` python3 -m http.server --directory docs 4173 npx serve docs ``` -If you install the recommended [Live Preview](https://marketplace.visualstudio.com/items?itemName=ms-vscode.live-server) extension you can skip all of that: click the preview button on any page under `docs/`, or run **Live Preview: Show Debug Preview** for breakpoints. `.vscode/settings.json` already points its server root at `docs/`. +Alternatively, install the recommended [Live Preview](https://marketplace.visualstudio.com/items?itemName=ms-vscode.live-server) extension and click the preview button on a page under `docs/`, or run **Live Preview: Show Debug Preview** for breakpoints. `.vscode/settings.json` sets the server root to `docs/`. -**Run Task** carries the rest: +**Run Task** provides these tasks: -| Task | What it does | +| Task | Action | |---|---| -| `site: serve` / `site: stop` | The server F5 uses, on its own | +| `site: serve` / `site: stop` | Start or stop the F5 server without the debugger | | `analysis 2026-08: verify` | Recompute every figure and extract from its published source | | `analysis 2026-08: regenerate` | Run every generator; a file that differs afterwards was hand-edited | -Opening a page from the filesystem does not work: every index renders its list from a JSON registry, and a browser blocks `fetch` on a `file://` origin. The pages say so when it happens rather than appearing empty. +The library index pages require a server to load lists from JSON registries: browsers block `fetch` on a `file://` origin. Affected pages display a message rather than an empty list. + +The “Work in progress” bar appears only within active analyses without a recorded decision. Each analysis's `analysis.json` controls visibility: `status` must be `active`, with no `concluded`, `recommendation`, `decision`, `decided`, or `supersededBy` value. A missing decision does not imply that an analysis will issue a recommendation. Library landing pages and indexes carry no banner. + +Closing the bar hides the notice across pages of that analysis for the browser tab's session, without affecting other analyses. With JavaScript disabled or status unavailable, the notice stays hidden. Folder-based copies read analysis status from the generated bundle. + +Banner regression checks: `node docs/analysis/2026-08-automating-technical-hardening-guidance/tools/bannercheck.js`. ## Contributing -Contributions of high-quality OSCAL examples are welcome. Please ensure examples are realistic, well-structured, and follow OSCAL best practices. +Contribute OSCAL examples grounded in implementation practice, organized by model, and following OSCAL best practices. ## License diff --git a/docs/analysis/2026-08-automating-technical-hardening-guidance/README.md b/docs/analysis/2026-08-automating-technical-hardening-guidance/README.md index 57b7013..12226c5 100644 --- a/docs/analysis/2026-08-automating-technical-hardening-guidance/README.md +++ b/docs/analysis/2026-08-automating-technical-hardening-guidance/README.md @@ -3,89 +3,79 @@ A comparison site for the OSCAL Foundation Technical Focus Group on Automated Assessments. -Three organizations have published working OSCAL content that answers the same -question three different ways: **where does technical hardening guidance live in -OSCAL, and how does an automated check attach to it?** This site puts the three -side by side against the same six questions, writes the same two rules out in all -three shapes so that the only thing differing between the columns is the -modelling, and names the join key that makes each one work. - -**The JSON on the question pages is ours.** The three groups ship content for -different products, so quoting what each one published would set a certificate -control beside an object-store rule beside a Linux STIG, which compares subject -matter rather than modelling. So two rules were chosen once, both published -Ubuntu 24.04 LTS requirements keeping their real identifiers, and written out in -each approach's own shape by `tools/pattern_examples.py`. What is authored is the -encoding and nothing else: every property carries the namespace of the approach -it illustrates and none of them is ours. What the three groups have actually -published is inventoried separately, on `oscal-artifacts.html`, one row per -document with its contents counted from the file rather than described. Extracts -at declared pointers into the published files are still carried where a claim -rests on one, and they are still re-derived on every build. - -Published hardening guidance from four publishers was read as input: CIS, DISA, -CISA and AWS. The introduction tabulates it one row per benchmark or guide, in a -disclosure that is closed until it is wanted, and every row carries what stands -behind it. A file icon downloads one of the 16 documents recorded under -`sources/`, 29.0 MB in all, and names the file, its type and its size on -hover; a globe opens -the publisher's own page for guidance that is not a document to hold. Every one -of them opens in a new tab. **Only inputs are listed.** The OSCAL written from -this guidance is published separately, so `sources/oscal/` is in the repository -but carries no row: `tools/sources_files.py` excludes it from the inventory, as -it excludes `disa/U_Readme_SRG_and_STIG.pdf`, which is about the STIG packages -as a set rather than about any one guide. `verify.py --sources` recomputes every -count from the corpora, -confirms every link resolves to a file, confirms no file sits in `sources/` unlinked, -and confirms nothing from the excluded tree is linked. Terms differ by publisher and -each file carries its own. - -**The CIS material is here on a recorded basis.** `sources/cis/` holds the two -CIS Benchmarks in both their JSON and PDF form, 19.7 MB of the 29.0 MB. CIS -licenses its Benchmarks under its own Agreed Terms of Use, which on their face -say a Benchmark may not be redistributed or posted on any website. The material -is here because CIS is a stakeholder receiving this brief and confirmed that -copying and linking infringes nothing. That call is recorded in `BUILD-LOG.md`, -and each of the four files carries the basis rather than a general grant in its -own terms line: *"Reproduced here with the publisher's participation in this +Three organizations have published OSCAL content addressing the question: +**where does technical hardening guidance live in +OSCAL, and how does an automated check attach to it?** The analysis compares +three approaches against six questions, encodes +the same two rules in all three model structures to isolate modelling +differences, and identifies the join key for each approach. + +**The JSON encodings on the question pages were authored for the analysis.** +The published content covers different products: a certificate control, an +object-store rule, and a Linux STIG would compare subject matter rather than +modelling. Instead, `tools/pattern_examples.py` encodes two published Ubuntu +24.04 LTS requirements in each approach's structure, retaining the published +identifiers. Only the encoding is authored; property namespaces come from the +illustrated approaches, not the analysis authors. `oscal-artifacts.html` +inventories the published content separately, one row per document, with +contents counted from each file. Extracts supporting claims retain declared +pointers into published files and are re-derived on every build. + +The analysis uses published hardening guidance from CIS, DISA, CISA, and AWS. +The introduction lists one row per benchmark or guide in a collapsed disclosure, +with links to source material. File icons download the 16 documents recorded +under `sources/`, totalling 29.0 MB; hover text gives the filename, type, and +size. Globe icons link to publisher pages for guidance without a downloadable +document. All source links open in new tabs. **Only inputs are listed.** OSCAL +content derived from the guidance is published separately. +`tools/sources_files.py` excludes `sources/oscal/` from the inventory and also +excludes `disa/U_Readme_SRG_and_STIG.pdf`, which describes STIG packages rather +than an individual guide. `verify.py --sources` recomputes counts from the +corpora and checks link resolution, coverage of included files in `sources/`, +and absence of links into the excluded tree. Each file lists publisher terms. + +**The basis for including CIS material is recorded.** `sources/cis/` holds two +CIS Benchmarks in JSON and PDF, accounting for 19.7 MB of the 29.0 MB. On their +face, CIS's Agreed Terms of Use prohibit redistributing a Benchmark or posting a +Benchmark on a website. CIS is a stakeholder receiving the brief and confirmed that copying +and linking infringes nothing. `BUILD-LOG.md` records the confirmation. Each of +the four files states the basis for inclusion, not a general grant: +*"Reproduced here with the publisher's participation in this review. The Benchmarks carry CIS's own Agreed Terms of Use; consult the publisher before reusing them elsewhere."* **A fork or a mirror made outside that conversation does not inherit the basis.** The DISA material is a work of the United States Government, 13 files and 9.3 MB, 12 of them with a row. -**It makes no recommendation.** There is no ranking, no score, and no summary -that adds up to one. Where an approach carries a cost, the cost is stated and the -reader prices it. +**No recommendation is made.** The analysis provides no ranking, score, or +implied recommendation. Costs are stated for readers to evaluate. ## Status Ready to publish, not published. Eight pages, all built: `index.html`, `six-questions.html`, the three approach pages, `scenario.html`, `questions.html` -and `oscal-artifacts.html`. What remains is not a build step: - -- **Proponent review has not happened.** No request for corrections has been sent - to the publishers of the three bodies of content, so none has been received and - nobody has declined. This is stated on the site rather than left implicit. -- Four checks cannot run without network and are skipped locally. They run in CI. -- Two more skip wherever the pre-read and the position paper are not on disk. - Both are Word documents, both live beside the site rather than in it, and - neither is in the corpora repository the workflow checks out, so `--criteria` - and `--questions` skip their verbatim halves in CI as well as locally. Put the - two documents at the root of the corpora directory and both run. +and `oscal-artifacts.html`. Remaining review and verification items: + +- **Proponent review has not happened.** No correction requests have been sent + to the publishers; no corrections have been received, and no publisher has + declined review. The site states the review status. +- Four checks require network access, skip locally, and run in CI. +- The verbatim portions of `--criteria` and `--questions` skip when the pre-read + and position paper are absent. The two Word documents reside outside the site + and are absent from the corpora repository checked out by the workflow, so the + verbatim checks skip in CI and locally. Place both documents at the corpora + directory root to enable the checks. `BUILD-LOG.md` records every phase, every decision, and the open items for each review gate. ## Viewing the site -This analysis is one area of the Pattern Library site and is served with it -rather than on its own. It carried its own `serve.cmd` and `tools/serve.py` -when it was a standalone repository; both are gone, because a second server -rooted here would serve this folder as if it were the whole site and hide the -thing a reader arrives through. +The analysis is served within the Pattern Library. The former standalone +`serve.cmd` and `tools/serve.py` have been removed. A server rooted in the +analysis folder would exclude the library landing pages. -From the repository root, serve `docs/` and open this area from the analysis -index. Pressing F5 in VS Code does both: +From the repository root, serve `docs/` and open the analysis from the analysis +index. Press F5 in VS Code to start the server and open the site: ``` python3 -m http.server --directory docs 4173 @@ -93,8 +83,8 @@ npx serve docs ``` Or install the recommended Live Preview extension and click the preview button -on any page; `.vscode/settings.json` already points its server root at `docs/`. -Either way this analysis is at: +on a page; `.vscode/settings.json` sets the server root to `docs/`. +The analysis path is: ``` /analysis/2026-08-automating-technical-hardening-guidance/ @@ -102,47 +92,44 @@ Either way this analysis is at: ### Why a server, and what happens without one -Every page is markup plus data: the markup carries hooks and `assets/site.js` -fills them from `data/` at load time, so regenerating data never touches markup. -A browser gives a `file://` page an opaque origin and blocks `fetch()`, so a page -opened by double-clicking cannot read its own data files. +Page markup contains hooks populated by `assets/site.js` from `data/` at load +time. Regenerating data leaves markup unchanged. Browsers assign opaque origins +to `file://` pages and block `fetch()`, preventing direct access to data files. -Opening `index.html` directly still works. `assets/bundle.js` carries a copy of -every data file and every diagram, and the page reads that instead and says so in -one line at the top: +Opening `index.html` directly still works: `assets/bundle.js` contains copies +of every data file and diagram. The page reads the bundle and displays a notice: > Opened from a folder rather than a server, so this page read its content from > `assets/bundle.js`, a mirror of `data/` that the build checks byte for byte ... -**That line is the site telling you which copy you are reading, not an error.** -The content is identical: `tools/verify.py --bundle` rebuilds the mirror and fails -on any difference. Serving the site reads `data/` directly and the line does not -appear. GitHub Pages serves over HTTP, so a published copy never shows it. +**The notice identifies the content source, not an error.** +`tools/verify.py --bundle` rebuilds the mirror and fails on any content +difference. Served pages read `data/` directly without the notice. GitHub Pages +serves over HTTP, so published pages do not display the notice. ## Verifying it -One command: +Install dependencies and run verification: ``` pip install pyyaml pillow python tools/verify.py --all ``` -Pillow is not optional: `verify.py` imports `svgrender`, which imports it at -module scope to rasterise the diagrams for `--diagrams`. +Pillow is required: `verify.py` imports `svgrender`, which imports Pillow at +module scope to rasterise diagrams for `--diagrams`. -The corpora are read from `TFG_CORPORA` when it is set, and from the -`corpora_root` in `tools/manifest.yaml` when it is not. This site sits under -`docs/analysis/` in a repository the corpora are not a sibling of, so set it: +Verification reads corpora from `TFG_CORPORA`, falling back to `corpora_root` +in `tools/manifest.yaml` when the environment variable is unset. The site resides +under `docs/analysis/`, not beside the corpora. Set `TFG_CORPORA`: ``` export TFG_CORPORA=/path/to/tfg-automated-assessments ``` -Eighteen checks. Each prints `PASS` or `FAIL` with detail. A check that cannot run -in the current environment prints `SKIP` with the reason and the exact command it -would have run, and is reported separately from the checks that passed. **A skip is -never counted as a pass.** +Eighteen checks report `PASS`, `FAIL`, or `SKIP`. A skipped check reports the +reason and command separately from passed checks. **A skip is never counted as +a pass.** ``` python tools/verify.py --snippets re-extract every extract and diff @@ -168,29 +155,27 @@ python tools/verify.py --pages run each page and inspect what it rendere `node tools/pagecheck.js` runs behind `--pages` and can be run alone, including against one page: `node tools/pagecheck.js six-questions.html`. -**In CI**, `--strict` turns every skip into a failure, on a runner that has the -network and the validators. That is what stops a skipped check from staying -skipped. See `.github/workflows/verify-2026-08-hardening-guidance.yml` at the -root of this repository. +**In CI**, `--strict` treats every skip as a failure on a runner with network +access and validators. See `.github/workflows/verify-2026-08-hardening-guidance.yml` +at the repository root. ### What needs network -| Check | Needs | Command it would run | +| Check | Needs | Command | |---|---|---| | `--schema`, second half | The published NIST 1.2.1 schemas | `curl` the schema, compare each stored fragment | | `--conformance` | An OSCAL validator | `pip install compliance-trestle` | | `--a11y`, second half | axe-core and a headless browser | `npm install --no-save axe-core puppeteer` | -| `--links`, external half | The links themselves | `curl -o /dev/null -w '%{http_code}'` per link | +| `--links`, external half | External link targets | `curl -o /dev/null -w '%{http_code}'` per link | ## How the content is produced -Nothing on this site is transcribed, and nothing is typed twice. Content on this -site comes from one of two places and always says which. An **extract** is taken -from a real published file at a declared JSON pointer and is re-derived on every -build. An **encoding** is written by a generator from data held in that -generator, is labelled on its own bar as ours, and is regenerated and diffed on -every build so it cannot be hand-edited. Every figure, in either case, is -recomputed from the corpora rather than stated. +The site generates content rather than transcribing or duplicating source +material. Labels distinguish two content types. An **extract** comes from a +published file at a declared JSON pointer and is re-derived on every build. An +**encoding** comes from generator-held data, is labelled as authored for the +analysis, and is regenerated and diffed on every build to detect hand edits. +Every figure is recomputed from the corpora. ``` python tools/extract.py # rebuild data/snippets and data/provenance.json @@ -204,42 +189,38 @@ python tools/bundle.py # rebuild the offline fallback python tools/verify.py --all # prove the site says what the files say ``` -That is the order the workflow runs them in, and it is the order they depend on -each other in: the bundle mirrors everything upstream of it, so it goes last. -Four of them take `--check`, which compares instead of writing and exits non-zero -if the committed file is stale: `pattern_examples.py`, `oscal_artifacts.py`, -`sources_files.py` and `bundle.py`. `approach_pages.py` does the same job the -other way round: it records the hash of every page it writes and refuses to -overwrite one that has been hand-edited since, so an edit made in the page rather -than in the data stops the build instead of disappearing into it. +The workflow runs generators in the listed dependency order. The bundle runs +last to mirror all preceding output. Four generators accept `--check` to compare +without writing and exit non-zero for stale committed files: +`pattern_examples.py`, `oscal_artifacts.py`, `sources_files.py`, and `bundle.py`. +`approach_pages.py` records generated-page hashes and refuses to overwrite +hand-edited pages. Editing a generated page instead of source data stops the +build rather than losing the edit. `tools/manifest.yaml` is the declarative source of truth for the extracts. Each -of its 29 entries names a source file, an RFC 6901 pointer, the question the -extract illustrates, and any trimming applied. Trimming is always declared and is -marked in the rendered output; there is no silent truncation. +of the 29 entries names a source file, an RFC 6901 pointer, the illustrated +question, and any trimming applied. Trimming is declared and marked in rendered +output; there is no silent truncation. `data/provenance.json` records the SHA-256 of both the source file and the extracted content. -`tools/pattern_examples.py` is the source of truth for the encodings. Two rules, -one binary and one carrying a value, because the pair is what shows where each -approach puts a parameter. The pair is written into each of the seven question -rows in each approach's shape, plus one composite that puts the whole chain for -one rule in three panes. Where an approach answers a question with nothing, as -assessment-first does for 6a, nothing is encoded and nothing is invented. -`--example` holds the encodings to the disciplines the extracts are held to: real -published identifiers, a real namespace on every property that carries one and -none of them ours, nobody named, and the same rules in the same order in every -column. - -`verify.py` does more than diff. It asserts named facts against the corpora, -recomputes every figure the site cites, and, when the two source documents are on -disk, opens them to confirm that everything the site marks verbatim occurs in the -document it names: the fifteen criteria and their attributions against the -pre-read, the eleven evidence-register items, and the position paper's ten open -questions. When they are not on disk it skips those by name rather than passing -them. The editorial rules were checked against the development plan by a -`--methodology` phase, which went when the methodology page it checked was -removed; they are no longer machine-checked. +`tools/pattern_examples.py` is the source of truth for encodings. Two rules, one +binary and one carrying a value, demonstrate parameter placement. The generator +encodes the rules for each of seven question rows in each approach's structure. +A composite shows one rule's complete chain in three panes. Unanswered questions, +such as assessment-first's 6a, receive no encoding. `--example` checks the same +constraints as extracts: published identifiers, approach namespaces where +required rather than invented namespaces, no named parties, and identical +rule order across columns. + +`verify.py` asserts facts against the corpora and recomputes cited figures. +When source documents are available, verification checks verbatim material +against the named sources: fifteen criteria and attributions against the +pre-read, eleven evidence-register items, and ten open questions from the +position paper. Missing documents produce named skips, not passes. The former +`--methodology` phase checked editorial rules against the development plan. +Removal of the methodology page also removed the check; editorial rules are no +longer machine-checked by that phase. ## Layout @@ -278,141 +259,125 @@ tools/ axe_run.mjs serves the site and runs axe-core over every page ``` -Eight of those are generators and the files they produce must not be edited by -hand: `data/snippets/*.json` and `data/provenance.json`, +Eight tools are generators. Do not hand-edit generated files: +`data/snippets/*.json` and `data/provenance.json`, `data/pattern-examples.json`, `data/oscal-artifacts.json`, `data/source-files.json`, `assets/diagrams/*.svg`, `data/criteria-fill.json`, the -three approach pages, and `assets/bundle.js`. The workflow regenerates all of -them and fails if the result differs from what was committed. +three approach pages, and `assets/bundle.js`. The workflow regenerates the files +and fails on differences from committed versions. -`tools/diagrams.py` builds 19 diagrams and writes 9. The other 10, the base layer -map, the three stakeholder variants and the rest, were only ever on a component -gallery that has been removed. The builders are kept, because the drawing code -belongs in one place, and the `PUBLISHED` constant at the foot of the file is the -list of what reaches `assets/diagrams/`. Add a name there to publish one. +`tools/diagrams.py` builds 19 diagrams and writes 9. The remaining 10 appeared +only in a removed component gallery, including the base layer map and three +stakeholder variants. The builders remain together in the drawing code. +`PUBLISHED`, at the foot of the file, lists diagrams written to +`assets/diagrams/`. Add a diagram name to `PUBLISHED` to publish the diagram. ## Editorial policy -Thirteen rules govern every page. They live in section 3 of the development -plan, `TFG-Rules-and-Checks-Site-Plan.md`, including the seven that a later -decision superseded in part. They were also published on the site, on a -methodology page that has since been removed. The load-bearing ones: +Thirteen rules govern every page, as recorded in section 3 of the development +plan, `TFG-Rules-and-Checks-Site-Plan.md`. A later decision partly superseded +seven rules. The rules also appeared on the removed methodology page. Key rules: -- **The site authors no evaluation criteria.** The fifteen come from the pre-read - unchanged. Their arithmetic is checked on every build, and their wording is - checked against the pre-read on any build that can reach it. +- **The site authors no evaluation criteria.** The fifteen criteria come from + the pre-read unchanged. Every build checks arithmetic; builds with access to + the pre-read also check wording. - **The site carries no quotations and names nobody.** Every characterization - traces to a JSON pointer into a published file, or to the document the claim - comes from. Structural names and option letters only. `tools/pagecheck.js` - enforces it per page, over rendered prose rather than markup, with a file path - exempt because a path is provenance. **`oscal-artifacts.html` is exempt by - name**, and is required to name publishers rather than merely permitted to: an - inventory of who published what is useless with the publisher redacted, and - that page weighs nothing and states no position. Every other page is held to - the rule. -- **Equal budget is enforced by construction**, not by judgement. One generator - writes the three approach pages and refuses to write them if their word counts - differ by more than ten per cent. -- **Non-verbal encoding counts as editorializing.** A badge, a hollow cell or a - hatch fill argues in a channel a word count cannot see, so every annotation type - applies to all three approaches or to none. + traces to a JSON pointer in a published file or a source document. Use + structural names and option letters only. `tools/pagecheck.js` checks rendered + prose per page, exempting file paths as provenance. + **`oscal-artifacts.html` is exempt by name** and must name publishers to + identify document provenance. The inventory states no position. All other + pages follow the rule. +- **Equal budget is enforced by construction.** One generator writes the three + approach pages and rejects word-count differences above ten per cent. +- **Non-verbal encoding counts as editorializing.** Badges, hollow cells, and + hatch fills convey arguments outside word counts. Every annotation type + applies to all three approaches or none. - **Consequences, not verdicts.** State the cost; let the reader price it. - **Every figure carries its denominator**, and every figure is recomputed from the corpora on every build. -One rule could not be met. Rule 5 asks for the same number of extracts on each -approach page, which is unreachable for a structural reason rather than an -editorial one: an unanswered question has nothing to extract, and a corpus that -answers a question with one construct needs one extract where a corpus that -answers it with three needs three. The counts today are 1, 1 and 2. Each page -states its own count and the reason instead, in identical wording, and says that -the count is not a measure of how well documented an approach is. The departure is -stated on each page rather than quietly met in a weaker form. +Rule 5, requiring equal extract counts on approach pages, could not be met. +Unanswered questions provide no extract; an answer using one construct needs one +extract, while an answer using three constructs needs three. The counts are 1, +1, and 2. Each page discloses the count and reason with otherwise identical +wording, including a warning that extract counts do not measure documentation +quality. Each page states the departure from the rule. ## Contributing a correction -**If something about your approach is stated wrongly here, that correction is -more valuable than anything else you could send.** Open an issue or a pull -request. Corrections are published on the site, unedited, with the correction -attributed to whoever made it. - -Four kinds are worth more than the others, and none of them is a matter of -opinion: - -1. **A characterization of your approach that is wrong.** Section 7 of each - approach page states the case for that approach as its proponents would state - it. If it is not the case you would make, say so; that is the one thing no - check on this site can catch. -2. **An encoding written in the wrong shape.** The JSON on the question pages is - ours, not yours: two rules written out in each approach's shape so the columns - are comparable. The build checks that the shape is well formed, uses real - identifiers and invents no namespace. It cannot check that it is the shape you - would actually write, and that is the correction worth the most. -3. **An extract that is unrepresentative.** Every extract is provably what the - file says at the pointer declared. Whether it is the right fragment to show is - a judgement, and the site's author made it. -4. **A figure that is out of date.** Every count is recomputed from the corpora - on each build, so a figure here is what the files said when they were read. If - a publisher has since corrected one, that is worth reporting. - -If an extract looks wrong, run `python tools/verify.py --snippets`. If that -passes and it still looks wrong, the manifest pointer is aimed at the wrong -place, which is a real defect. If an encoding looks wrong, the defect is in -`tools/pattern_examples.py` rather than in a pointer, and -`python tools/verify.py --example` is the check that covers it. - -A criterion cannot be added here. Editorial rule 4 forbids the site from -authoring evaluation criteria, so a sixteenth goes to the working group and this -site follows. +**Report mischaracterizations of an approach through an issue or pull request.** +Corrections are published unedited and attributed to the contributor. + +Priority correction types: + +1. **Mischaracterization of an approach.** Section 7 of each approach page + presents the proponents' case. Proponents should report differences from the + intended argument; automated checks cannot establish agreement. +2. **Incorrect encoding.** The analysis authors encoded two rules in each + approach's structure for comparison. The build checks structure, published + identifiers, and absence of invented namespaces, but cannot confirm that + proponents would use the same encoding. +3. **Unrepresentative extract.** Verification proves the extract matches the + declared file pointer. Selecting a representative fragment remains an + editorial judgement by the analysis author. +4. **Outdated figure.** Counts are recomputed from the corpora on each build. + Report publisher corrections made since the files were read. + +For a suspected extract error, run `python tools/verify.py --snippets`. A +passing check with an unrepresentative extract indicates a manifest-pointer +selection defect. Encoding defects belong in `tools/pattern_examples.py`, not +the manifest; run `python tools/verify.py --example` to check encodings. + +Editorial rule 4 forbids authoring evaluation criteria for the site. Submit a +sixteenth criterion to the working group before inclusion in the analysis. ## Adding a fourth approach -The site is built so that this is a data change and a page, not a rewrite. +Adding an approach requires data changes and a page, not a site rewrite. 1. **Add the corpus.** Put the published files where `corpora_root` in `tools/manifest.yaml` can reach them. 2. **Declare the extracts.** Add manifest entries with a source file, an RFC 6901 - pointer, and the question each one illustrates. Run `python tools/extract.py`. -3. **Add the approach to `data/six-questions.json`.** One entry under `approaches` with - a structural name, an option letter, a primary model, which of the three views - of a rule its proponents hold, which reader it serves first, and its status - annotation. Then one `matrix` cell for each of the seven question rows, each - with a state, a note that becomes its tooltip, and the extracts that evidence - it. An unanswered cell must say which of the three kinds of unanswered it is. -4. **Write the two rules in your shape.** Add a fourth column to + pointer, and the illustrated question. Run `python tools/extract.py`. +3. **Add the approach to `data/six-questions.json`.** Add an `approaches` entry + with a structural name, option letter, primary model, proponents' view of a + rule among the three views, primary reader, and status annotation. Add a + `matrix` cell for each of the seven question rows, with a state, tooltip note, + and supporting extracts. Classify unanswered cells using one of the three + unanswered states. +4. **Encode the two rules in the approach's structure.** Add a fourth column to `tools/pattern_examples.py`: the same two rules, in the same order, for every - question row the approach answers, each block naming the OSCAL construct that - carries it and carrying your own namespace on any property that needs one. A - question the approach answers with nothing gets no block. + answered question row. Each block names the containing OSCAL construct and + uses the approach's namespace on properties requiring a namespace. + Unanswered questions receive no block. 5. **Add a one-line summary** to the same `data/six-questions.json` entry, within ten per cent of the length of the others. 6. **Add the page content** to `tools/approach_pages.py`: the gist, the per-question - prose, the case and the questions against it, and the three status sentences. - The generator supplies the structure and refuses to write pages that break the - budget. + prose, the case for the approach, questions about the approach, and three + status sentences. The generator supplies the structure and rejects pages + exceeding the budget. 7. **Regenerate and verify.** Run the generators in the order given under *How the content is produced*, then `python tools/verify.py --all`. -The checks will tell you what is missing. `--matrix` will reject an unclassified -cell, `--example` will reject a column whose rules do not match the others, -`--budget` will reject an unequal page, and `--criteria` will reject a criteria -table that does not answer all fifteen for the new approach. +Checks identify omissions: `--matrix` rejects unclassified cells, `--example` +rejects columns with mismatched rules, `--budget` rejects unequal pages, and +`--criteria` requires answers to all fifteen criteria for the added approach. -Two more things. The inventory page needs a fourth entry in the `PUBLISHERS` list -in `tools/oscal_artifacts.py`, naming the corpus directory and the option letter; -everything else on that page is counted off the files and nothing about it is -described by hand. And `sources/` lists the hardening guidance read as input, not -the OSCAL written from it, so a new corpus does not go there. +Add a fourth entry to `PUBLISHERS` in `tools/oscal_artifacts.py`, naming the +corpus directory and option letter. The inventory derives the remaining content +from files rather than hand-authored descriptions. Keep the OSCAL corpus outside +`sources/`, which lists input hardening guidance, not derived OSCAL content. ## License -CC BY 4.0. See `LICENSE`. The site's own text, diagrams and encodings are under -that licence. +CC BY 4.0 applies to analysis-authored text, diagrams, and encodings. See +`LICENSE`. -Nothing else here is. The three OSCAL corpora belong to their publishers and are -not redistributed; only extracts at declared pointers, with provenance in -`data/provenance.json`. Under `sources/`, the DISA material is a work of the -United States Government; the CIS Benchmarks are here on the publisher's -participation in this review rather than on a grant in their Agreed Terms of -Use, which is why that basis is written on each of those files rather than -assumed. Each file in `data/source-files.json` carries its own terms. +The licence does not cover source material. The three OSCAL corpora belong to +the publishers and are not redistributed; only extracts at declared pointers +are included, with provenance in `data/provenance.json`. Under `sources/`, DISA +material is a work of the United States Government. CIS Benchmarks are included +on the basis of publisher participation in the review, not a grant under the +Agreed Terms of Use. Each CIS file states the basis for inclusion. +`data/source-files.json` records terms for each file. diff --git a/docs/analysis/2026-08-automating-technical-hardening-guidance/assessment-first.html b/docs/analysis/2026-08-automating-technical-hardening-guidance/assessment-first.html index 7dce04a..80ece04 100644 --- a/docs/analysis/2026-08-automating-technical-hardening-guidance/assessment-first.html +++ b/docs/analysis/2026-08-automating-technical-hardening-guidance/assessment-first.html @@ -4,16 +4,16 @@ Assessment-first | Automating Technical Hardening Guidance with OSCAL - + - + - +