Skip to content

Add TFG automated assessments analysis site and tools - #10

Merged
brian-ruf merged 8 commits into
mainfrom
add-automated-assessments-analysis
Aug 25, 2026
Merged

brian-ruf merged 8 commits into
mainfrom
add-automated-assessments-analysis

Conversation

@pjavan

@pjavan pjavan commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Add analysis and recommendation areas

Important

Work in progress, not final. Opened for review of the structure and the lifecycle rules. Details below will still move.

The repository currently holds worked examples only. There is no place to record the analysis behind them, or the recommendations that analysis produces.

This PR adds support for analysis, recommendations with the first analysis included.


The lifecycle

Analyses  ------>  Recommendations  ------>  Patterns
  what               what we                what we
we weighed          recommend               publish

Analyses. An analysis is an effort that debates one question about OSCAL. Each gets its own dated area under docs/analysis/ and is retained after it finishes, so a later reader can see what was weighed.

Status: draft, active, concluded, superseded.

Recommendations. A recommendation is what an analysis produces when it reaches one. It lives in its own area under docs/recommendations/ so it can be cited directly and superseded independently of the analysis behind it.

Status: proposed, accepted, superseded.

Retention. An area is never renamed, moved or deleted, and a concluded analysis is not edited to match a later view. A superseding effort gets its own area, and the old one is marked as superseded by it.

Linking. An analysis names the recommendation that settled it, and a recommendation names the analysis it came from. Concluding an analysis sets its recommendation field and stamps concluded, and its page then renders a banner pointing forward.


Adding an analysis

Each index renders from a registry file, so adding an effort means adding a folder and appending one object. No page is edited and no build runs.

File Drives
docs/analysis/analyses.json the Analyses index
docs/recommendations/recommendations.json the Recommendations index

Each analysis also carries an analysis.json beside its index.html holding the question, status, dates, the approaches compared, and the recommendation it produced.

Recommendations ships empty

There are no recommendations yet. The area is built anyway, with an empty registry and a page that states the position, because the open analysis makes no recommendation by design rather than because it is unfinished. A blank page would invite a reader to assume a verdict is pending. analysis.json carries makesRecommendation: false for the same reason.


The first analysis

Automating Technical Hardening Guidance with OSCAL, from the OSCAL Foundation Technical Focus Group on Automated Assessments. Opened 2026-08-12, active.

Where does technical hardening guidance live in OSCAL, and how does an automated check attach to it?

Three organizations have published working OSCAL content answering this three different ways. The site puts them side by side against the same six questions (Catalog-first, Component-first, Assessment-first) and writes the same two rules out in all three shapes, so the only thing differing between the columns is the modelling rather than the subject matter.

Eight pages. Published guidance from CIS, DISA, CISA and AWS is inventoried as input: 16 documents, 29.0 MB, one row per benchmark with its provenance.


VS Code

Press F5. The Pattern Library configuration runs the site: serve task, which serves docs/ on port 4173, then opens the landing page in Edge with the debugger attached. Breakpoints in docs/assets/site.js or in an analysis's scripts bind to the served file. Ending the session stops the server.

Run and Debug
Pattern Library (Edge / Chrome) landing page, debugger attached
Debug the 2026-08 analysis harness steps through that analysis's verify.py
Run Task
site: serve / site: stop the server on its own
analysis 2026-08: verify recompute every figure and extract from source
analysis 2026-08: regenerate run every generator for that analysis

The serve task is a stock python3 -m http.server over docs/, so nothing in this repository serves the site. Any static server works the same way. Live Preview is recommended in extensions.json with its server root already set to docs/; installing it replaces the task.

Opening a page from the filesystem does not work, because each index reads its registry over fetch and a browser blocks that on a file:// origin. The pages report it rather than appearing empty.


CI/CD

pages.yml runs on a push to main that touches docs/, and publishes docs/ to GitHub Pages. There is no build step: what is in the folder is what is served. Two guards run first:

  • Every path is relative. A project site is served under /Pattern-Library/, so an absolute path resolves against the wrong root.
  • Every analysis the registry names is published, with its metadata. An entry pointing at nothing renders as an ordinary card with a dead link, which nothing except a reader clicking it would catch.

verify-2026-08-hardening-guidance.yml runs that analysis's own harness, and a paths: filter scopes it to that analysis's folder. It runs twice: an offline pass covering what a contributor can reproduce on a laptop, then --strict, which counts every skipped check as a failure so a skip cannot go unnoticed. The schedule reruns it weekly, since the external schemas and links it checks can change without anyone touching the repository.

Each analysis brings its own workflow on the same naming convention.


Notes for reviewers

  • Pages is not enabled yet. Settings, then Pages, then Source, then GitHub Actions. The github-pages environment lint warning clears on first deploy.
  • The CIS material ships on a recorded basis, not a licence grant. CIS is a stakeholder in this review and confirmed that copying and linking infringes nothing. The call is recorded in BUILD-LOG.md, and each file carries the basis in its own terms line. A fork made outside that conversation does not inherit it.
  • Known gaps, deliberately left: a stale not_shipped entry for CIS in data/source-files.json (nothing renders it, but it is a stale licensing assertion), and no favicon.

pjavan added 8 commits August 25, 2026 10:55
Copies tfg-rules-and-checks-site into
docs/analysis/2026-08-automating-technical-hardening-guidance/ with no
edits, so the next commit's relocation changes are reviewable on their own.

Excluded from the copy:
  *:Zone.Identifier, *:sec.endpointdlp   129 NTFS alternate-data-stream
                                         files that materialised on the
                                         WSL side; not content
  __pycache__/, *.pyc, build/            generated

The site is unbuilt in this state: tools/manifest.yaml still points at
../../tfg-automated-assessments, which does not resolve from this depth.
The site was written to sit at a repository root with the corpora as its
sibling. Nested under docs/analysis/ neither holds, so:

corpora    tools/extract.py grows corpora_root(), which reads TFG_CORPORA
           first and falls back to the manifest path. Four call sites
           resolved this independently, two of them hardcoded; they now
           share one. No relative path reaches every caller's checkout,
           because the corpora are not this repository's sibling either.

CI         .github/workflows/verify.yml moves to the repository root as
           verify-2026-08-hardening-guidance.yml, scoped by a paths filter
           so it runs on the analysis folder rather than on every push.
           The repository is checked out under site/ so the corpora stay
           outside the working tree the byte-identical check git-diffs.

Two defects found by moving, both pre-existing:

  - verify.py never created build/grayscale before writing 16 renders to
    it. build/ is gitignored, so --diagrams failed on any fresh clone.
  - CI installed pyyaml only. verify.py imports svgrender imports PIL at
    module scope, so the harness could not have started. Pillow and
    fonts-dejavu-core, which svgrender loads by absolute path, are now
    installed.

Verified from the new location: 2631/2639, all 8 generators byte-identical.
The 8 failures are 5 NIST schema 404s, 2 unreachable namespace URIs and a
missing local oscal-cli; all 8 reproduce against the unmoved original.
The site shipped its CIS material and its OSCAL tree between the README
being written and the build ending, and the README still described the
state before that.

  - 17 documents under sources/ is 16, and 29.3 MB is 29.0 MB
  - sources/oscal/ and sources/cis/ are no longer gitignored. oscal has
    no row because sources_files.py excludes it from the inventory, not
    because it is absent
  - the paragraph telling a reader to download the Benchmarks from
    cisecurity.org to rebuild described a repository that no longer
    needs it. Replaced with the basis BUILD-LOG records, and with the
    fact that a fork outside that conversation does not inherit it
  - compare.html has been scenario.html since before the build ended
    (README twice, CORRECTIONS.md template once)

Also brought forward from the migration: pillow is required, TFG_CORPORA
is how the corpora are found, and the workflow has moved and been renamed.
The Foundation's primary, #4E79D8, is 4.16:1 on the light background and
4.36:1 on the dark one. verify.py --a11y holds a link to 4.5 in both, so
adopting the brand colour as the accent would have failed the check this
site's stylesheet is written around.

Its hue and saturation are adopted instead, hue 221 and 0.64, and only
lightness moves — to wherever the contrast the site already had comes
back:

              was              now
  light   #0B5FA5 6.57:1   #2958BC 6.53:1
  dark    #6EB5F5 8.29:1   #96B0E8 8.36:1

Every accent pair is within 0.07 of where it was, and the hue is the
Foundation's rather than the cyan-leaning 205 it had been. --focus tracks
--accent as it did before.

The six question hues, the three approach hues, the layer hues and the
code surface are untouched: each is held to a measured property that a
brand match would have cost. --css and --a11y pass 158/158; the diagrams
carry var() rather than baked values, so nothing regenerates.
Both variants the Foundation ships, downscaled from 1024x472 to 208x96
and served at 2.25rem: the blue mark on the light theme, the reversed
white one on dark. Two <img> rather than <picture>, because the theme
here is a data-theme attribute a reader toggles rather than a media
query, and <picture> cannot follow that.

The mark is the one thing on the page drawn in the brand blue at full
strength, 4.16:1. WCAG exempts a logotype from the ratio it holds text
to, which is the reason --accent had to move off that colour and this
did not.

alt="" on both: the link is already named by the title beside them, and
a second accessible name would be announced twice.

The header is written in six places -- assets/shell.html, the copy in
tools/approach_pages.py, and the four pages no generator owns -- so all
six carry it and the four generated pages were rebuilt. All 8 pages now
show the mark. 2631/2639, unchanged from before the theme work.
docs/ is the site and there is no build step. The deploy workflow adds
one thing: summit/ copied to patterns/summit/, so the pages that link the
artifacts are same-origin with them rather than pointing at a
branch-pinned raw.githubusercontent URL.

  docs/            ->  _site/
  summit/          ->  _site/patterns/summit/

Three areas, each with its own lifecycle:

  patterns/         the worked examples. Names all seven models and says
                    which two are published, because which models an
                    example is missing is part of what it is
  analysis/         one dated area per effort, rendered from
                    analyses.json. Adding an effort means adding an area
                    and appending to that file; no page is edited
  recommendations/  an empty registry and a page that says so. Built now
                    rather than when the first one lands, so the shape is
                    proven while it costs nothing

The recommendations page states that the open analysis makes no
recommendation by design rather than leaving a reader to infer one is
pending. analysis.json carries makesRecommendation: false for the same
reason.

Two guards run before deploy: every path is relative, because a project
site is served under /Pattern-Library/ and an absolute path resolves
against the wrong root; and every analysis the registry names is
published with its metadata, because a registry entry pointing at
nothing renders as an ordinary card with a dead link.

The shell follows prefers-color-scheme rather than carrying a toggle.
An analysis keeps its own toggle and its own stored choice, so crossing
between them can change the theme; that is the cost of each analysis
owning its look and being frozen with it.

Verified against the assembled tree: 16 paths resolve, both registries
render, the empty state shows, and the analysis site serves from its
nested location with no broken asset. The harness is unmoved at
2631/2639.
- Introduce `serve.py` to serve the Pattern Library site locally.
- Create VS Code launch configurations for site and analysis debugging.
- Add tasks for serving the site and managing analysis workflows.
- Update `.gitignore` to exclude the assembled preview tree.
- Enhance README with instructions for running the site.
@brian-ruf

Copy link
Copy Markdown
Contributor

After conversation with @pjavan , we agreed to publish this as-is so that it can be published on Pages and people can review online. this is a work in progress and additional PRs will be issued with updates.

@brian-ruf
brian-ruf merged commit 7665e74 into main Aug 25, 2026
2 of 6 checks passed
@pjavan
pjavan deleted the add-automated-assessments-analysis branch September 9, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants