Skip to content

Add NIST CMVP module validation status fetcher - #355

Open
claude[bot] wants to merge 10 commits into
mainfrom
claude/nist-cmvp-fetcher
Open

claude[bot] wants to merge 10 commits into
mainfrom
claude/nist-cmvp-fetcher

Conversation

@claude

@claude claude Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Requested by Isaac Teuscher · Slack thread

Before: No fetcher could show that the cryptographic modules a system relies on are FIPS validated. Someone had to open each NIST CMVP certificate page by hand and read its status and sunset date.

After: The new nist_cmvp category has one fetcher, nist_cmvp_module_validation_status. You list certificate numbers in the manifest. The fetcher reads each certificate page on csrc.nist.gov and records whether the module is currently validated, with the reason when it is not.

run:
  fetchers:
    - use: nist_cmvp_module_validation_status
      config:
        certificates: "AWS-LC=4816, 4282"   # the "Label=" part is optional

How: The fetcher sends one GET per certificate to <base_url>/projects/cryptographic-module-validation-program/certificate/<n>. Requests go one at a time, with a 30 s timeout and 2 retries on timeouts, connection errors and 5xx. The User-Agent is paramify-fetchers nist_cmvp_module_validation_status/0.1.0. The page is parsed with the standard library html.parser, and rows are found by label text, not by id or class. A certificate is validated only when all of these are true:

  • the page heading is Certificate #<n> for that number
  • Status is Active
  • Standard is FIPS 140-2 or FIPS 140-3
  • Sunset Date parses as M/D/YYYY and is on or after the run date (UTC)

Every other certificate is not validated and gets one reason: historical, revoked, sunset_passed, status_unknown, not_found (HTTP 404, or the page is for another number), parse_error or fetch_error. A parse_error or fetch_error writes metadata.partial_failure and api_failures[], and the run exits 1 with partial_failure (target_unreachable if every certificate failed). The other certificates are still checked and written. The other reasons are verdicts, so the run exits 0. base_url (env NIST_CMVP_BASE_URL, default https://csrc.nist.gov) is a category setting. The structure mirrors fedramp_marketplace (public data, IDs in config, no secrets). The code is self-contained because a category may not import another category's _shared/.

KSIs: KSI-SVC-SIN (Securing Information) and KSI-SVC-VRI (Validating Resource Integrity). Both carry SC-13 in FedRAMP's crosswalk. VRI is the weaker fit, so drop it if reviewers prefer a strict reading. docs/ksi_mapping.md is regenerated. The regeneration also adds aws_config_inventory, which was already missing from the doc on main.

Live check (csrc.nist.gov, 2026-11-13). I ran the fetcher end to end, both directly and through paramify run:

Cert HTTP Result
4816 200 validated (AWS-LC Cryptographic Module (static), Amazon Web Services Inc., FIPS 140-3, Active, sunset 2029-09-30)
4282 200 historical (FIPS 140-2, "Moved to historical list due to sunsetting")
4710 200 historical (FIPS 140-3, "Replaced by certificate #5386")
1381 200 revoked
99999 404 not_found (NIST answers an unknown number with a real 404 "Page Not Found")

The test fixtures (tests/fixtures/nist_cmvp/synthetic_*.html) are synthetic, as CONTRIBUTING.md asks. They keep the markup of the live 4816, 4282 and 1381 pages (page chrome, algorithm lists, related files and vendor contacts removed), but the certificate numbers (7001 to 7003), module names, vendors, addresses and labs are made up. The other test cases are edits of the 7001 page: an Active FIPS 140-2 page with a 9/21/2026 sunset, a missing Status, an unknown Status, an unparseable sunset, FIPS 140-1, a missing heading, a page for another number, an HTTP 404, 5xx retries and an unreachable host.

Logo: The tile uses the NIST agency logo (the "NIST" wordmark) at Isaac's direction, so the tile matches the others. It is sourced from csrc.nist.gov (https://csrc.nist.gov/CSRC/media/images/svg/nist-logo.svg), cleaned to plain paths, padded to a square viewBox, and saved as fetchers/logos/nist_cmvp.svg with a white nist_cmvp-dark.svg for GitHub dark theme. The FIPS 140 certification mark was deliberately not used: NIST reserves it for vendors of validated modules (source).

Proposed validators (not committed, per the onboard-platform rule that the public repo carries fetchers, not new validators). All four are for EVD-NIST-CMVP-MODULE-VALIDATION. Each opens with MATCH_COUNT NOT_EQUALS 0. Each number must be a plain JSON integer followed by , or }, so 0.0 and 0e0 cannot read as 0.

  • nist_cmvp_modules_validated (configuration): "certificates_not_validated":\s*(?<certificates_not_validated>\d+)(?=\s*[,}]), group 1 EQUALS 0
  • nist_cmvp_none_historical_or_revoked (configuration): "historical":\s*(?<historical>\d+)\s*,\s*"revoked":\s*(?<revoked>\d+)(?=\s*[,}]), groups 1 and 2 EQUALS 0
  • nist_cmvp_none_past_sunset (configuration): "sunset_passed":\s*(?<sunset_passed>\d+)(?=\s*[,}]), group 1 EQUALS 0
  • nist_cmvp_certificates_resolved (completeness): "certificates_total":\s*(?<certificates_total>[1-9]\d*)(?=\s*[,}])[\s\S]*?"not_found":\s*(?<not_found>\d+)\s*,\s*"parse_error":\s*(?<parse_error>\d+)\s*,\s*"fetch_error":\s*(?<fetch_error>\d+)(?=\s*[,}]), groups 2 to 4 EQUALS 0

I proved each one with paramify validators check: 23 cases, 0 failed, 0 vacuous passes. The cases cover a compliant artifact, a non-compliant one, a renamed key, a failed collection, and the float or exponent forms. I also scored real envelopes. 4816 alone passes the set. 4816 plus 4282 fails none_historical_or_revoked and modules_validated. The set also belongs in validators/common/collection_succeeded.yaml through sync.

Checks: pytest -q -n auto 10219 passed, 91 skipped. tests/test_nist_cmvp.py has 28 tests. ruff check framework/ and mypy framework/ are clean, and ruff and mypy are also clean on the new fetcher. paramify list, catalog, ksi, describe and tools/cli_goldens.py pass. nist_cmvp is text-format checked by default under main's scheme, which checks every fetcher not on TEXT_FORMAT_PENDING, so the instructions layout is enforced without a registration.

Not verified

  • Only five live certificates were checked, and none of them is Active with a sunset date in the past. On the live site, FIPS 140-2 certificates past their 9/21/2026 sunset already show Historical. The sunset_passed path is tested only against an edited page.
  • How NIST answers rate limiting or a very large number of requests. Runs are sequential, but a long list has not been tried.
  • The onboarding sandbox gates (claim, sandbox, slate, build notes, teardown) do not apply: the data is public, so there is no tenant to provision and no .onboarding/ state. The check_onboarding.py checks that read the repo were run directly and pass: comments concise, TLS verification on by default, no validators on the branch.
  • No docs/README.md "Platform setup" link was added. That list covers credential setup. FedRAMP Marketplace has a fetchers/fedramp_marketplace/README.md but is not linked there, and neither are Qualys, Sophos, Cornerstone or Oracle Cloud. No guide asks for the link.

Generated by Claude Code

New nist_cmvp category with one fetcher,
nist_cmvp_module_validation_status. It reads the public CMVP certificate
page for each certificate number in the `certificates` config and reports
whether the module is currently FIPS 140-2 or FIPS 140-3 validated:
the page is for that number, Status is Active, and the sunset date has
not passed. Anything else is not validated, with a reason code.

Tests run against a local HTTP server with fixtures trimmed from the
live pages for certificates 4816 (Active), 4282 (Historical) and 1381
(Revoked). docs/ksi_mapping.md is regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Text tile, no logo: NIST reserves the FIPS 140 mark for validated-module
vendors and gives no general permission for third-party use of its logo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Replace the text-only tile with a logo tile that matches the others.
The NIST agency wordmark comes from csrc.nist.gov, padded to a square
viewBox, with a white variant for GitHub dark theme.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
CONTRIBUTING.md asks for synthetic data in tests and fixtures. The three
certificate pages were trimmed copies of live csrc.nist.gov pages with real
vendor names, addresses and labs. They keep the same markup but now carry
made-up certificate numbers (7001-7003), names, addresses and labs, and are
named synthetic_*.html like the web and zendesk fixtures.

The category file's comment is one line, per the onboard-platform comment
rule (check_onboarding.py warned on the 6-line block).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
@claude
claude Bot marked this pull request as ready for review November 13, 2026 20:17
Resolve CHANGELOG [Unreleased] conflict by keeping both the NIST CMVP
entry and main's Azure selected-subscription entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Resolve the README Supported services conflict by keeping main's Sophos
and Cornerstone tiles and adding the NIST CMVP tile after them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Brings in #350, #354, #361, #362 and #363. CHANGELOG [Unreleased] keeps
the NIST CMVP entry alongside main's new Azure entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
@claude

claude Bot commented Nov 14, 2026

Copy link
Copy Markdown
Author

CI note: the red test (py 3.12) job on bbae31e (job) is not caused by this PR. It's inherited from main.

Failing tests (2 failed, 9873 passed):

  • tests/test_azure_private_endpoint_scope.py::test_related_lookups_and_discovery_use_selected_mode[True]: assert collector.ok is False
  • tests/test_azure_private_endpoint_scope.py::test_main_records_lookup_scope[True]: assert native.main() == 0 returns 1

Both log API call failed: network.NetworkManagementClient (init) (ModuleNotFoundError: No module named 'azure').

Also fails on main. The same two tests fail on main at d8c6ddb in test (py 3.11/3.12/3.13/3.14/3.15): CI run 43561431999 (for example the py 3.12 job). The two CI runs on main before it (for #361 and #362) were cancelled, so d8c6ddb is the first completed run that includes the change.

Where it comes from: #361 (fix(azure): scope private endpoint related reads), which added tests/test_azure_private_endpoint_scope.py. The test stubs azure.mgmt.network in sys.modules, but in the strict=True case (AZURE_TARGET_SUBSCRIPTION_ONLY=true), arm_client_kwargs() calls read_boundaries.sdk_arm_boundary_options(). That function imports azure.core.pipeline.policies, which the test doesn't stub. The generic test (py *) jobs install only .[dev,tui], without the Azure SDK, so the import fails. The test passes in the azure-sdk (surface) job, which does install the SDK. The strict=False cases pass everywhere.

This PR's changes (the nist_cmvp fetcher, its tests and docs) don't touch any Azure code or this test. The fix belongs on main: either stub azure.core.pipeline.policies in the test fixture or move the test to the azure-sdk job.


Generated by Claude Code

Brings in the pipeline fetchers work (#334) and wiz_inventory (#349).
CHANGELOG [Unreleased] keeps both sides' Added entries;
docs/ksi_mapping.md is main's, regenerated with tools/gen_ksi_mapping.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Picks up #371 (private endpoint scope tests run without the Azure SDK)
and #365 (azure-mgmt-containerinstance requirement bump).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Resolve tests/test_contracts.py by taking main's TEXT_FORMAT_PENDING
denylist: every fetcher is now text-format checked by default, so the
nist_cmvp entry in the old TEXT_FORMAT_FETCHERS allowlist is no longer
needed and nist_cmvp stays enforced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WA75NTunK2CFNDjwSpwg8m
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant