Skip to content

W6: incident evaluation and reporting (IEC-CSO-EFR/DPR/EFI/IIR/OIR/FIR/AIR, VER-TFR-IRI/NRI) - #506

Closed
sam-aydlette wants to merge 2 commits into
feat/frr-recordsfrom
feat/incidents
Closed

sam-aydlette wants to merge 2 commits into
feat/frr-recordsfrom
feat/incidents

Conversation

@sam-aydlette

@sam-aydlette sam-aydlette commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

SCN-Type: not-significant
SCN-Rationale: PROVISIONAL, Sam to confirm the type: adds a scoped IAM role and an incident-report workflow.

Changed

  • Incident register data/incidents.json (schema infrastructure/schemas/incidents.schema.json, stdlib fail-closed loader scripts/_incidents.py following _decisions.py). Each incident records detection, an IEC-CSO-EFR evaluation (time, reportable true/false, basis, reason, federal customer data considered), Incident Rating estimates (IEC-CSO-EFI; rating, customer effect, one or several agencies, reason, time) and the reports issued (Initial → Ongoing → Final, each with time and the items it sets), plus resolved_at / recovery_complete_at and exercise. A reportable incident is rated at the IEC-CSO-DPR default (N5, read from the rule text) until an estimate exists; an estimate whose rating is not the one the EFI bullets give for its effect and agency count is refused.
  • Rules as data (scripts/_rules.py): Class C IIR/OIR/FIR clocks, the EFI rating rubric and the DPR default are read from the pinned corpus; a changed shape fails closed.
  • Report builder scripts/build-incident-report.py: every report goes through _fedramp_schema.write_validated against the vendored incident report schema. Ongoing and Final reports restate every earlier item and mark each one "updated" or "no update". The builder also writes the incident log incident-reports.json and .md (evaluations and reasons, rating history, each report with sha256, each clock with due/issued/on time, reports still due). --check exits 1 when a report is overdue.
  • Triage (policies/triage.rego incident_triggers / incident_errors): VER-TFR-IRI (IRV+LEV+PAIN>N3) and VER-TFR-NRI (NIRV+LEV+N5). The thresholds are parsed from the Class C rule text, and a changed text fails closed. A finding stops triggering once partially mitigated to N3 / N4 or below, and false positives and fully mitigated findings never trigger. Adopted rules are listed in the register policy (both adopted, per docs/policies/ir-policy.md). A trigger that no unresolved incident names puts an Initial report due from the finding's first evaluation at the N5 clock. The VDR builder and its golden report are unchanged.
  • Reconciliation invariant (m): every finding or CVE an open, non-exercise incident names must be in the VDR report. It is registered as reconcile.m (block) in the gate-mode register.
  • AIR workflow .github/workflows/incident-reports.yml runs on push of data/incidents.json to main, on workflow_dispatch, twice daily, and after each successful nightly. Steps: verify the published VDR signature, build, sign (cosign keyless), redaction-gate, publish under .well-known/incident-*, invalidate, check freshness at the edge, then check the report clocks. A failure raises one reused GitHub issue.
  • Bootstrap role infrastructure/bootstrap/incident-reports.tf: s3:PutObject on .well-known/incident-* plus cloudfront:CreateInvalidation. Trust is limited to the incident-reports environment subject. It is classified in the trust-root group, has IIW text, and appears as boundary flow L.
  • Deploy: builds and signs the incident reports, and adds them to the publication redaction gate, the publish step and the round-trip check.
  • Deploy preflight (step "Incident report publisher preflight", early in the deploy job, before AWS credentials and Terraform; scripts/check-incident-publisher.py). It fails the deploy if:
    • the repo variable AWS_INCIDENT_REPORTS_ROLE_ARN is unset, is not an IAM role ARN, or names a role other than github-actions-incident-reports-oidc (the name is tied to the bootstrap file by a test; the ARN and account are never printed);
    • the incident-reports environment does not exist (404), has no custom branch policy, or its branch policies are anything other than exactly main.
    • If the job token cannot read the environment (any other API error), the step says "not observed" and checks only the variable. If the branch-policy list alone cannot be read, it says so and passes on the custom-policy flag.
    • Known-bad fixture: tests/fixtures/incidents/preflight-missing-variable.json (variable missing, environment correct) fails for the variable alone.
  • Proof of the publisher: incident-reports.yml gains workflow_call. A new deploy-workflow job incident-publisher (needs deploy) calls it, so it assumes its own role from the incident-reports environment, builds, signs, publishes and checks the clocks. If it fails, the run fails and notify raises the deployment-failure issue. Only CLOUDFRONT_DISTRIBUTION_ID is passed, not secrets: inherit.
  • Records follow the deployed state: the IIR/OIR/FIR/AIR gaps state only what remains true after a successful deploy.
  • Watchdog: new metric IncidentReportsOverdue, computed from the published log's obligations_due on AWS's clock; a missing or unreadable log counts as 1. New alarm incident-report-overdue. The log has a freshness-config entry (24h, signed).
  • OCR item 7 is now derived from the register; exercises are never counted.
  • Docs (AI-drafted, for review): docs/policies/ir-policy.md, docs/incident-response.md (reportability evaluation step before containment, and a new "Reporting FedRAMP Reportable Incidents (IEC)" section), docs/cr26-applicability.md, docs/runbooks/evidence-sla-alarm.md, CLAUDE.md map.
  • Exercise, dated 2026-11-05, marked exercise: true:
    • INC-2026-001: a simulated IRV LEV N5 vulnerability. Initial report, two Ongoing reports and a Final report, all on time. Rated N5 by default, then estimated N3.
    • INC-2026-002: a simulated page change, evaluated not reportable with its reason.
    • Both timelines are simulated and say so. tests/test_incidents.py runs the exercise end to end on every make check.

Notification is "published only". Nothing is sent to FedRAMP or to any agency. The operator has not authorized sending, and this system has no certification and no agency customers. Every report, the log and the policy say so.

Effect

After a successful deploy (which requires the prerequisites under Needs human):

  • /.well-known/incident-reports.json (signed) and .md, plus four signed exercise reports incident-INC-2026-001-0{1..4}-*.json, are served.
  • The watchdog publishes a fifth metric and alarm.
  • Reconciliation also checks (m).
  • The OCR's incident attestation points to the log.
  • The live VDR (emitted 2026-11-05T17:59Z) has no IRI/NRI triggers, so no clock starts on merge.

Records (python3 scripts/_frr.py, 154 rules), before → after: Implemented 6 → 11, Partially Implemented 93 → 97, Not Implemented 42 → 33, Not Applicable 13 → 13. The follow-up commit changes no statuses, only gap text.

  • Implemented: IEC-CSO-EFR, IEC-CSO-DPR, IEC-CSO-EFI, VER-TFR-IRI, VER-TFR-NRI.
  • Partially Implemented (closes_with: sam): IEC-CSO-IIR, IEC-CSO-OIR, IEC-CSO-FIR, IEC-CSO-AIR. Each gap is: published, not delivered to FedRAMP or agencies, by the operator's decision. For AIR, everything up to publication is automated (and proved by every deploy). What is left is reaching all affected parties (that same decision). A person still records the incident's facts in the register.

Verified

  • make check passes (ruff, mypy, pytest, opa fmt/check/test). Policy coverage is 95.7%, 49/49 policy tests pass.
  • Regal lint on infrastructure/policy is clean. check-policy-annotations.py, check-exceptions.py and check-wasm-safe.py pass.
  • The new tests in tests/test_incidents.py, the watchdog incident tests and the rego incident tests fail on the base, because the code does not exist there.
  • Each of 17 committed known-bad registers fails for its stated reason. The known-bad register for (m) fails with its specific message. Mutated rules fail closed.
  • The builder ran against the live published VDR: 0 triggers, clocks ok.
  • Preflight: fails the known-bad fixture and passes the good one. Tests cover a wrong role name, a user ARN, a bare name, a missing environment, an unrestricted or protected-branches policy, extra branches, and unreadable environments. The ARN's account is not printed. The deploy test checks that the step precedes Configure AWS credentials and Terraform Apply in the deploy job and that incident-publisher is wired into notify. The new tests fail on the previous commit (the script does not exist there).
  • CodeGuard ran over both commits. The one finding, secrets: inherit on the new job (least privilege), was fixed by passing only the CloudFront secret.
  • Read-only check of the live repo on 2026-11-05: AWS_INCIDENT_REPORTS_ROLE_ARN is not set and the incident-reports environment does not exist.
  • Earlier CodeGuard run over the first commit had no findings to fix. Checked: no secrets; pinned actions; least-privilege role with no wildcard actions; the VDR's signature is verified before use; filenames come only from the ID regex; the redaction gate covers every published file.

Residual

  • Reports are published, not delivered, so IIR/OIR/FIR/AIR stay Partially Implemented.
  • The deploy now depends on the incident publisher. An overdue report (the clock step) or an outage of the publisher makes the deploy run fail after the main deploy job has applied. This is intended: the records say the publisher works.
  • The register's report times are recorded by the operator; the published log's emitted_at and signatures show when publication happened.
  • Opening an incident for a VDR trigger is a human act. Until it happens, the Initial clock runs and the alarms fire. If a trigger clears before anyone opens an incident, nothing records it beyond the published logs of that time.
  • Business days skip weekends but count federal holidays, which can only make a deadline earlier.
  • The watchdog evaluates in 2 of 3 hourly periods, so it pages about 2 hours after a missed deadline. The workflow's own check runs on each trigger.
  • verify-published.sh and check-manual-settings.py do not yet cover the incident artifacts or the new environment.
  • Cost: about $0.40/month for the new metric and alarm, plus up to about 60 CloudFront invalidation paths a month (within the free 1,000 alongside existing use).

Needs human

  • Change type unconfirmed. The SCN-Type: not-significant above is provisional. The change adds a scoped IAM role, a new unattended publishing workflow and a deploy dependency on it, so it may be adaptive. Sam to confirm; no SCN record is added here.
  • Before merging, do these, or every deploy of main fails at the preflight. They are not set today:
    1. Bootstrap apply of infrastructure/bootstrap/incident-reports.tf from a tree synced to main.
    2. Set the repo variable AWS_INCIDENT_REPORTS_ROLE_ARN from the incident_reports_role_arn output.
    3. Create the GitHub Environment incident-reports: no reviewer, deployment branches = custom, exactly main.
  • Main stack: the watchdog alarm and metric are applied by the deploy after merge.
  • Decisions to confirm:
    • publish-only notification;
    • NRI adoption;
    • federal incident coordinator text (URL contact, no email);
    • the policy decided_by: Operator;
    • reconcile.m mode block (recorded as proposed);
    • the simulated exercise content, which was authored by the AI. Re-run it yourself if you want a human-performed exercise on record;
    • the 24h freshness window for the log.

Couldn't verify

  • The workflow, the preflight step and the incident-publisher job were not run in Actions. Not run: the role assumption, publishing, cosign signing in CI, and the alarm in AWS.
  • Whether GITHUB_TOKEN (contents: read) can read /environments/incident-reports and its branch policies was not observed. If it cannot, the preflight says so, and the incident-publisher job still fails the deploy when the role cannot be assumed from that environment.
  • Whether a workflow-level concurrency group in a called workflow applies when it is called from the deploy is untested. The groups differ, so no deadlock is possible either way.
  • make check-full / reconcile against live artifacts (needs AWS) was not run.

🤖 Generated with Claude Code

sam-aydlette and others added 2 commits November 5, 2026 14:36
…/NRI triggers

- data/incidents.json + schema + fail-closed loader (scripts/_incidents.py):
  every incident carries its FedRAMP reportability evaluation and reason;
  default N5 until an IEC-CSO-EFI estimate (checked against the rule's rubric).
- scripts/build-incident-report.py renders Initial/Ongoing/Final reports in
  FedRAMP's incident report schema (validated before write) and the incident
  log; Class C clocks and the rating scale are read from the pinned rules.
- policies/triage.rego incident_triggers: VER-TFR-IRI and VER-TFR-NRI, with
  thresholds parsed from the rule text (fail closed on a changed text).
- Reconciliation invariant (m): an open incident's findings are in the VDR.
- incident-reports.yml publishes unattended (publish-only; nothing is sent);
  bootstrap role scoped to .well-known/incident-*; the deploy also publishes;
  watchdog metric and alarm for overdue reports; freshness entry.
- Dated exercise (2026-11-05, exercise: true): IIR -> 2x OIR -> FIR, plus a
  not-reportable evaluation; run end to end in tests.
- FRR records: EFR, DPR, EFI, VER-TFR-IRI, VER-TFR-NRI Implemented; IIR, OIR,
  FIR, AIR Partially Implemented (publish-only; bootstrap apply pending).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s state the deployed system

The deploy fails before applying anything unless AWS_INCIDENT_REPORTS_ROLE_ARN
names the bootstrap incident-reports role and the incident-reports environment
deploys only from main (scripts/check-incident-publisher.py; known-bad fixture
with the variable missing). After the deploy job, the deploy calls
incident-reports.yml (now also workflow_call) on its own role, passing only the
CloudFront secret, so a green deploy proves the publisher publishes.

IEC-CSO-IIR/OIR/FIR/AIR gaps now state only what holds after a successful
deploy: published, not delivered to FedRAMP or agencies, by the operator's
decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sam-aydlette

Copy link
Copy Markdown
Owner Author

Consolidated into #508.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant