Skip to content

Commit cf20ce2

Browse files
judeperCopilot
judeper
and
Copilot
committed
Add automated assessment engine for 78 governance controls
Introduce assessment/ directory with programmatic tenant assessment: - manifest/controls.json: machine-readable definitions for all 78 controls with checks, zone thresholds, pass conditions, and manual questions - 5 PowerShell collectors (PPAC, Graph, Purview, SharePoint, Sentinel) that gather tenant configuration via APIs and write JSON output - engine/score.py: Python scoring engine that evaluates checks against collected data, derives maturity scores (0-4), and sets confidence - engine/report.py: generates assessment-prefilled.md, manual-questionnaire.md, and assessment-summary.json using jinja2 templates - run-assessment.ps1: orchestrator with -Zone, -AuthMode, -SkipCollectors - 11 pytest tests (7 scoring + 4 report) with fixture data for 5 representative controls covering pass/fail/manual/missing-data scenarios - assessment/output/ gitignored (customer data must not be committed) - Updated README.md, AGENTS.md, copilot-instructions.md, CLAUDE.md to document the assessment engine Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9b0d7fe commit cf20ce2

27 files changed

Lines changed: 8719 additions & 0 deletions

‎.claude/claude.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,13 @@ FSI-AgentGov/
114114
│ ├── extract_assessment_data.py # Assessment data extraction (78 controls → JSON)
115115
│ ├── compile_researcher_package.py # Research package generator
116116
│ └── hooks/ # Claude Code hooks
117+
├── assessment/ # Automated assessment engine
118+
│ ├── manifest/controls.json # Machine-readable 78-control definitions
119+
│ ├── collectors/ # 5 PowerShell collectors (PPAC, Graph, Purview, SharePoint, Sentinel)
120+
│ ├── engine/ # Python scoring (score.py) and report generator (report.py)
121+
│ ├── tests/ # pytest tests with fixture data
122+
│ ├── run-assessment.ps1 # Main orchestrator
123+
│ └── output/ # Run outputs (gitignored — customer data)
117124
├── data/ # Runtime data (state files)
118125
├── reports/ # Generated reports
119126
│ └── learn-changes/ # Learn documentation change reports
@@ -234,6 +241,9 @@ python scripts/learn_monitor.py --dry-run --limit 5
234241

235242
# Regenerate researcher package after control changes
236243
python scripts/compile_researcher_package.py
244+
245+
# Run assessment engine tests
246+
cd assessment && pip install -r requirements.txt && pytest tests/ -v
237247
```
238248

239249
### What "Pass" Means

‎.github/copilot-instructions.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,13 @@ docs/
9090
├── images/ # Screenshot verification (LOCAL ONLY - gitignored)
9191
└── downloads/ # Excel templates for admins
9292
scripts/ # Validation scripts (verify_controls.py, verify_templates.py, extract_assessment_data.py)
93+
assessment/ # Automated assessment engine (collectors, scoring, reports)
94+
│ ├── manifest/controls.json # Machine-readable 78-control definitions
95+
│ ├── collectors/ # 5 PowerShell data collectors (PPAC, Graph, Purview, SharePoint, Sentinel)
96+
│ ├── engine/ # Python scoring (score.py) and report generator (report.py)
97+
│ ├── tests/ # pytest tests with fixture data
98+
│ ├── run-assessment.ps1 # Main orchestrator
99+
│ └── output/ # Run outputs (gitignored — customer data)
93100
releases/ # Release artifacts by version
94101
mkdocs.yml # Site navigation and configuration
95102
.config/wt.toml # Worktrunk project hooks (worktree management)
@@ -246,8 +253,26 @@ python scripts/verify_templates.py
246253

247254
# Validate Excel templates (control counts, version references)
248255
python scripts/verify_excel_templates.py
256+
257+
# Run assessment engine tests
258+
cd assessment && pip install -r requirements.txt && pytest tests/ -v
249259
```
250260

261+
## Automated Assessment Engine
262+
263+
The `assessment/` directory contains a programmatic assessment engine that collects Microsoft 365 tenant configuration via APIs, scores all 78 controls against zone thresholds, and generates pre-filled assessment reports.
264+
265+
**Key components:**
266+
- `manifest/controls.json` — machine-readable definitions for all 78 controls with checks, zone thresholds, and manual questions
267+
- `collectors/` — 5 PowerShell collectors (PPAC, Graph, Purview, SharePoint, Sentinel) that write JSON to `output/collected/`
268+
- `engine/score.py` — evaluates checks against collected data, derives maturity scores (0–4)
269+
- `engine/report.py` — generates `assessment-prefilled.md`, `manual-questionnaire.md`, and `assessment-summary.json`
270+
- `run-assessment.ps1` — orchestrator with `-Zone`, `-AuthMode`, `-SkipCollectors` parameters
271+
272+
**Usage:** `.\assessment\run-assessment.ps1 -TenantId <id> -Zone 2 -AuthMode Interactive -CustomerName "Contoso"`
273+
274+
See `assessment/README.md` for full prerequisites and usage documentation.
275+
251276
## Worktree Management (Parallel Agent Runs)
252277

253278
```bash

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,6 @@ docs/javascripts/assessment-data.json
5757
# GSD planning state (kept locally, not tracked)
5858
.planning/
5959
.codex/
60+
61+
# Assessment engine outputs (customer data — never commit)
62+
assessment/output/

‎AGENTS.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,8 +211,26 @@ Additional validations (when applicable):
211211
```bash
212212
python scripts/verify_excel_templates.py # After template changes
213213
python scripts/compile_researcher_package.py # After pillar control changes
214+
cd assessment && pytest tests/ -v # After assessment engine changes
214215
```
215216

217+
## Automated Assessment Engine
218+
219+
The `assessment/` directory contains a programmatic assessment engine that collects tenant configuration via APIs, scores all 78 controls against zone thresholds, and generates pre-filled reports with a focused manual questionnaire.
220+
221+
**Structure:**
222+
- `manifest/controls.json` — machine-readable 78-control manifest (checks, zone thresholds, manual questions)
223+
- `collectors/` — 5 PowerShell collectors: Collect-PPAC, Collect-Graph, Collect-Purview, Collect-SharePoint, Collect-Sentinel
224+
- `engine/score.py` — Python scoring engine (evaluates pass_conditions, derives maturity 0–4, sets confidence)
225+
- `engine/report.py` — generates assessment-prefilled.md, manual-questionnaire.md, assessment-summary.json
226+
- `run-assessment.ps1` — orchestrator (validates, runs collectors, calls scoring + reporting)
227+
- `tests/` — pytest suite with fixture data for 5 representative controls
228+
229+
**When editing assessment code:**
230+
1. Run `cd assessment && pytest tests/ -v` to verify no regressions
231+
2. If modifying `controls.json`, ensure all 78 entries are present with required schema fields
232+
3. `assessment/output/` is gitignored — customer data must never be committed
233+
216234
## Advanced Implementations
217235

218236
Complex multi-control solutions in `docs/playbooks/advanced-implementations/`:

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,13 @@ Companion automation lives in the **[FSI-AgentGov-Solutions](https://github.com/
157157
- **Administrator Excel Templates** - Role-specific checklists and dashboards (see [Downloads](docs/downloads/index.md))
158158
- **Offline Deliverables** - This repository ships **web docs + Excel templates only** (no Word/PDF document bundle)
159159

160+
### Automated Assessment Engine
161+
The `assessment/` directory provides programmatic governance assessment:
162+
- **5 PowerShell collectors** gather tenant configuration from PPAC, Graph, Purview, SharePoint, and Sentinel
163+
- **Python scoring engine** evaluates 78 controls against zone thresholds (maturity 0–4)
164+
- **Report generator** produces a pre-filled assessment with evidence tables plus a focused manual questionnaire for the ~30 controls requiring human attestation
165+
- See [Assessment README](assessment/README.md) for prerequisites and usage
166+
160167
---
161168

162169
## 🚀 Quick Start

‎assessment/README.md‎

Lines changed: 271 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,271 @@
1+
# FSI-AgentGov Assessment Engine
2+
3+
Automated governance assessment for Microsoft 365 Copilot Studio deployments in
4+
Financial Services. Collects tenant configuration via APIs, scores controls
5+
against zone-specific thresholds, and generates a pre-filled assessment with a
6+
focused manual questionnaire.
7+
8+
## Architecture
9+
10+
```
11+
run-assessment.ps1 ← Orchestrator (PowerShell)
12+
├── collectors/ ← Data collection scripts (one per API surface)
13+
│ ├── Collect-PPAC.ps1
14+
│ ├── Collect-Graph.ps1
15+
│ ├── Collect-Purview.ps1
16+
│ ├── Collect-SharePoint.ps1
17+
│ └── Collect-Sentinel.ps1
18+
├── engine/ ← Scoring & report generation (Python)
19+
│ ├── score.py
20+
│ └── report.py
21+
├── manifest/
22+
│ └── controls.json ← 78-control definition manifest
23+
├── tests/ ← Automated tests
24+
│ ├── fixtures/ ← Synthetic test data
25+
│ ├── test_score.py
26+
│ └── test_report.py
27+
└── output/ ← Generated assessment artifacts
28+
├── collected/ ← Raw collector JSON
29+
├── scores.json
30+
├── assessment-prefilled.md
31+
├── manual-questionnaire.md
32+
└── assessment-summary.json
33+
```
34+
35+
## Prerequisites
36+
37+
### PowerShell Modules
38+
39+
| Module | Collector | Purpose |
40+
|--------|-----------|---------|
41+
| `Microsoft.PowerApps.Administration.PowerShell` | PPAC | Environment, DLP, role assignment data |
42+
| `Microsoft.Graph.Authentication` | Graph | Entra ID authentication |
43+
| `Microsoft.Graph.Identity.SignIns` | Graph | Conditional Access policies |
44+
| `Microsoft.Graph.Groups` | Graph | Security group enumeration |
45+
| `ExchangeOnlineManagement` | Purview | Audit log, retention, compliance policies |
46+
| `PnP.PowerShell` | SharePoint | Site permissions, sharing, grounding validation |
47+
| `Az.OperationalInsights` | Sentinel | Log Analytics workspace & connector status |
48+
49+
### Python
50+
51+
- **Python 3.10+** required
52+
- Install dependencies:
53+
54+
```bash
55+
cd assessment
56+
pip install -r requirements.txt
57+
```
58+
59+
### Entra ID Permissions
60+
61+
#### Interactive Mode
62+
63+
Sign in as a Global Admin or a user with **all** of the following roles:
64+
65+
| Role | Scope |
66+
|------|-------|
67+
| Power Platform Administrator | PPAC collector |
68+
| Compliance Administrator | Purview collector |
69+
| Security Reader | Graph collector |
70+
| SharePoint Administrator | SharePoint collector |
71+
| Log Analytics Reader | Sentinel collector |
72+
73+
#### Service Principal Mode
74+
75+
Register an Entra ID application with the following **application** permissions:
76+
77+
| Permission | API | Collectors |
78+
|-----------|-----|------------|
79+
| `Policy.Read.All` | Microsoft Graph | Graph |
80+
| `Group.Read.All` | Microsoft Graph | Graph |
81+
| `Directory.Read.All` | Microsoft Graph | Graph |
82+
| `AuditLog.Read.All` | Microsoft Graph | Graph |
83+
| `Sites.Read.All` | Microsoft Graph | SharePoint |
84+
| `Files.Read.All` | Microsoft Graph | SharePoint |
85+
| Power Platform Admin API consent | Power Platform | PPAC |
86+
87+
> **Note:** Some collectors (Purview, Sentinel) may still require interactive
88+
> authentication even in ServicePrincipal mode, depending on tenant
89+
> configuration. Use `-SkipCollectors` for those if needed.
90+
91+
## Quick Start
92+
93+
### Interactive Mode (Recommended)
94+
95+
```powershell
96+
.\run-assessment.ps1 `
97+
-TenantId "00000000-0000-0000-0000-000000000000" `
98+
-Zone 2 `
99+
-AuthMode Interactive `
100+
-CustomerName "Contoso Financial" `
101+
-SubscriptionId "00000000-0000-0000-0000-000000000001" `
102+
-ResourceGroup "rg-sentinel" `
103+
-WorkspaceName "sentinel-workspace"
104+
```
105+
106+
### Service Principal Mode
107+
108+
```powershell
109+
$secret = Read-Host -AsSecureString "Client Secret"
110+
111+
.\run-assessment.ps1 `
112+
-TenantId "00000000-0000-0000-0000-000000000000" `
113+
-Zone 3 `
114+
-AuthMode ServicePrincipal `
115+
-ClientId "00000000-0000-0000-0000-000000000002" `
116+
-ClientSecret $secret `
117+
-CustomerName "Contoso Financial" `
118+
-ApprovedSitesCsv ".\approved-sites.csv" `
119+
-SubscriptionId "00000000-0000-0000-0000-000000000001" `
120+
-ResourceGroup "rg-sentinel" `
121+
-WorkspaceName "sentinel-workspace"
122+
```
123+
124+
### Skipping Collectors
125+
126+
Skip one or more collectors when their prerequisites aren't available:
127+
128+
```powershell
129+
.\run-assessment.ps1 `
130+
-TenantId "..." `
131+
-Zone 2 `
132+
-AuthMode Interactive `
133+
-CustomerName "Contoso Financial" `
134+
-SkipCollectors @("Sentinel", "Purview")
135+
```
136+
137+
> When a collector is skipped, controls that depend on its data are scored with
138+
> **confidence: low** and flagged for manual review.
139+
140+
### Custom Output Directory
141+
142+
```powershell
143+
.\run-assessment.ps1 ... -OutputDir "C:\Assessments\contoso-2026-03"
144+
```
145+
146+
## Zones
147+
148+
The FSI-AgentGov framework defines three deployment zones with increasing
149+
governance requirements:
150+
151+
| Zone | Name | Description |
152+
|------|------|-------------|
153+
| 1 | **Standard** | Internal agents, low-risk data, minimal regulatory overlap |
154+
| 2 | **Sensitive** | Customer-facing or PII-handling agents, moderate compliance |
155+
| 3 | **Regulated** | Agents in regulated workloads (banking, insurance, capital markets) |
156+
157+
Higher zones require more checks to pass per control, resulting in higher
158+
maturity thresholds. A zone-3 assessment is the most stringent.
159+
160+
## Output Files
161+
162+
| File | Format | Description |
163+
|------|--------|-------------|
164+
| `output/collected/*.json` | JSON | Raw data snapshots from each collector |
165+
| `output/scores.json` | JSON | Per-control scores, check results, confidence, and evidence |
166+
| `output/assessment-prefilled.md` | Markdown | Pre-filled assessment report organized by pillar and control |
167+
| `output/manual-questionnaire.md` | Markdown | Interview questions for controls requiring manual validation |
168+
| `output/assessment-summary.json` | JSON | Machine-readable summary for dashboards and CI integration |
169+
170+
## Maturity Scale
171+
172+
Each control receives a maturity score from 0–4:
173+
174+
| Score | Level | Description |
175+
|-------|-------|-------------|
176+
| 0 | **Not Implemented** | Control is absent or all checks failed |
177+
| 1 | **Baseline** (25%) | Minimal implementation — meets zone-1 threshold |
178+
| 2 | **Recommended** (50%) | Standard compliance — meets zone-2 threshold |
179+
| 3 | **Advanced** (75%) | Strong implementation with enhanced protections |
180+
| 4 | **Fully Regulated** (100%) | Complete implementation — meets zone-3 threshold |
181+
182+
A control's maturity score is determined by comparing the number of passing
183+
checks against the zone-specific threshold in the controls manifest. If the
184+
passing count is below the minimum, the score is 0 (Not Implemented).
185+
186+
## Confidence Levels
187+
188+
Each control score includes a confidence indicator:
189+
190+
| Level | Meaning |
191+
|-------|---------|
192+
| **High** | All required API calls returned valid data |
193+
| **Medium** | Some data sources returned partial results or warnings |
194+
| **Low** | Required data sources were unavailable, null, or errored |
195+
196+
Low-confidence scores are highlighted in the assessment report and
197+
automatically added to the manual questionnaire for validation.
198+
199+
## Collector Exit Codes
200+
201+
| Code | Meaning | Orchestrator Behavior |
202+
|------|---------|----------------------|
203+
| 0 | **Success** | All data collected cleanly |
204+
| 1 | **Partial** | Some API calls failed; partial data saved | Logged as warning; run continues |
205+
| 2 | **Failure** | Collector could not produce usable data | Logged as error; run continues |
206+
207+
The orchestrator never halts on collector failures. Downstream scoring handles
208+
missing data by lowering confidence.
209+
210+
## Running Tests
211+
212+
```bash
213+
cd assessment
214+
pip install -r requirements.txt
215+
pytest tests/ -v
216+
```
217+
218+
### Test Coverage
219+
220+
| File | Tests | Focus |
221+
|------|-------|-------|
222+
| `tests/test_score.py` | 7 | Zone thresholds, maturity scoring, confidence, summaries |
223+
| `tests/test_report.py` | 4 | Output file generation, Markdown structure, JSON schema |
224+
225+
### Test Fixtures
226+
227+
Test fixtures in `tests/fixtures/` provide synthetic tenant data for
228+
deterministic, offline testing:
229+
230+
| Fixture | Contents |
231+
|---------|----------|
232+
| `controls_subset.json` | 5-control manifest (controls 1.1, 1.3, 2.1, 3.1, 4.4) |
233+
| `ppac.json` | Power Platform environment, DLP, and role assignment data |
234+
| `graph.json` | Conditional Access policies and Entra ID configuration |
235+
| `purview.json` | Audit log config, retention policies |
236+
| `sharepoint.json` | Site inventory, sharing settings, grounding scope |
237+
| `sentinel.json` | Log Analytics workspace and connector status |
238+
| `expected_scores.json` | Expected zone-2 scoring output for validation |
239+
240+
## Idempotency
241+
242+
Running the orchestrator multiple times with the same parameters cleanly
243+
overwrites the output directory. No append-only files or cumulative state is
244+
maintained between runs.
245+
246+
## Troubleshooting
247+
248+
### "Python 3 is required but was not found"
249+
250+
Ensure Python 3.10+ is installed and available as `python3`, `python`, or `py`
251+
on your PATH.
252+
253+
### Collector authentication errors
254+
255+
- **Interactive mode:** Ensure you have the required admin roles and that your
256+
browser session isn't blocked by Conditional Access.
257+
- **ServicePrincipal mode:** Verify the app registration has the correct API
258+
permissions and that admin consent has been granted.
259+
260+
### Missing Sentinel parameters
261+
262+
If you don't use Microsoft Sentinel, skip its collector:
263+
264+
```powershell
265+
.\run-assessment.ps1 ... -SkipCollectors @("Sentinel")
266+
```
267+
268+
### Partial collector results
269+
270+
When a collector returns exit code 1, check the `_metadata.warnings` array in
271+
its output JSON for details on which API calls failed.

0 commit comments

Comments
 (0)