M365 Sheppard Agent 🛡️

Enterprise-grade Microsoft 365 security auditor for Managed Service Providers — CLI and interactive terminal UI

📋 Table of Contents

• Overview
• Quick Start
• Terminal UI
• Project Status
• Architecture
• Control Library
• Usage Examples
• Configuration
• Development
• Testing
• Roadmap
• Contributing

🎯 Overview

M365 Sheppard Agent is a production-ready security auditing tool that systematically evaluates Microsoft 365 tenant security posture against 84 controls derived from industry best practices (CIS, NIST 800-53, ISO 27001, CoreView) plus external threat intelligence (OSINT).

Why M365 Sheppard?

• 🔍 Comprehensive Coverage - 84 controls across 9 categories (security + business + OSINT)
• ⚡ Real-time Evaluation - Direct Microsoft Graph API integration
• 🎯 ROI-Based Prioritization - Security impact ÷ implementation effort ranking
• 🖥️ Interactive Terminal UI - Browse controls, run audits, and triage findings without leaving the terminal
• 📊 Professional Reporting - JSON and HTML outputs for dashboards and compliance
• 🔒 Secure by Design - AST-validated code execution, least-privilege permissions
• 🚀 MSP-Ready - Multi-tenant support with parallel execution
• 💻 CLI-First - Scriptable, automation-ready, CI/CD integration

Key Differentiators

Unlike vulnerability scanners or compliance tools, M365 Sheppard provides:

1. Actionable Recommendations - Every finding includes remediation steps with complexity ratings (1-5)
2. Intelligent Prioritization - ROI algorithm ranks fixes by security impact vs. effort
3. MSP Operations - Built for managing multiple tenants at scale
4. Open Architecture - YAML-based control definitions for easy customization
5. Zero Dependencies - No agents, no special permissions beyond read-only Graph API

🚀 Quick Start

Prerequisites

• Python 3.11+ (Download)
• Microsoft 365 tenant with appropriate licensing
• Azure AD app registration with Graph API permissions:
  • SecurityEvents.Read.All
  • Organization.Read.All
  • Policy.Read.ConditionalAccess
  • User.Read.All
  • Directory.Read.All

Installation

# Clone the repository
git clone https://github.com/mstanton/m365-sheppard.git
cd m365-sheppard

# Install with uv (recommended - fastest)
uv sync

# Or install with pip
pip install -e .

# Verify installation
uv run sentinel --version
# Output: M365 Sheppard Agent v0.1.0

First Run

# 1. Initialize configuration
uv run sentinel config init

# 2. Configure your tenant (edit ~/.m365sheppard/config.yaml)
# Add tenant details and authentication credentials

# 3. Set environment variables
export TENANT1_CLIENT_SECRET="your-secret-here"

# 4. Run your first audit
uv run sentinel audit run --tenant your-tenant-name

# 5. View results
cat reports/your-tenant-name_*_audit.json

# ...or do steps 4-5 interactively in the terminal UI
uv run sentinel tui

🖥️ Terminal UI

The TUI is the primary interface for interactive work: browsing the control library, launching audits, and triaging findings — all inside the terminal with a cyberpunk aesthetic. Launch it with:

uv run sentinel tui

Tabs

Tab	What it does
1 · Dashboard	Control library overview: totals per category, severity breakdown
2 · Controls	Browse all 84 controls with live text/severity/category filtering and full detail (description, remediation, compliance)
3 · Audit Run	Launch an audit against a configured tenant with per-category progress bars and a live status stream for every control
4 · Findings	Load any audit report (latest is auto-detected) and triage findings sorted by severity
5 · OSINT	Scan a tenant's users for breach exposure (HIBP) with per-account breach drill-down. Live-only — full addresses are never written to disk

Findings Triage

Selecting a finding shows the full picture in a scrollable detail pane:

• Expected vs. current state — the configuration gap at a glance
• Metrics & evidence — the concrete data the control observed (affected accounts, policy names, DNS records, …)
• Remediation plan — numbered steps with per-step time estimates, complexity rating, total effort hours, and a direct admin-portal link
• Compliance mappings — CIS / NIST / ISO references per finding
• Data-availability warning — flagged when Graph permissions were missing, so partial results are never mistaken for clean ones

Keyboard Reference

Key	Action
1-5	Jump to tab
/	Focus the controls filter
Ctrl+R	Reset filters
Ctrl+G	Run audit
j / k	Move through tables
?	Help overlay
q	Quit

Audits started from the TUI use the same AuditEngine as the CLI — same controls, same sandbox, same report format. When a run finishes, the report is exported and loaded straight into the Findings tab.

📊 Project Status

✅ Phases 1-3 Complete

What's Working:

• ✅ 84 YAML Control Files - Fully implemented across all 9 categories
• ✅ Graph API Integration - Real data collection from Microsoft 365
• ✅ Safe Evaluation Engine - AST-validated sandbox (imports, dunder access, and introspection builtins blocked)
• ✅ Trusted OSINT Executors - Network-based checks (DNS lookups) run as reviewed in-repo code, dispatched by check.type
• ✅ Interactive Terminal UI - Textual-based dashboard, controls browser, audit runner, and findings triage
• ✅ JSON/HTML Reporting - Professional audit reports with metrics
• ✅ Configuration Management - YAML config with environment variable secrets
• ✅ Multi-Tenant Support - Parallel execution for MSP workflows
• ✅ Complexity Scoring - ROI ranking with 1-5 effort ratings
• ✅ Response Caching, Structured Logging, Scheduled Audits, Webhooks
• ✅ CLI Interface - Complete Typer-based command structure
• ✅ Unit Tests - 337 passing tests (engine, sandbox, scoring, OSINT, TUI)

🔧 Implementation Statistics

Control Definitions: 84 YAML files
Categories Covered: 9 (IAM, ATP, DGV, DEV, APP, MON, OPS, BIZ, OSINT)
Graph API Endpoints: 20+
OSINT Integrations: DNS health checks (SPF/DKIM/DMARC), breach detection (HIBP)
Tests: 337 passing (core engine/controls well covered; TUI and Graph
       client layers have targeted smoke coverage)

📈 Real-World Validation

The agent has been run against live Microsoft 365 tenants and produces complete audit reports with scored findings and remediation guidance. Reports land in the configured reports/ directory (tenant data stays local and is never committed).

🏗️ Architecture

Project Structure

m365-sheppard/
├── src/m365_sheppard/          # Core application code
│   ├── cli/                    # Typer-based CLI commands
│   │   ├── main.py            # App entry point
│   │   └── commands/          # Audit, config, etc.
│   ├── core/                  # Business logic
│   │   ├── engine.py          # Audit orchestration
│   │   ├── control.py         # YAML control loader
│   │   ├── evaluator.py       # AST-sandboxed code evaluation
│   │   └── scoring.py         # ROI calculations
│   ├── integrations/          # External services
│   │   ├── graph/            # MS Graph API client
│   │   └── osint/            # DNS graders + trusted OSINT executors
│   ├── models/               # Pydantic data models
│   ├── output/               # Report generators
│   ├── services/             # Scheduler + webhook notifications
│   ├── tui/                  # Textual terminal UI
│   └── utils/                # Configuration, logging, cache, history
├── controls/                  # YAML control definitions
│   ├── iam/                  # Identity & Access (18 controls)
│   ├── atp/                  # Advanced Threat Protection (12)
│   ├── dgv/                  # Data Governance (10)
│   ├── dev/                  # Device Management (8)
│   ├── app/                  # Application Security (6)
│   ├── mon/                  # Monitoring & Response (10)
│   ├── ops/                  # Security Operations (6)
│   ├── biz/                  # Business Optimization (12)
│   └── osint/                # Open Source Intelligence (1)
├── tests/                    # Test suite
│   ├── unit/                 # Unit tests
│   └── integration/          # Integration tests (future)
├── docs/                     # Documentation
├── reports/                  # Audit output directory
└── templates/                # Report templates

Technology Stack

Component	Technology	Purpose
CLI Framework	Typer + Rich	Scriptable command-line interface
Terminal UI	Textual	Interactive dashboard, audit runner, findings triage
API Client	msgraph-sdk	Microsoft Graph API integration
Authentication	azure-identity	OAuth 2.0 / Client credentials
Data Models	Pydantic v2	Type-safe data validation
Configuration	pydantic-settings	Environment-aware config
Templating	Jinja2	HTML report generation
Testing	pytest + pytest-asyncio	Async-friendly test suite
Code Quality	ruff + mypy	Linting and type checking
Logging	structlog	Structured logging
Caching	diskcache	API response caching

Execution Flow

graph TD
    A[CLI Command] --> B[Load Configuration]
    B --> C[Initialize Graph Client]
    C --> D[Authenticate with Azure AD]
    D --> E[Load Control Definitions]
    E --> F{For Each Control}
    F --> G[Fetch Graph API Data]
    G --> H[Execute Evaluation Logic]
    H --> I[Calculate Complexity]
    I --> J[Generate Finding]
    J --> F
    F --> K[Calculate ROI Scores]
    K --> L[Prioritize Findings]
    L --> M[Generate Reports]
    M --> N[Export JSON/HTML]

Loading

📚 Control Library

84 Controls Across 9 Categories

1️⃣ Identity & Access Management (IAM) - 18 Controls

Control ID	Name	Severity	Complexity
IAM-001	MFA for All Users	Critical	⭐⭐ 2/5
IAM-002	Conditional Access Policies	Critical	⭐⭐⭐ 3/5
IAM-003	Password Protection Policies	High	⭐⭐ 2/5
IAM-005	Privileged Identity Management	High	⭐⭐⭐⭐ 4/5
IAM-006	Guest User Access Controls	Medium	⭐⭐ 2/5
IAM-007	Sign-in Risk Policies	High	⭐⭐⭐ 3/5
IAM-008	User Risk Policies	High	⭐⭐⭐ 3/5
IAM-009	Break-Glass Accounts	High	⭐⭐ 2/5
IAM-010	Session Controls	Medium	⭐⭐⭐ 3/5
IAM-011	Self-Service Password Reset	Medium	⭐ 1/5
IAM-012	Smart Lockout	Medium	⭐ 1/5
IAM-013	Directory Synchronization	Medium	⭐⭐⭐ 3/5
IAM-014	Legacy Authentication Blocking	High	⭐⭐ 2/5
IAM-015	Named Locations Configuration	Low	⭐⭐ 2/5
IAM-016	Terms of Use Policies	Low	⭐⭐ 2/5

2️⃣ Advanced Threat Protection (ATP) - 12 Controls

Control ID	Name	Severity	Complexity
ATP-001	Safe Attachments for Exchange	Critical	⭐⭐ 2/5
ATP-002	Safe Links for Exchange	High	⭐⭐ 2/5
ATP-003	Advanced Anti-phishing	Critical	⭐⭐⭐ 3/5
ATP-004	Anti-spam Policies	High	⭐⭐ 2/5
ATP-005	Anti-malware Policies	High	⭐⭐ 2/5
ATP-006	SPF Configuration	High	⭐⭐ 2/5
ATP-007	DKIM Signing	High	⭐⭐ 2/5
ATP-008	DMARC Policy	High	⭐⭐⭐ 3/5
ATP-009	Safe Attachments for SharePoint/OneDrive	Medium	⭐ 1/5
ATP-010	Quarantine Notifications	Low	⭐ 1/5
ATP-011	Connection Filter (IP Lists)	Medium	⭐⭐ 2/5
ATP-012	Zero-hour Auto Purge	High	⭐ 1/5

3️⃣ Data Governance (DGV) - 10 Controls

Includes DLP policies, sensitivity labels, data loss prevention, retention policies, and information protection.

4️⃣ Device Management (DEV) - 8 Controls

Covers Intune compliance, BitLocker enforcement, MDM enrollment, and device configuration.

5️⃣ Application Security (APP) - 6 Controls

OAuth consent policies, risky app detection, service principal management.

6️⃣ Monitoring & Response (MON) - 10 Controls

Audit logging, alert policies, security monitoring, incident response, and threat detection.

7️⃣ Security Operations (OPS) - 6 Controls

Security contacts, privileged access workstations, emergency access accounts, security posture reviews, vulnerability management, and third-party risk assessment.

8️⃣ Business Optimization (BIZ) - 12 Controls

Control ID	Name	Severity	Complexity
BIZ-001	Unused License Detection	Medium	⭐ 1/5
BIZ-002	Inactive User Cleanup	Medium	⭐⭐ 2/5
BIZ-003	Shared Mailbox Licensing	Low	⭐ 1/5
BIZ-004	External Sharing Exposure	High	⭐⭐⭐ 3/5
BIZ-005	Stale Guest Account Cleanup	Medium	⭐⭐ 2/5
BIZ-006	Email Forwarding Detection	High	⭐⭐ 2/5
BIZ-007	Unmanaged Mobile Devices	Medium	⭐⭐⭐ 3/5
BIZ-008	Mailbox Delegation Audit	Medium	⭐⭐ 2/5
BIZ-009	Distribution List Sprawl	Low	⭐⭐ 2/5
BIZ-010	Teams External Access	Medium	⭐⭐ 2/5
BIZ-011	OneDrive Sync Restrictions	Medium	⭐⭐⭐ 3/5
BIZ-012	Password Never Expires	Medium	⭐⭐ 2/5

Business Optimization Focus:

• Cost Savings (BIZ-001, 002, 003) - License waste detection and cleanup ($3K-$5K/year savings potential)
• Security Hardening (BIZ-004, 005, 006, 011) - Zero-cost security improvements for Business Standard tenants
• Compliance & Governance (BIZ-007, 008, 009, 010, 012) - Audit readiness and policy enforcement

9️⃣ Open Source Intelligence (OSINT) - 1 Control

Control ID	Name	Severity	Complexity
OSINT-002	Email Authentication DNS Health Check	High	⭐⭐⭐ 3/5

OSINT Focus:

• External Validation - Real-time DNS lookups for SPF, DKIM, DMARC records (not simulated)
• Letter Grades - A-F grading system clients understand (e.g., "Your domain gets a C grade")
• Specific Findings - Identifies exact issues ("SPF uses ~all instead of -all")
• Microsoft 365 Optimized - Checks selector1 and selector2 DKIM selectors
• Zero Cost - Uses public DNS, no API keys required
• Replaces ATP-006 - ATP-006 simulated checks, OSINT-002 provides real external verification

Control Definition Format

Each control is defined in YAML with the following structure:

control_id: "IAM-001"
category: "IAM"

metadata:
  name: "Multi-Factor Authentication for All Users"
  description: "Ensures MFA is enforced for all user accounts"
  severity: "critical"
  framework_mappings:
    cis: ["1.1.1", "1.1.2"]
    nist: ["IA-2(1)", "IA-2(2)"]
    iso27001: ["A.9.4.2"]

check:
  type: "graph_api"
  data_requirements:
    - endpoint: "/policies/conditionalAccessPolicies"
      permission: "Policy.Read.ConditionalAccess"

  evaluation: |
    def evaluate(data, tenant_context):
        policies = data.get("policies", [])
        mfa_policies = [p for p in policies if "mfa" in p.get("conditions", {}).get("applications", {}).get("includeApplications", [])]

        return {
            "status": "pass" if len(mfa_policies) > 0 else "fail",
            "metrics": {
                "total_policies": len(policies),
                "mfa_policies": len(mfa_policies)
            },
            "evidence": {
                "policy_names": [p["displayName"] for p in mfa_policies[:5]]
            }
        }

remediation:
  summary: "Configure MFA enforcement through Conditional Access"
  steps:
    - "Navigate to Azure AD > Security > Conditional Access"
    - "Create new policy: 'Require MFA for all users'"
    - "Configure: All users, All cloud apps, Grant: Require MFA"
    - "Enable policy and test with non-admin account"

  documentation:
    - title: "Microsoft Documentation"
      url: "https://learn.microsoft.com/en-us/entra/identity/authentication/tutorial-enable-azure-mfa"

complexity:
  base_score: 2
  factors:
    requires_licenses: ["Microsoft Entra ID P1"]
    user_impact: "medium"
    rollback_difficulty: "low"

💻 Usage Examples

Interactive (Terminal UI)

# Launch the TUI — browse controls, run audits, triage findings
uv run sentinel tui

Basic Audit Commands

# Full audit of all categories
uv run sentinel audit run --tenant contoso

# Quick scan (critical/high severity only)
uv run sentinel audit run --tenant contoso --quick

# Skip external OSINT checks (DNS lookups)
uv run sentinel audit run --tenant contoso --skip-osint

# Specific categories
uv run sentinel audit run --tenant contoso --categories IAM,ATP

# List available controls (filter by category/severity)
uv run sentinel audit list-controls --category IAM

# Show a specific control definition
uv run sentinel audit show IAM-001

# Convert an existing JSON report to HTML
uv run sentinel audit to-html reports/contoso_audit.json

Advanced Usage

# Export format: json, html, or both
uv run sentinel audit run --tenant contoso --format both

# Verbose / JSON structured logs
uv run sentinel audit run --tenant contoso --verbose --log-json

# OSINT checks against a domain (no tenant needed)
uv run sentinel osint domain contoso.com

# Audit history and trends
uv run sentinel history list contoso
uv run sentinel history trend contoso

# Scheduled audits + webhook notifications
uv run sentinel schedule add nightly --tenant contoso --cron "0 2 * * *"
uv run sentinel schedule daemon
uv run sentinel webhook add slack --url https://hooks.example.com/sentinel --event audit.completed

Configuration Management

# Initialize new configuration
uv run sentinel config init

# Show current configuration (secrets redacted)
uv run sentinel config show

# Show a specific tenant's configuration
uv run sentinel config tenant contoso

# Test tenant connectivity/authentication
uv run sentinel config test contoso

🔧 Configuration

Configuration File Location

The agent looks for configuration in the following order:

1. ~/.m365sheppard/config.yaml (user-specific)
2. ./config.yaml (project directory)
3. Environment variables (override file settings)

Configuration File Structure

version: "1"

settings:
  output_dir: "./reports"
  controls_dir: "./controls"
  log_level: "INFO"
  cache_enabled: true
  cache_ttl: 3600  # seconds

tenants:
  - name: "contoso"
    tenant_id: "12345678-1234-1234-1234-123456789abc"
    display_name: "Contoso Ltd"
    authentication:
      method: "client_credentials"
      client_id: "your-app-client-id"
      client_secret_env: "CONTOSO_CLIENT_SECRET"
      tenant_id: "12345678-1234-1234-1234-123456789abc"

    # Optional: Override default settings
    output_dir: "./reports/contoso"

  - name: "fabrikam"
    tenant_id: "87654321-4321-4321-4321-cba987654321"
    display_name: "Fabrikam Inc"
    authentication:
      method: "certificate"
      client_id: "your-app-client-id"
      certificate_path: "/path/to/cert.pem"
      tenant_id: "87654321-4321-4321-4321-cba987654321"

Environment Variables

Create a .env file in the project root:

# Tenant authentication secrets
CONTOSO_CLIENT_SECRET=your-secret-here
FABRIKAM_CLIENT_SECRET=another-secret

# API settings (optional)
GRAPH_API_VERSION=v1.0
REQUEST_TIMEOUT=30
RATE_LIMIT_PER_MINUTE=100

# Output configuration (optional)
DEFAULT_OUTPUT_FORMAT=json
LOG_LEVEL=INFO

Azure AD App Registration

Required API Permissions

Navigate to Azure Portal → Azure Active Directory → App registrations → Your App → API permissions

Microsoft Graph (Application permissions):

• SecurityEvents.Read.All - Read security events and reports
• Organization.Read.All - Read organization information
• Policy.Read.ConditionalAccess - Read conditional access policies
• User.Read.All - Read all users' full profiles
• Directory.Read.All - Read directory data
• AuditLog.Read.All - Read audit log data (optional but recommended)
• SecurityActions.Read.All - Read security actions (optional)

Grant Admin Consent

After adding permissions, click "Grant admin consent for [Your Organization]"

Create Client Secret

1. Navigate to "Certificates & secrets"
2. Click "New client secret"
3. Add description: "M365 Sheppard Agent"
4. Set expiration (recommended: 12 months)
5. Copy the secret value immediately (store in .env file)

🧪 Testing

Running Tests

# Install development dependencies
uv sync --extra dev

# Run all tests
uv run pytest

# Run with verbose output
uv run pytest -v

# Run specific test file
uv run pytest tests/unit/test_scoring.py

# Run with coverage report
uv run pytest --cov=src/m365_sheppard --cov-report=html

# Run only unit tests
uv run pytest tests/unit/

# Run integration tests (requires Graph API credentials)
uv run pytest tests/integration/ -m integration

Test Coverage

337 tests cover the security-critical paths end to end: the AST sandbox (every documented bypass vector), the audit engine (dispatch, scoring, OSINT executors), configuration models, and the TUI (headless pilot tests for boot, browsing, filtering, findings triage, and audit wiring). Coverage is deliberately concentrated where correctness matters most — the large presentation layers (TUI rendering, Graph client plumbing) have targeted smoke tests rather than line-by-line coverage.

Code Quality Checks

# Lint code with ruff
uv run ruff check src/ tests/

# Format code with ruff
uv run ruff format src/ tests/

# Type checking with mypy
uv run mypy src/

# All quality checks
uv run ruff check src/ && uv run mypy src/ && uv run pytest

🗺️ Roadmap

✅ Phase 1: Foundation (Completed Q4 2024)

• ✅ Project structure and architecture
• ✅ Pydantic data models
• ✅ CLI framework with Typer
• ✅ Graph API authentication
• ✅ YAML control definition schema
• ✅ Basic unit tests

✅ Phase 2: Core Functionality (Completed Q1 2025)

• ✅ Safe control evaluation engine
• ✅ Configuration file management
• ✅ JSON/HTML report generation
• ✅ Real Graph API data collection
• ✅ 84 control definitions (all categories including OSINT)
• ✅ ROI-based prioritization
• ✅ Multi-tenant support
• ✅ Comprehensive testing

✅ Phase 3: Advanced Features (Complete)

• ✅ Response caching with diskcache
• ✅ Structured logging with structlog
• ✅ OSINT integration (HIBP, DNS health checks)
• ✅ Trusted executor path for network-based controls
• ✅ Historical trending and analytics
• ✅ Scheduled auditing (cron-like daemon)
• ✅ Webhook notifications
• ✅ Interactive terminal UI (dashboard, controls browser, audit runner, findings triage)

🔮 Phase 4: Enterprise Features (Planned)

• 🔮 REST API for external integrations
• 🔮 Ticket system integration (ServiceNow, Jira)
• 🔮 Automated remediation workflows
• 🔮 Compliance report templates (SOC 2, ISO 27001)
• 🔮 AI-powered remediation suggestions
• 🔮 Benchmark data (industry averages)
• 🔮 TUI: multi-tenant portfolio view and diff between audit runs

🤝 Contributing

We welcome contributions from the community!

Development Workflow

1. Fork the repository

   git clone https://github.com/your-username/m365-sheppard.git
   cd m365-sheppard

2. Setup development environment

   uv sync --extra dev

3. Create feature branch

   git checkout -b feature/your-feature-name

4. Make changes and test

   # Run tests
   uv run pytest
   
   # Check code quality
   uv run ruff check src/
   uv run mypy src/

5. Submit pull request

   • Write clear commit messages
   • Update documentation as needed
   • Ensure all tests pass
   • Add tests for new functionality

Contribution Guidelines

• Follow PEP 8 style guidelines
• Use type hints for all function signatures
• Write async-first code (use async/await for I/O operations)
• Add docstrings to public functions and classes
• Maintain 80%+ test coverage for new code
• Update CLAUDE.md with new patterns or conventions

Adding New Controls

1. Create YAML file in appropriate category directory:

   controls/iam/IAM-017.yaml

2. Follow the control schema (see existing controls for examples)

3. Test loading:

   uv run sentinel audit show-control IAM-017

4. Submit PR with control definition and test cases

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Microsoft Graph Team - For comprehensive security APIs
• CIS, NIST, ISO - For security control frameworks
• CoreView - For MSP best practices guidance
• Security Community - For feedback and contributions

📞 Support

Getting Help

Channel	Use For	Response Time
GitHub Issues	Bug reports, Feature requests	1-2 business days
GitHub Discussions	General questions, Ideas	2-3 business days
Documentation	How-to guides, API reference	Self-service

Troubleshooting

See docs/TROUBLESHOOTING.md for common issues and solutions.

Security Issues

Report security vulnerabilities to: security@m365sentinel.dev

---

Made with 🔒 for better Microsoft 365 security

Current Version: 0.1.0 | Last Updated: July 2026 | Status: Phase 3 Complete