From a3d4817ec6cf4430effaffff1e7c714ca14c896b Mon Sep 17 00:00:00 2001 From: Connor Dalton Date: Fri, 13 Nov 2026 11:15:38 -0700 Subject: [PATCH 1/4] Check every fetcher's text format by default, with a shrink-only pending list TEXT_FORMAT_PENDING (one entry per line, capped by a frozen baseline file) replaces the opt-in list. A pending fetcher that already passes must leave it. Rewrite the descriptions of two Entra fetchers and the AI services fetcher's instructions so they are checked. State the limits in the guide and skills. Co-Authored-By: Claude Sonnet 5.5 --- .claude/skills/create-fetcher/SKILL.md | 6 +- .claude/skills/onboard-platform/SKILL.md | 5 +- docs/authoring_fetcher_guide.md | 8 +- .../ai_services_configuration/fetcher.yaml | 159 ++++---- .../entra_app_registrations/fetcher.yaml | 14 +- .../azure/entra_sign_in_logs/fetcher.yaml | 3 +- tests/test_azure_resource_exclusions_ai.py | 5 + tests/test_contracts.py | 366 ++++++++++++------ tests/text_format_baseline.txt | 213 ++++++++++ 9 files changed, 588 insertions(+), 191 deletions(-) create mode 100644 tests/text_format_baseline.txt diff --git a/.claude/skills/create-fetcher/SKILL.md b/.claude/skills/create-fetcher/SKILL.md index b577981a0..567a0435d 100644 --- a/.claude/skills/create-fetcher/SKILL.md +++ b/.claude/skills/create-fetcher/SKILL.md @@ -140,7 +140,11 @@ Read these before building (skim, don't summarize back): `docs/authoring_fetcher_guide.md` ("The `description` and `instructions` text"): a one-sentence description, then plain-text instructions in a `|` block with Script (`//`), Access, Commands Executed, Output - and Summary fields. Paramify shows them as typed, so no markdown. + and Summary fields. Paramify shows them as typed, so no markdown. Hard limits: + the description is one sentence of at most 30 words ending in a period, with + no fetcher name; a summary field line is at most 800 characters; the + instructions at most 20,000. `tests/test_contracts.py` checks every new + fetcher's text. - `category` — from the tool name. **Ask the user (batch into ONE `AskUserQuestion`, ≤3 questions):** diff --git a/.claude/skills/onboard-platform/SKILL.md b/.claude/skills/onboard-platform/SKILL.md index dc1c40d29..f327c8b0f 100644 --- a/.claude/skills/onboard-platform/SKILL.md +++ b/.claude/skills/onboard-platform/SKILL.md @@ -101,7 +101,10 @@ person. The conventions it reads are in `references/state.md`. `teardown.sh`. The derivation of each field goes in the fetcher's `evidence_set.instructions` (laid out as in `docs/authoring_fetcher_guide.md`: Script, Access, Commands Executed, Output, Summary fields), where an assessor - reads it; the build story goes + reads it. Hard limits: the description is one sentence of at most 30 words + ending in a period, with no fetcher name; no markdown; a summary field line + at most 800 characters; the instructions at most 20,000. The contract test + checks every new fetcher's text. The build story goes in `notes/`. The third run wrote 45–74-line module docstrings restating both. Committed code never cites `.onboarding/`, which is gitignored. - **In the public repo, the branch carries fetchers, never new validators.** diff --git a/docs/authoring_fetcher_guide.md b/docs/authoring_fetcher_guide.md index 07b7b13c6..b641f8a71 100644 --- a/docs/authoring_fetcher_guide.md +++ b/docs/authoring_fetcher_guide.md @@ -163,7 +163,9 @@ It fails on: [`azure/alert_rules`](../fetchers/azure/alert_rules/fetcher.yaml) is a full example. `tests/test_contracts.py` checks: -- the layout, for the fetchers listed in `TEXT_FORMAT_FETCHERS`; +- the layout, for every fetcher except those on `TEXT_FORMAT_PENDING` (older + fetchers that still fail it; the list only shrinks, and a new fetcher is + never added to it); - that a test calls `assert_summary_documented` for every fetcher whose instructions list summary fields. Fetchers that listed them before the check existed wait in `SUMMARY_DOCS_PENDING`, which is meant only to shrink. Its @@ -171,8 +173,8 @@ example. `tests/test_contracts.py` checks: raising the count, which shows in review. A new fetcher calls the fixture instead. -Until a platform is converted, its instructions only need to open with the -`Script:` path. +A fetcher on `TEXT_FORMAT_PENDING` only needs instructions that open with the +`Script:` path, until its text is rewritten and it leaves the list. ## 3. Write `fetcher.py` and smoke-test it diff --git a/fetchers/azure/ai_services_configuration/fetcher.yaml b/fetchers/azure/ai_services_configuration/fetcher.yaml index 8393a3c50..228c4790e 100644 --- a/fetchers/azure/ai_services_configuration/fetcher.yaml +++ b/fetchers/azure/ai_services_configuration/fetcher.yaml @@ -1,15 +1,6 @@ name: azure_ai_services_configuration version: 0.2.2 -description: > - Inventories Azure AI services accounts (Microsoft.CognitiveServices/accounts — Azure AI Foundry, - Azure OpenAI, Language, Speech, Document Intelligence and the other kinds) and Azure AI Search - services in the subscription. For every account and search service: how reachable it is from the - internet, whether API keys still work or only Microsoft Entra ID sign-in does, customer-managed key - encryption, managed identity and whether audit logs are exported. For Foundry and OpenAI accounts - also: every model deployment with its publisher, where it may process data and the content filter and - Prompt Shields protection it runs under; every content filter policy, flagging custom ones weaker - than Microsoft's default; every Foundry project; and every connection agents use to reach other - systems, with whether it holds a stored secret. +description: Shows how exposed, authenticated, encrypted and logged each Azure AI services account and AI Search service is, including model deployments, content filters and agent connections. category: azure config_schema: @@ -76,58 +67,96 @@ ksis: evidence_set: reference_id: EVD-AZURE-AI-SERVICES-CONFIG name: Azure AI Services Configuration - instructions: > - Script: azure/ai_services_configuration/fetcher.py. Azure SDK calls: azure-mgmt-cognitiveservices CognitiveServicesManagementClient - accounts.list(); for accounts of kind AIServices or OpenAI also deployments.list, rai_policies.list - and projects.list (resource_group, account); azure-mgmt-monitor diagnostic_settings.list for each - account and search service. Read as ARM JSON through the same authenticated client, because the SDK - drops connection auth types it has no model for and the repo has no AI Search SDK: GET - {account}/connections and {project}/connections (api-version 2025-06-01) and - /subscriptions/{id}/providers/Microsoft.Search/searchServices (api-version 2025-05-01). No call reads - keys, connection credentials or model content; connection targets are recorded without query string. - Accounts: network_exposure is private when publicNetworkAccess is Disabled, restricted when it is - not but the network rules default to Deny, and open otherwise (Azure treats unset values as Enabled - and Allow). private_endpoint_but_public_access marks an account with a private endpoint that is - still reachable publicly. api_keys_enabled is false only when disableLocalAuth is true (it defaults - to false). customer_managed_key is true when encryption.keySource is anything other than - Microsoft.CognitiveServices. audit_logs_exported is true when a diagnostic setting with a - destination (including a Marketplace partner such as Datadog) enables the audit category or the - audit or allLogs category group, false otherwise, and null when diagnostic settings could not be - read. The category is Audit for an AI services account and OperationLogs for an AI Search service, - which has no Audit category. - Deployments: model_publisher is the model format (OpenAI, Anthropic, MoonshotAI, ...) and - third_party_publisher is true for publishers other than OpenAI and Microsoft. processing_scope comes - from the deployment type: global (Global* types, may process data in any Azure region), data_zone - (DataZone* types, within a geography), regional (Standard, ProvisionedManaged, Batch) or unknown. - content_filter judges the hate, sexual, violence and self-harm filters, jailbreak_protection and - indirect_attack_protection the Prompt Shields filters; each is blocking when the deployment's content - filter (RAI) policy has them enabled and blocking, non_blocking when not, none when the deployment - names no policy, missing when the named policy is not on the account, and unknown when policies - could not be listed. Microsoft's default policy blocks jailbreaks but does not enable indirect-attack - detection. version_upgrade_option shows whether the model is upgraded automatically. - Content filter policies: custom is true for user-managed policies; weaker_than_default marks a custom - policy that does not block every harm category and jailbreaks, or that sets any harm filter's - severity_threshold looser than Microsoft's DefaultV2 (Medium on prompts and completions). A threshold is - the lowest severity blocked, so High blocks only the most severe content and is looser than Medium; - harm_filters_looser_than_default names those filters and harm_thresholds_at_least_default is false. - A missing or unrecognised threshold leaves both null (unknown, counted in - summary.content_filter_policies_weaker_unknown), never "at least default"; in_use is whether any deployment uses - it; asynchronous_filtering marks policies that let output stream before filtering completes; - non_blocking_filters lists every filter that is off or annotate-only. - Connections: listed at account and project level (the account list repeats its projects' - connections; summary counts each once per account). uses_stored_secret is true for auth types that - keep a credential in the connection (ApiKey, CustomKeys, PAT, UsernamePassword, AccessKey, SAS, - AccountKey, ServicePrincipal, OAuth2), false for Entra ID and managed identity types, and null for - unrecognised types (counted in connections_auth_type_unrecognized). remote_tool_connections are tools - and MCP servers agents can call. grounding_with_bing_connections counts Grounding with Bing Search - connections, which Microsoft documents as sending queries outside the Azure compliance boundary. - Search services: network_exposure is private when publicNetworkAccess is disabled, restricted when - IP rules are set, open otherwise; auth_mode is entra_only, api_key_only or entra_or_api_key; - cmk_enforcement is the service's customer-managed key enforcement. - A per-account list that could not be read is null and sets metadata.partial_failure. When the account or search service list itself was not read (a refused call, or no credential or subscription), its results list stays empty and the summary counts of that kind are null. provider_registration_status - (registered / not_registered / unknown) is the Microsoft.CognitiveServices resource provider's state. - Accounts and search services the resource_exclusions patterns take out are listed under excluded instead of accounts and search_services. - resource_exclusion_patterns lists the configured patterns, empty when none are set; resource_exclusion_patterns_unmatched the ones that matched no - account or search service, such as a mistyped name, or null when either list could not be read; excluded_accounts and excluded_search_services count those left out of scope because a pattern matched - their resource group, region or id; in_scope_accounts and in_scope_search_services count those read and in scope. They, and the total_* and other counts of a kind, are null when that list could not be read; 0 means the - patterns took out every one, which a check can catch by requiring more than 0. + instructions: | + Script: azure/ai_services_configuration/fetcher.py + + Access: Reader on the subscription. + + Commands Executed: + - subscriptions.list() — picks a subscription when none is set + - providers.get("Microsoft.CognitiveServices") — tells "not registered" from "no accounts" + - CognitiveServicesManagementClient accounts.list() — every Azure AI services account (Foundry, OpenAI, Language, Speech, Document Intelligence and other kinds) with its network, key, encryption and identity settings + - deployments.list(), rai_policies.list() and projects.list() (per Foundry or OpenAI account) — model deployments, content filter policies and Foundry projects + - GET {account}/connections and {project}/connections (ARM REST, api-version 2025-06-01; per account and per project) — connections agents use to reach other systems, recorded without credentials or query strings + - GET /subscriptions/{id}/providers/Microsoft.Search/searchServices (ARM REST, api-version 2025-05-01) — every AI Search service with its network, key and encryption settings + - diagnostic_settings.list() (per account and search service) — whether audit logs are exported + + Output: azure_ai_services_configuration_.json — each AI services account, search service, model deployment, content filter policy, project and connection with the verdicts below. Resources the resource_exclusions patterns take out are listed under excluded instead. + + Summary fields: + + What was checked? + - total_accounts — Number of AI services accounts checked; null when the account list could not be read + - accounts_by_kind — Number of accounts for each kind of account (AIServices, OpenAI, and so on), with one entry per kind found + - total_search_services — Number of AI Search services checked; null when the search service list could not be read + - provider_registration_status — Whether the AI services provider (Microsoft.CognitiveServices) is registered on the subscription: registered, not_registered or unknown + + Which resources are left out of scope? + - resource_exclusion_patterns — List of the configured resource_exclusions patterns (rg:, location: or id:); empty when none are set, so every resource is in scope + - resource_exclusion_patterns_unmatched — List of configured patterns that matched no account or search service, such as a mistyped name; null when either list could not be read + - excluded_accounts — Number of accounts left out of scope because a pattern matched their resource group, region or id; none of their settings are read and no other count includes them + - excluded_search_services — Number of search services left out of scope for the same reason + - in_scope_accounts — Number of accounts read and in scope; 0 means the patterns took out every account, which a check can catch by requiring more than 0 + - in_scope_search_services — Number of search services read and in scope; 0 means the patterns took out every one + + Can the accounts be reached from the internet? + - accounts_open_to_all_networks — Number of accounts reachable from any network (public access is on and the network rules allow all) + - accounts_network_restricted — Number of accounts with public access on but the network rules denying by default + - accounts_private_only — Number of accounts with public network access disabled + - accounts_with_private_endpoints — Number of accounts with at least one private endpoint connection + - accounts_with_private_endpoint_but_public_access — Number of accounts that have a private endpoint but can still be reached publicly + - accounts_with_outbound_restricted — Number of accounts that restrict their own outbound network access + - search_services_open_to_all_networks — Number of search services reachable from any network (public access is on and no IP rules are set) + + Do keys still work, or only Microsoft Entra ID sign-in? + - accounts_with_api_keys_enabled — Number of accounts where API keys still work (local authentication is not disabled) + - accounts_entra_only — Number of accounts where API keys are disabled so only Microsoft Entra ID sign-in works + - accounts_entra_only_percentage — Percentage of accounts that accept only Entra ID sign-in = accounts_entra_only / total_accounts × 100 (a whole number, 0–100); 0 when total_accounts is 0 + - accounts_without_managed_identity — Number of accounts with no managed identity + - search_services_with_api_keys_enabled — Number of search services where API keys still work + + Is data encrypted with a customer-managed key? + - accounts_with_customer_managed_key — Number of accounts encrypted with a key the customer manages instead of the Microsoft-managed key + - search_services_cmk_enforced — Number of search services that enforce customer-managed key encryption + + Are audit logs exported? + - accounts_with_diagnostic_settings — Number of accounts with at least one diagnostic setting + - accounts_audit_logs_exported — Number of accounts whose diagnostic setting sends audit logs to a destination + - accounts_audit_logs_unknown — Number of accounts whose diagnostic settings could not be read + - search_services_audit_logs_exported — Number of search services whose diagnostic setting sends operation logs to a destination + + Where may model deployments process data, and who publishes the models? + - total_deployments — Number of model deployments across all accounts + - deployments_global_processing — Number of deployments that may process data in any Azure region (Global deployment types) + - deployments_data_zone_processing — Number of deployments that process data within one geography (DataZone deployment types) + - deployments_regional_processing — Number of deployments that process data in the account's own region (Standard, ProvisionedManaged or Batch types) + - deployments_processing_scope_unknown — Number of deployments whose deployment type is not recognised + - deployments_third_party_publisher — Number of deployments whose model publisher is neither OpenAI nor Microsoft + - deployments_by_model_publisher — Number of deployments for each model publisher (OpenAI, Anthropic and so on; unknown when none is reported), with one entry per publisher found + - deployments_with_automatic_version_upgrade — Number of deployments whose model is upgraded automatically when a new default version is available or the current one expires + + Do deployments run under a content filter? + - deployments_content_filter_blocking — Number of deployments whose content filter policy has the hate, sexual, violence and self-harm filters enabled and blocking + - deployments_content_filter_non_blocking — Number of deployments whose policy has any of those filters off or annotate-only + - deployments_content_filter_none — Number of deployments that name no content filter policy + - deployments_content_filter_missing — Number of deployments that name a policy that is not on the account + - deployments_content_filter_unknown — Number of deployments whose account's policies could not be listed + - deployments_jailbreak_protection — Number of deployments whose policy blocks jailbreak attempts (Prompt Shields) + - deployments_indirect_attack_protection — Number of deployments whose policy blocks indirect attacks hidden in documents or tool output (Prompt Shields) + - deployments_asynchronous_filtering — Number of deployments whose policy lets output stream before filtering completes + + Are custom content filter policies weaker than Microsoft's default? + - content_filter_policies_custom — Number of content filter policies the customer manages + - content_filter_policies_weaker_than_default — Number of custom policies that do not block every harm category and jailbreaks, or set a severity threshold looser than Microsoft's default (a threshold is the lowest severity blocked, so High is looser than Medium) + - content_filter_policies_weaker_unknown — Number of policies whose thresholds are missing or unrecognised, so the comparison is unknown + - content_filter_policies_custom_unused — Number of custom policies that no deployment uses + + What can agents reach, and with what credentials? + - total_projects — Number of Foundry projects across all accounts + - total_connections — Number of connections across all accounts and projects (an account list that repeats a project's connection counts it once) + - connections_using_stored_secrets — Number of connections that keep a credential (key, token, password, SAS or service principal secret) instead of using Entra ID or a managed identity + - connections_unauthenticated — Number of connections that use no authentication + - connections_auth_type_unrecognized — Number of connections whose authentication type is not recognised, so whether they hold a secret is unknown + - connections_unknown_accounts — Number of Foundry or OpenAI accounts whose connections could not be read + - remote_tool_connections — Number of connections that are tools or MCP servers agents can call + - grounding_with_bing_connections — Number of Grounding with Bing Search connections, which send queries outside the Azure compliance boundary diff --git a/fetchers/azure/entra_app_registrations/fetcher.yaml b/fetchers/azure/entra_app_registrations/fetcher.yaml index b97bd1aea..78922b801 100644 --- a/fetchers/azure/entra_app_registrations/fetcher.yaml +++ b/fetchers/azure/entra_app_registrations/fetcher.yaml @@ -1,14 +1,6 @@ name: azure_entra_app_registrations version: 0.2.1 -description: > - Every Microsoft Entra ID app registration with each of its client secrets and - certificate credentials and that credential's expiry date, so credential rotation - is evidenceable per credential. Three expiry states are kept distinct — already - expired, inside the 30-day rotation window, and no expiry date at all — because a - never-expiring credential never appears in an "expiring soon" report by - construction. Rotation compliance is reported over credentials, excluding apps that - hold none. Each credential also carries the lifetime it was issued for, its end - date minus its start date. +description: Lists every Microsoft Entra ID app registration with each client secret and certificate, its expiry state and issued lifetime, so credential rotation can be shown per credential. category: azure supports_targets: true @@ -96,8 +88,8 @@ evidence_set: Do credentials expire and get rotated? - total_credentials — Number of secrets and certificates across all app registrations; null if the applications read failed or stopped part-way - - password_credentials — Number of those that are client secrets; null if the applications read failed or stopped part-way - - certificate_credentials — Number of those that are certificates; null if the applications read failed or stopped part-way + - password_credentials — Number of credentials that are client secrets; null if the applications read failed or stopped part-way + - certificate_credentials — Number of credentials that are certificates; null if the applications read failed or stopped part-way - expired_credentials — Number of credentials whose end date has passed; null if the applications read failed or stopped part-way - credentials_expiring_soon — Number of credentials that expire within rotation_threshold_days days and haven't expired; null if the applications read failed or stopped part-way - credentials_without_expiry — Number of credentials with no end date; null if the applications read failed or stopped part-way diff --git a/fetchers/azure/entra_sign_in_logs/fetcher.yaml b/fetchers/azure/entra_sign_in_logs/fetcher.yaml index 0d513d525..6561d1671 100644 --- a/fetchers/azure/entra_sign_in_logs/fetcher.yaml +++ b/fetchers/azure/entra_sign_in_logs/fetcher.yaml @@ -1,7 +1,6 @@ name: azure_entra_sign_in_logs version: 0.3.0 -description: >- - Shows from the workspace's Entra sign-in logs whether successful sign-ins required MFA and phishing-resistant methods, the changes made to the authentication and Conditional Access policies, and, as counts only, the non-interactive user sign-ins and the service-principal (application) sign-ins that detections and least-privilege reviews depend on. +description: Shows from Entra sign-in logs whether successful sign-ins required MFA and phishing-resistant methods, what changed in authentication and Conditional Access policies, and counts of non-interactive and service-principal sign-ins. category: azure supports_targets: true diff --git a/tests/test_azure_resource_exclusions_ai.py b/tests/test_azure_resource_exclusions_ai.py index 6f77f2fbc..46c049e47 100644 --- a/tests/test_azure_resource_exclusions_ai.py +++ b/tests/test_azure_resource_exclusions_ai.py @@ -256,6 +256,11 @@ def test_services_without_patterns_adds_only_empty_exclusion_fields(monkeypatch, } +def test_services_summary_matches_the_instructions(monkeypatch, tmp_path, assert_summary_documented): + _, evidence, *_ = run_services(monkeypatch, tmp_path, "rg:rg-out") + assert_summary_documented("azure/ai_services_configuration", evidence["summary"]) + + def test_services_exclude_by_group_and_region_and_skip_account_reads(monkeypatch, tmp_path): code, evidence, calls, paths, diag = run_services(monkeypatch, tmp_path, "rg:rg-out, location:regioneast, rg:typo") summary, results = evidence["summary"], evidence["results"] diff --git a/tests/test_contracts.py b/tests/test_contracts.py index 181753c90..264821f7f 100644 --- a/tests/test_contracts.py +++ b/tests/test_contracts.py @@ -220,108 +220,249 @@ def test_instructions_open_with_the_entry_path() -> None: # --------------------------------------------------------------------------- # # Description and instructions text (docs/authoring_fetcher_guide.md). Paramify shows -# both as plain text, so the layout is checked here for each converted fetcher; -# add a category (or category/short_name) once its fetchers are converted. +# both as plain text, so the layout is checked here for every fetcher not on TEXT_FORMAT_PENDING. # --------------------------------------------------------------------------- # -TEXT_FORMAT_FETCHERS = ( - "aws/route53_resolver_dns_security", - "azure/activity_log_alerts", - "azure/ai_hub_configuration", - "azure/aks_cluster_configuration", - "azure/alert_rules", - "azure/api_management_configuration", - "azure/app_gateway_backend_tls", - "azure/app_gateway_tls_listeners", - "azure/app_gateway_waf", - "azure/app_service_backups", - "azure/app_service_configuration", - "azure/app_service_deployments", - "azure/app_service_plans", - "azure/backup_recovery_status", - "azure/backup_vault_protection", - "azure/bastion_configuration", - "azure/change_provenance", - "azure/change_work_items", - "azure/container_registry_configuration", - "azure/container_workloads_configuration", - "azure/cosmosdb_configuration", - "azure/data_services_configuration", - "azure/databricks_workspace_configuration", - "azure/ddos_protection", - "azure/defender_assessments", - "azure/defender_plans", - "azure/defender_regulatory_compliance", - "azure/defender_vulnerability_findings", - "azure/devops_audit_streams", - "azure/devops_identities", - "azure/devops_permissions", - "azure/devops_pipelines", - "azure/devops_service_connections", - "azure/devops_work_items", - "azure/diagnostic_settings", - "azure/disk_encryption_status", - "azure/entra_access_reviews", - "azure/entra_authentication_policy", - "azure/entra_conditional_access_policies", - "azure/entra_diagnostic_settings", - "azure/entra_dormant_identities", - "azure/entra_emergency_accounts", - "azure/entra_groups", - "azure/entra_mfa_status", - "azure/entra_pim_assignments", - "azure/entra_privileged_roles", - "azure/entra_risky_users", - "azure/entra_service_principals", - "azure/entra_sign_ins", - "azure/firewall_configuration", - "azure/front_door_waf_policies", - "azure/function_app_configuration", - "azure/key_vault_configuration", - "azure/key_vault_key_rotation", - "azure/log_analytics_deletion_rights", - "azure/log_analytics_query_audit", - "azure/log_analytics_workspaces", - "azure/log_storage_immutability", - "azure/managed_identity_configuration", - "azure/managed_redis_configuration", - "azure/mysql_configuration", - "azure/network_flow_logs", - "azure/network_security_groups", - "azure/network_traffic_activity", - "azure/pim_activations", - "azure/policy_assignments", - "azure/policy_compliance", - "azure/postgresql_configuration", - "azure/private_dns_zone_configuration", - "azure/private_endpoint_configuration", - "azure/rbac_custom_roles", - "azure/redis_cache_configuration", - "azure/residual_resources", - "azure/resource_diagnostic_settings", - "azure/resource_inventory", - "azure/resource_locks", - "azure/sentinel_analytics_rules", - "azure/sentinel_automation_rules", - "azure/sentinel_incidents", - "azure/sentinel_playbooks", - "azure/service_availability", - "azure/service_health_advisories", - "azure/sql_encryption_status", - "azure/sql_managed_instance_backups", - "azure/sql_managed_instance_configuration", - "azure/sql_managed_instance_encryption", - "azure/sql_server_configuration", - "azure/storage_encryption_status", - "azure/virtual_wan_configuration", - "azure/vm_hardening_status", - "azure/vnet_egress_routing", - "azure/vpn_gateway_configuration", - "azure/waf_ddos_activity", - "azure/zone_redundancy", - "oci", - "qualys", - "sophos", +# Every fetcher is checked except these, which still fail _text_format_problems. Shrink-only: a batch deletes +# its own lines (one per line, so batches don't collide), and a new fetcher is never added here. The frozen +# tests/text_format_baseline.txt is the ceiling this list can't outgrow; never edit it. +TEXT_FORMAT_PENDING = ( + # aws + "aws/access_analyzer_findings", + "aws/acm_certificate_status", + "aws/apigateway_tls_enforcement", + "aws/athena_encryption_status", + "aws/auto_scaling_high_availability", + "aws/backup_recovery_high_availability", + "aws/backup_validation", + "aws/block_storage_encryption_status", + "aws/cloudformation_drift", + "aws/cloudfront_distribution_security", + "aws/cloudtrail_configuration", + "aws/cloudwatch_high_availability", + "aws/codeartifact_encryption_status", + "aws/codebuild_pipeline_config", + "aws/codepipeline_config", + "aws/component_ssl_enforcement_status", + "aws/config_conformance_packs", + "aws/config_inventory", + "aws/config_monitoring", + "aws/database_high_availability", + "aws/detect_new_aws_resource", + "aws/discovery_beta", + "aws/dlm_lifecycle_policies", + "aws/dms_encryption_status", + "aws/documentdb_encryption_status", + "aws/dynamodb_encryption_status", + "aws/dynamodb_pitr_status", + "aws/ebs_snapshot_status", + "aws/ec2_public_exposure", + "aws/ecr_image_scanning", + "aws/efs_encryption_status", + "aws/efs_high_availability", + "aws/eks_high_availability", + "aws/eks_least_privilege", + "aws/elasticache_encryption_status", + "aws/emr_encryption_status", + "aws/firehose_encryption_status", + "aws/fsx_encryption_status", + "aws/glacier_encryption_status", + "aws/global_accelerator_ha", + "aws/glue_encryption_status", + "aws/guard_duty", + "aws/guard_duty_findings", + "aws/iam_identity_center", + "aws/iam_mfa_status", + "aws/iam_password_policy", + "aws/iam_policies", + "aws/iam_roles", + "aws/iam_users_groups", + "aws/inspector_ecr_findings", + "aws/inspector_vulnerability_scanning", + "aws/kafka_encryption_status", + "aws/kinesis_encryption_status", + "aws/kms_key_rotation", + "aws/load_balancer_encryption_status", + "aws/load_balancer_high_availability", + "aws/macie_data_discovery", + "aws/memorydb_encryption_status", + "aws/neptune_encryption_status", + "aws/network_acls", + "aws/network_firewall_rules", + "aws/network_resilience_high_availability", + "aws/opensearch_encryption_status", + "aws/organizations_scp", + "aws/rds_encryption_status", + "aws/rds_tls_configuration", + "aws/redshift_encryption_status", + "aws/resource_inventory", + "aws/route53_high_availability", + "aws/s3_encryption_status", + "aws/sagemaker_encryption_status", + "aws/secrets_manager_rotation", + "aws/security_groups", + "aws/securityhub_status", + "aws/shield_dos_protection", + "aws/sns_encryption_status", + "aws/sqs_encryption_status", + "aws/ssm_patch_compliance", + "aws/transfer_tls_enforcement", + "aws/vpc_flow_logs", + "aws/vpc_network_segmentation", + "aws/waf_all_rules", + "aws/waf_dos_rules", + # betterstack + "betterstack/public_status_page", + # checkov + "checkov/bicep", + "checkov/kubernetes", + "checkov/terraform", + # cornerstone + "cornerstone/incident_response_training", + "cornerstone/role_based_training", + "cornerstone/secure_development_training", + "cornerstone/security_awareness_training", + # crowdstrike + "crowdstrike/detections", + "crowdstrike/filevantage", + "crowdstrike/firewall_policies", + "crowdstrike/hosts", + "crowdstrike/prevention_policies", + "crowdstrike/spotlight_vulnerabilities", + "crowdstrike/zero_trust_assessment", + # datadog + "datadog/agent_hosts", + "datadog/apm_services", + "datadog/containers", + "datadog/incident_timelines", + "datadog/incidents_list", + "datadog/infra_agent_checks", + "datadog/log_archives", + "datadog/log_indexes", + "datadog/log_pipelines", + "datadog/monitors_list", + "datadog/siem_configuration", + "datadog/siem_detection_rules", + "datadog/siem_signals", + # demo + "demo/hello", + "demo/policy_check", + # fedramp_marketplace + "fedramp_marketplace/vendor_certification_status", + "fedramp_marketplace/workspace_vendor_certification_status", + # gcp + "gcp/api_keys_inventory", + "gcp/bigquery_dataset_configuration", + "gcp/cloud_logging_configuration", + "gcp/cloud_sql_backup_configuration", + "gcp/cloud_sql_encryption_status", + "gcp/cloud_sql_network_configuration", + "gcp/cloud_storage_encryption_status", + "gcp/compute_instance_configuration", + "gcp/dns_configuration", + "gcp/firewall_rules", + "gcp/gke_cluster_configuration", + "gcp/iam_custom_roles", + "gcp/iam_policy_bindings", + "gcp/iam_service_accounts", + "gcp/kms_key_configuration", + "gcp/load_balancer_tls_configuration", + "gcp/persistent_disk_encryption_status", + "gcp/secret_manager_configuration", + "gcp/vpc_network_configuration", + # github + "github/access_review", + "github/actions_security", + "github/audit_log", + "github/branch_protection", + "github/code_scanning_alerts", + "github/dependabot_alerts", + "github/deployment_controls", + "github/non_user_credentials", + "github/org_security_settings", + "github/pull_request_checks", + "github/release_integrity", + "github/repository_inventory", + "github/secret_scanning", + "github/significant_change_notifications", + "github/vulnerability_disclosure", + "github/workflow_runs", + # gitlab + "gitlab/ci_cd_pipeline_config", + "gitlab/merge_request_summary", + "gitlab/project_summary", + "gitlab/significant_change_notifications", + # jira + "jira/access_roles", + "jira/audit_records", + "jira/change_requests", + "jira/incident_postmortems", + "jira/remediation_items", + "jira/site_inventory", + # k8s + "k8s/eks_microservice_segmentation", + "k8s/eks_pod_inventory", + "k8s/kubectl_security", + # knowbe4 + "knowbe4/developer_specific_training", + "knowbe4/discovery_beta", + "knowbe4/high_risk_training", + "knowbe4/module_based_summary", + "knowbe4/security_awareness_training", + # okta + "okta/authenticators", + "okta/automated_account_management", + "okta/just_in_time_authorization", + "okta/least_privilege", + "okta/non_user_accounts_authentication", + "okta/passwordless_authentication", + "okta/phishing_resistant_mfa", + "okta/suspicious_activity_management", + # paramify + "paramify/accepted_vulnerabilities", + "paramify/historical_ver_activity", + "paramify/vulnerability_detail_report", + # rippling + "rippling/all_employees", + "rippling/current_employees", + "rippling/devices", + # sentinelone + "sentinelone/activities", + "sentinelone/agents", + "sentinelone/cloud_detection_rules", + "sentinelone/user_config", + "sentinelone/xdr_assets", + # servicenow + "servicenow/cases", + "servicenow/changes", + # splunk + "splunk/alert_delivery", + "splunk/alert_rules", + "splunk/data_inputs", + "splunk/index_activity", + "splunk/index_retention", + "splunk/log_source_freshness", + "splunk/role_index_access", + # web + "web/page_text", + "web/vulnerability_disclosure", + # wiz + "wiz/attack_surface_findings", + "wiz/cloud_configuration_posture", + "wiz/code_findings", + "wiz/container_vulnerabilities", + "wiz/file_integrity_monitoring", + "wiz/host_configuration_posture", + "wiz/infrastructure_vulnerabilities", + "wiz/issues_report", + "wiz/posture_issues", + "wiz/scan_coverage", + "wiz/stig_compliance_report", + "wiz/tenant_security_settings", + "wiz/threat_detections", + "wiz/vulnerability_findings", + # zendesk + "zendesk/help_center_section", ) # Fetchers whose instructions listed summary fields before their tests checked them; each leaves once its tests # call assert_summary_documented, and SUMMARY_DOCS_PENDING_MAX drops with it. A new fetcher calls the fixture instead. @@ -370,7 +511,8 @@ def _matches(yaml_path: Path, prefix: str) -> bool: return rel == prefix or rel.startswith(prefix + "/") -CONVERTED_YAMLS = [y for y in FETCHER_YAMLS if any(_matches(y, p) for p in TEXT_FORMAT_FETCHERS)] +CHECKED_YAMLS = [y for y in FETCHER_YAMLS if _short(y) not in TEXT_FORMAT_PENDING] +PENDING_YAMLS = [y for y in FETCHER_YAMLS if _short(y) in TEXT_FORMAT_PENDING] def _description_problems(label: str, desc: str, name: str) -> list[str]: @@ -445,20 +587,28 @@ def _text_format_problems(yaml_path: Path, spec: dict) -> list[str]: return problems -def test_text_format_fetchers_exist() -> None: - """Every TEXT_FORMAT_FETCHERS entry names real fetchers, so a typo can't switch the check off.""" - assert TEXT_FORMAT_FETCHERS - missing = [p for p in TEXT_FORMAT_FETCHERS if not any(_matches(y, p) for y in FETCHER_YAMLS)] - assert not missing, f"no fetcher matches {missing}" +def test_text_format_pending_names_real_fetchers_within_the_baseline() -> None: + """Pending entries are real fetchers and only ever shrink from the frozen baseline.""" + baseline = set((Path(__file__).parent / "text_format_baseline.txt").read_text().split()) + assert len(set(TEXT_FORMAT_PENDING)) == len(TEXT_FORMAT_PENDING), "TEXT_FORMAT_PENDING lists a fetcher twice" + assert not set(TEXT_FORMAT_PENDING) - baseline, f"not in the baseline, fix the text instead: {sorted(set(TEXT_FORMAT_PENDING) - baseline)}" + missing = set(TEXT_FORMAT_PENDING) - {_short(y) for y in FETCHER_YAMLS} + assert not missing, f"no fetcher matches {sorted(missing)}" -@pytest.mark.parametrize("yaml_path", CONVERTED_YAMLS, ids=_rel_ids(CONVERTED_YAMLS)) -def test_converted_fetcher_text_follows_the_format(yaml_path: Path) -> None: +@pytest.mark.parametrize("yaml_path", CHECKED_YAMLS, ids=_rel_ids(CHECKED_YAMLS)) +def test_fetcher_text_follows_the_format(yaml_path: Path) -> None: spec = yaml.safe_load(yaml_path.read_text()) problems = _text_format_problems(yaml_path, spec) assert not problems, f"{_short(yaml_path)}:\n " + "\n ".join(problems) +def test_pending_fetchers_still_fail_the_format() -> None: + """A pending fetcher whose text now passes must leave TEXT_FORMAT_PENDING.""" + fixed = [_short(y) for y in PENDING_YAMLS if not _text_format_problems(y, yaml.safe_load(y.read_text()))] + assert not fixed, f"remove from TEXT_FORMAT_PENDING, their text now passes: {fixed}" + + def _oversized(spec: dict) -> int | None: n = len((spec.get("evidence_set") or {}).get("instructions") or "") return n if n > MAX_INSTRUCTIONS_CHARS else None diff --git a/tests/text_format_baseline.txt b/tests/text_format_baseline.txt new file mode 100644 index 000000000..398c3fb97 --- /dev/null +++ b/tests/text_format_baseline.txt @@ -0,0 +1,213 @@ +aws/access_analyzer_findings +aws/acm_certificate_status +aws/apigateway_tls_enforcement +aws/athena_encryption_status +aws/auto_scaling_high_availability +aws/backup_recovery_high_availability +aws/backup_validation +aws/block_storage_encryption_status +aws/cloudformation_drift +aws/cloudfront_distribution_security +aws/cloudtrail_configuration +aws/cloudwatch_high_availability +aws/codeartifact_encryption_status +aws/codebuild_pipeline_config +aws/codepipeline_config +aws/component_ssl_enforcement_status +aws/config_conformance_packs +aws/config_inventory +aws/config_monitoring +aws/database_high_availability +aws/detect_new_aws_resource +aws/discovery_beta +aws/dlm_lifecycle_policies +aws/dms_encryption_status +aws/documentdb_encryption_status +aws/dynamodb_encryption_status +aws/dynamodb_pitr_status +aws/ebs_snapshot_status +aws/ec2_public_exposure +aws/ecr_image_scanning +aws/efs_encryption_status +aws/efs_high_availability +aws/eks_high_availability +aws/eks_least_privilege +aws/elasticache_encryption_status +aws/emr_encryption_status +aws/firehose_encryption_status +aws/fsx_encryption_status +aws/glacier_encryption_status +aws/global_accelerator_ha +aws/glue_encryption_status +aws/guard_duty +aws/guard_duty_findings +aws/iam_identity_center +aws/iam_mfa_status +aws/iam_password_policy +aws/iam_policies +aws/iam_roles +aws/iam_users_groups +aws/inspector_ecr_findings +aws/inspector_vulnerability_scanning +aws/kafka_encryption_status +aws/kinesis_encryption_status +aws/kms_key_rotation +aws/load_balancer_encryption_status +aws/load_balancer_high_availability +aws/macie_data_discovery +aws/memorydb_encryption_status +aws/neptune_encryption_status +aws/network_acls +aws/network_firewall_rules +aws/network_resilience_high_availability +aws/opensearch_encryption_status +aws/organizations_scp +aws/rds_encryption_status +aws/rds_tls_configuration +aws/redshift_encryption_status +aws/resource_inventory +aws/route53_high_availability +aws/s3_encryption_status +aws/sagemaker_encryption_status +aws/secrets_manager_rotation +aws/security_groups +aws/securityhub_status +aws/shield_dos_protection +aws/sns_encryption_status +aws/sqs_encryption_status +aws/ssm_patch_compliance +aws/transfer_tls_enforcement +aws/vpc_flow_logs +aws/vpc_network_segmentation +aws/waf_all_rules +aws/waf_dos_rules +betterstack/public_status_page +checkov/bicep +checkov/kubernetes +checkov/terraform +cornerstone/incident_response_training +cornerstone/role_based_training +cornerstone/secure_development_training +cornerstone/security_awareness_training +crowdstrike/detections +crowdstrike/filevantage +crowdstrike/firewall_policies +crowdstrike/hosts +crowdstrike/prevention_policies +crowdstrike/spotlight_vulnerabilities +crowdstrike/zero_trust_assessment +datadog/agent_hosts +datadog/apm_services +datadog/containers +datadog/incident_timelines +datadog/incidents_list +datadog/infra_agent_checks +datadog/log_archives +datadog/log_indexes +datadog/log_pipelines +datadog/monitors_list +datadog/siem_configuration +datadog/siem_detection_rules +datadog/siem_signals +demo/hello +demo/policy_check +fedramp_marketplace/vendor_certification_status +fedramp_marketplace/workspace_vendor_certification_status +gcp/api_keys_inventory +gcp/bigquery_dataset_configuration +gcp/cloud_logging_configuration +gcp/cloud_sql_backup_configuration +gcp/cloud_sql_encryption_status +gcp/cloud_sql_network_configuration +gcp/cloud_storage_encryption_status +gcp/compute_instance_configuration +gcp/dns_configuration +gcp/firewall_rules +gcp/gke_cluster_configuration +gcp/iam_custom_roles +gcp/iam_policy_bindings +gcp/iam_service_accounts +gcp/kms_key_configuration +gcp/load_balancer_tls_configuration +gcp/persistent_disk_encryption_status +gcp/secret_manager_configuration +gcp/vpc_network_configuration +github/access_review +github/actions_security +github/audit_log +github/branch_protection +github/code_scanning_alerts +github/dependabot_alerts +github/deployment_controls +github/non_user_credentials +github/org_security_settings +github/pull_request_checks +github/release_integrity +github/repository_inventory +github/secret_scanning +github/significant_change_notifications +github/vulnerability_disclosure +github/workflow_runs +gitlab/ci_cd_pipeline_config +gitlab/merge_request_summary +gitlab/project_summary +gitlab/significant_change_notifications +jira/access_roles +jira/audit_records +jira/change_requests +jira/incident_postmortems +jira/remediation_items +jira/site_inventory +k8s/eks_microservice_segmentation +k8s/eks_pod_inventory +k8s/kubectl_security +knowbe4/developer_specific_training +knowbe4/discovery_beta +knowbe4/high_risk_training +knowbe4/module_based_summary +knowbe4/security_awareness_training +okta/authenticators +okta/automated_account_management +okta/just_in_time_authorization +okta/least_privilege +okta/non_user_accounts_authentication +okta/passwordless_authentication +okta/phishing_resistant_mfa +okta/suspicious_activity_management +paramify/accepted_vulnerabilities +paramify/historical_ver_activity +paramify/vulnerability_detail_report +rippling/all_employees +rippling/current_employees +rippling/devices +sentinelone/activities +sentinelone/agents +sentinelone/cloud_detection_rules +sentinelone/user_config +sentinelone/xdr_assets +servicenow/cases +servicenow/changes +splunk/alert_delivery +splunk/alert_rules +splunk/data_inputs +splunk/index_activity +splunk/index_retention +splunk/log_source_freshness +splunk/role_index_access +web/page_text +web/vulnerability_disclosure +wiz/attack_surface_findings +wiz/cloud_configuration_posture +wiz/code_findings +wiz/container_vulnerabilities +wiz/file_integrity_monitoring +wiz/host_configuration_posture +wiz/infrastructure_vulnerabilities +wiz/issues_report +wiz/posture_issues +wiz/scan_coverage +wiz/stig_compliance_report +wiz/tenant_security_settings +wiz/threat_detections +wiz/vulnerability_findings +zendesk/help_center_section From 4781cf7967db0fca1a459deef4d47699ab4524cf Mon Sep 17 00:00:00 2001 From: Connor Dalton Date: Fri, 13 Nov 2026 11:26:38 -0700 Subject: [PATCH 2/4] Pin the text-format baseline and tighten the three fetchers' instructions Hash-pin the frozen baseline in the subset test, drop API versions and vague back-references from the AI services instructions, state null when a list is unread on every affected field, add the percentage formula to the app registrations fetcher, name audit logs in the sign-in logs description, and remove the unused prefix matcher. Co-Authored-By: Claude Sonnet 5.5 --- .../ai_services_configuration/fetcher.yaml | 106 +++++++++--------- .../entra_app_registrations/fetcher.yaml | 2 +- .../azure/entra_sign_in_logs/fetcher.yaml | 2 +- tests/test_contracts.py | 16 +-- 4 files changed, 64 insertions(+), 62 deletions(-) diff --git a/fetchers/azure/ai_services_configuration/fetcher.yaml b/fetchers/azure/ai_services_configuration/fetcher.yaml index 228c4790e..42298cb11 100644 --- a/fetchers/azure/ai_services_configuration/fetcher.yaml +++ b/fetchers/azure/ai_services_configuration/fetcher.yaml @@ -77,8 +77,8 @@ evidence_set: - providers.get("Microsoft.CognitiveServices") — tells "not registered" from "no accounts" - CognitiveServicesManagementClient accounts.list() — every Azure AI services account (Foundry, OpenAI, Language, Speech, Document Intelligence and other kinds) with its network, key, encryption and identity settings - deployments.list(), rai_policies.list() and projects.list() (per Foundry or OpenAI account) — model deployments, content filter policies and Foundry projects - - GET {account}/connections and {project}/connections (ARM REST, api-version 2025-06-01; per account and per project) — connections agents use to reach other systems, recorded without credentials or query strings - - GET /subscriptions/{id}/providers/Microsoft.Search/searchServices (ARM REST, api-version 2025-05-01) — every AI Search service with its network, key and encryption settings + - GET {account}/connections and {project}/connections (ARM REST; per account and per project) — connections agents use to reach other systems, recorded without credentials or query strings + - GET /subscriptions/{id}/providers/Microsoft.Search/searchServices (ARM REST) — every AI Search service with its network, key and encryption settings - diagnostic_settings.list() (per account and search service) — whether audit logs are exported Output: azure_ai_services_configuration_.json — each AI services account, search service, model deployment, content filter policy, project and connection with the verdicts below. Resources the resource_exclusions patterns take out are listed under excluded instead. @@ -87,76 +87,76 @@ evidence_set: What was checked? - total_accounts — Number of AI services accounts checked; null when the account list could not be read - - accounts_by_kind — Number of accounts for each kind of account (AIServices, OpenAI, and so on), with one entry per kind found + - accounts_by_kind — Number of accounts for each kind of account (AIServices, OpenAI, and so on), with one entry per kind found; null when the account list could not be read - total_search_services — Number of AI Search services checked; null when the search service list could not be read - provider_registration_status — Whether the AI services provider (Microsoft.CognitiveServices) is registered on the subscription: registered, not_registered or unknown Which resources are left out of scope? - resource_exclusion_patterns — List of the configured resource_exclusions patterns (rg:, location: or id:); empty when none are set, so every resource is in scope - resource_exclusion_patterns_unmatched — List of configured patterns that matched no account or search service, such as a mistyped name; null when either list could not be read - - excluded_accounts — Number of accounts left out of scope because a pattern matched their resource group, region or id; none of their settings are read and no other count includes them - - excluded_search_services — Number of search services left out of scope for the same reason - - in_scope_accounts — Number of accounts read and in scope; 0 means the patterns took out every account, which a check can catch by requiring more than 0 - - in_scope_search_services — Number of search services read and in scope; 0 means the patterns took out every one + - excluded_accounts — Number of accounts left out of scope because a pattern matched their resource group, region or id; none of their settings are read and no other count includes them; null when the account list could not be read + - excluded_search_services — Number of search services left out of scope because a pattern matched their resource group, region or id; null when the search service list could not be read + - in_scope_accounts — Number of accounts read and in scope; 0 means the patterns took out every account, which a check can catch by requiring more than 0; null when the account list could not be read + - in_scope_search_services — Number of search services read and in scope; 0 means the patterns took out every one; null when the search service list could not be read Can the accounts be reached from the internet? - - accounts_open_to_all_networks — Number of accounts reachable from any network (public access is on and the network rules allow all) - - accounts_network_restricted — Number of accounts with public access on but the network rules denying by default - - accounts_private_only — Number of accounts with public network access disabled - - accounts_with_private_endpoints — Number of accounts with at least one private endpoint connection - - accounts_with_private_endpoint_but_public_access — Number of accounts that have a private endpoint but can still be reached publicly - - accounts_with_outbound_restricted — Number of accounts that restrict their own outbound network access - - search_services_open_to_all_networks — Number of search services reachable from any network (public access is on and no IP rules are set) + - accounts_open_to_all_networks — Number of accounts reachable from any network (public access is on and the network rules allow all); null when the account list could not be read + - accounts_network_restricted — Number of accounts with public access on but the network rules denying by default; null when the account list could not be read + - accounts_private_only — Number of accounts with public network access disabled; null when the account list could not be read + - accounts_with_private_endpoints — Number of accounts with at least one private endpoint connection; null when the account list could not be read + - accounts_with_private_endpoint_but_public_access — Number of accounts that have a private endpoint but can still be reached publicly; null when the account list could not be read + - accounts_with_outbound_restricted — Number of accounts that restrict their own outbound network access; null when the account list could not be read + - search_services_open_to_all_networks — Number of search services reachable from any network (public access is on and no IP rules are set); null when the search service list could not be read Do keys still work, or only Microsoft Entra ID sign-in? - - accounts_with_api_keys_enabled — Number of accounts where API keys still work (local authentication is not disabled) - - accounts_entra_only — Number of accounts where API keys are disabled so only Microsoft Entra ID sign-in works - - accounts_entra_only_percentage — Percentage of accounts that accept only Entra ID sign-in = accounts_entra_only / total_accounts × 100 (a whole number, 0–100); 0 when total_accounts is 0 - - accounts_without_managed_identity — Number of accounts with no managed identity - - search_services_with_api_keys_enabled — Number of search services where API keys still work + - accounts_with_api_keys_enabled — Number of accounts where API keys still work (local authentication is not disabled); null when the account list could not be read + - accounts_entra_only — Number of accounts where API keys are disabled so only Microsoft Entra ID sign-in works; null when the account list could not be read + - accounts_entra_only_percentage — Percentage of accounts that accept only Entra ID sign-in = accounts_entra_only / total_accounts × 100 (a whole number, 0–100); 0 when total_accounts is 0; null when the account list could not be read + - accounts_without_managed_identity — Number of accounts with no managed identity; null when the account list could not be read + - search_services_with_api_keys_enabled — Number of search services where API keys still work; null when the search service list could not be read Is data encrypted with a customer-managed key? - - accounts_with_customer_managed_key — Number of accounts encrypted with a key the customer manages instead of the Microsoft-managed key - - search_services_cmk_enforced — Number of search services that enforce customer-managed key encryption + - accounts_with_customer_managed_key — Number of accounts encrypted with a key the customer manages instead of the Microsoft-managed key; null when the account list could not be read + - search_services_cmk_enforced — Number of search services that enforce customer-managed key encryption; null when the search service list could not be read Are audit logs exported? - - accounts_with_diagnostic_settings — Number of accounts with at least one diagnostic setting - - accounts_audit_logs_exported — Number of accounts whose diagnostic setting sends audit logs to a destination - - accounts_audit_logs_unknown — Number of accounts whose diagnostic settings could not be read - - search_services_audit_logs_exported — Number of search services whose diagnostic setting sends operation logs to a destination + - accounts_with_diagnostic_settings — Number of accounts with at least one diagnostic setting; null when the account list could not be read + - accounts_audit_logs_exported — Number of accounts whose diagnostic setting sends audit logs to a destination; null when the account list could not be read + - accounts_audit_logs_unknown — Number of accounts whose diagnostic settings could not be read; null when the account list could not be read + - search_services_audit_logs_exported — Number of search services whose diagnostic setting sends operation logs to a destination; null when the search service list could not be read Where may model deployments process data, and who publishes the models? - - total_deployments — Number of model deployments across all accounts - - deployments_global_processing — Number of deployments that may process data in any Azure region (Global deployment types) - - deployments_data_zone_processing — Number of deployments that process data within one geography (DataZone deployment types) - - deployments_regional_processing — Number of deployments that process data in the account's own region (Standard, ProvisionedManaged or Batch types) - - deployments_processing_scope_unknown — Number of deployments whose deployment type is not recognised - - deployments_third_party_publisher — Number of deployments whose model publisher is neither OpenAI nor Microsoft - - deployments_by_model_publisher — Number of deployments for each model publisher (OpenAI, Anthropic and so on; unknown when none is reported), with one entry per publisher found - - deployments_with_automatic_version_upgrade — Number of deployments whose model is upgraded automatically when a new default version is available or the current one expires + - total_deployments — Number of model deployments across all accounts; null when the account list could not be read + - deployments_global_processing — Number of deployments that may process data in any Azure region (Global deployment types); null when the account list could not be read + - deployments_data_zone_processing — Number of deployments that process data within one geography (DataZone deployment types); null when the account list could not be read + - deployments_regional_processing — Number of deployments that process data in the account's own region (Standard, ProvisionedManaged or Batch types); null when the account list could not be read + - deployments_processing_scope_unknown — Number of deployments whose deployment type is not recognised; null when the account list could not be read + - deployments_third_party_publisher — Number of deployments whose model publisher is neither OpenAI nor Microsoft; null when the account list could not be read + - deployments_by_model_publisher — Number of deployments for each model publisher (OpenAI, Anthropic and so on), with one entry per publisher found; a deployment with no publisher reported is counted under unknown; null when the account list could not be read + - deployments_with_automatic_version_upgrade — Number of deployments whose model is upgraded automatically when a new default version is available or the current one expires; null when the account list could not be read Do deployments run under a content filter? - - deployments_content_filter_blocking — Number of deployments whose content filter policy has the hate, sexual, violence and self-harm filters enabled and blocking - - deployments_content_filter_non_blocking — Number of deployments whose policy has any of those filters off or annotate-only - - deployments_content_filter_none — Number of deployments that name no content filter policy - - deployments_content_filter_missing — Number of deployments that name a policy that is not on the account - - deployments_content_filter_unknown — Number of deployments whose account's policies could not be listed - - deployments_jailbreak_protection — Number of deployments whose policy blocks jailbreak attempts (Prompt Shields) - - deployments_indirect_attack_protection — Number of deployments whose policy blocks indirect attacks hidden in documents or tool output (Prompt Shields) - - deployments_asynchronous_filtering — Number of deployments whose policy lets output stream before filtering completes + - deployments_content_filter_blocking — Number of deployments whose content filter policy has the hate, sexual, violence and self-harm filters enabled and blocking; null when the account list could not be read + - deployments_content_filter_non_blocking — Number of deployments whose content filter policy lacks a hate, sexual, violence or self-harm filter, or has one off or annotate-only; null when the account list could not be read + - deployments_content_filter_none — Number of deployments that name no content filter policy; null when the account list could not be read + - deployments_content_filter_missing — Number of deployments that name a policy that is not on the account; null when the account list could not be read + - deployments_content_filter_unknown — Number of deployments whose account's policies could not be listed; null when the account list could not be read + - deployments_jailbreak_protection — Number of deployments whose policy blocks jailbreak attempts (Prompt Shields); null when the account list could not be read + - deployments_indirect_attack_protection — Number of deployments whose policy blocks indirect attacks hidden in documents or tool output (Prompt Shields); null when the account list could not be read + - deployments_asynchronous_filtering — Number of deployments whose policy lets output stream before filtering completes; null when the account list could not be read Are custom content filter policies weaker than Microsoft's default? - - content_filter_policies_custom — Number of content filter policies the customer manages - - content_filter_policies_weaker_than_default — Number of custom policies that do not block every harm category and jailbreaks, or set a severity threshold looser than Microsoft's default (a threshold is the lowest severity blocked, so High is looser than Medium) - - content_filter_policies_weaker_unknown — Number of policies whose thresholds are missing or unrecognised, so the comparison is unknown - - content_filter_policies_custom_unused — Number of custom policies that no deployment uses + - content_filter_policies_custom — Number of content filter policies the customer manages; null when the account list could not be read + - content_filter_policies_weaker_than_default — Number of custom policies that do not block every harm category and jailbreaks, or set a severity threshold looser than Microsoft's default (a threshold is the lowest severity blocked, so High is looser than Medium); null when the account list could not be read + - content_filter_policies_weaker_unknown — Number of custom policies whose thresholds are missing or unrecognised, so the comparison is unknown; null when the account list could not be read + - content_filter_policies_custom_unused — Number of custom policies that no deployment uses; null when the account list could not be read What can agents reach, and with what credentials? - - total_projects — Number of Foundry projects across all accounts - - total_connections — Number of connections across all accounts and projects (an account list that repeats a project's connection counts it once) - - connections_using_stored_secrets — Number of connections that keep a credential (key, token, password, SAS or service principal secret) instead of using Entra ID or a managed identity - - connections_unauthenticated — Number of connections that use no authentication - - connections_auth_type_unrecognized — Number of connections whose authentication type is not recognised, so whether they hold a secret is unknown - - connections_unknown_accounts — Number of Foundry or OpenAI accounts whose connections could not be read - - remote_tool_connections — Number of connections that are tools or MCP servers agents can call - - grounding_with_bing_connections — Number of Grounding with Bing Search connections, which send queries outside the Azure compliance boundary + - total_projects — Number of Foundry projects across all accounts; null when the account list could not be read + - total_connections — Number of connections across all accounts and projects (an account list that repeats a project's connection counts it once); null when the account list could not be read + - connections_using_stored_secrets — Number of connections that keep a credential (key, token, password, SAS or service principal secret) instead of using Entra ID or a managed identity; null when the account list could not be read + - connections_unauthenticated — Number of connections that use no authentication; null when the account list could not be read + - connections_auth_type_unrecognized — Number of connections whose authentication type is not recognised, so whether they hold a secret is unknown; null when the account list could not be read + - connections_unknown_accounts — Number of Foundry or OpenAI accounts whose connections could not be read; null when the account list could not be read + - remote_tool_connections — Number of connections that are tools or MCP servers agents can call; null when the account list could not be read + - grounding_with_bing_connections — Number of Grounding with Bing Search connections, which send queries outside the Azure compliance boundary; null when the account list could not be read diff --git a/fetchers/azure/entra_app_registrations/fetcher.yaml b/fetchers/azure/entra_app_registrations/fetcher.yaml index 78922b801..da013b253 100644 --- a/fetchers/azure/entra_app_registrations/fetcher.yaml +++ b/fetchers/azure/entra_app_registrations/fetcher.yaml @@ -95,7 +95,7 @@ evidence_set: - credentials_without_expiry — Number of credentials with no end date; null if the applications read failed or stopped part-way - healthy_credentials — Number of credentials that have an end date, haven't expired and don't expire within rotation_threshold_days days; null if the applications read failed or stopped part-way - rotation_threshold_days — Days before expiry at which a credential counts as expiring soon (30) - - credential_rotation_compliance_percentage — Percentage of credentials that are healthy; 0 when there are none; null if the applications read failed or stopped part-way + - credential_rotation_compliance_percentage — Percentage of credentials that are healthy = healthy_credentials / total_credentials × 100 (a whole number, 0–100); 0 when total_credentials is 0; null if the applications read failed or stopped part-way - apps_with_expired_credentials — Number of app registrations holding an expired credential; null if the applications read failed or stopped part-way - apps_with_credentials_expiring_soon — Number of app registrations holding a credential expiring soon; null if the applications read failed or stopped part-way - apps_with_credentials_without_expiry — Number of app registrations holding a credential with no end date; null if the applications read failed or stopped part-way diff --git a/fetchers/azure/entra_sign_in_logs/fetcher.yaml b/fetchers/azure/entra_sign_in_logs/fetcher.yaml index 6561d1671..e14302e75 100644 --- a/fetchers/azure/entra_sign_in_logs/fetcher.yaml +++ b/fetchers/azure/entra_sign_in_logs/fetcher.yaml @@ -1,6 +1,6 @@ name: azure_entra_sign_in_logs version: 0.3.0 -description: Shows from Entra sign-in logs whether successful sign-ins required MFA and phishing-resistant methods, what changed in authentication and Conditional Access policies, and counts of non-interactive and service-principal sign-ins. +description: Shows from Entra sign-in and audit logs whether successful sign-ins required MFA and phishing-resistant methods, what changed in authentication and Conditional Access policies, and counts of non-interactive and service-principal sign-ins. category: azure supports_targets: true diff --git a/tests/test_contracts.py b/tests/test_contracts.py index 264821f7f..cdee19061 100644 --- a/tests/test_contracts.py +++ b/tests/test_contracts.py @@ -20,6 +20,7 @@ from __future__ import annotations import ast +import hashlib import json import re from pathlib import Path @@ -224,7 +225,8 @@ def test_instructions_open_with_the_entry_path() -> None: # --------------------------------------------------------------------------- # # Every fetcher is checked except these, which still fail _text_format_problems. Shrink-only: a batch deletes -# its own lines (one per line, so batches don't collide), and a new fetcher is never added here. The frozen +# only its own lines and keeps every "# " comment, and a new fetcher is never added here. Parallel +# batches take runs that don't touch; if a merge conflicts, keep both deletions. The frozen # tests/text_format_baseline.txt is the ceiling this list can't outgrow; never edit it. TEXT_FORMAT_PENDING = ( # aws @@ -506,11 +508,6 @@ def test_instructions_open_with_the_entry_path() -> None: COMMAND_LINE = re.compile(r"^- \S.* — \S") -def _matches(yaml_path: Path, prefix: str) -> bool: - rel = _short(yaml_path) - return rel == prefix or rel.startswith(prefix + "/") - - CHECKED_YAMLS = [y for y in FETCHER_YAMLS if _short(y) not in TEXT_FORMAT_PENDING] PENDING_YAMLS = [y for y in FETCHER_YAMLS if _short(y) in TEXT_FORMAT_PENDING] @@ -587,9 +584,14 @@ def _text_format_problems(yaml_path: Path, spec: dict) -> list[str]: return problems +BASELINE_SHA256 = "92d09c675c741fc09b56bb09fc95c4838103d2e74c3aa46417aebbbea02442b1" + + def test_text_format_pending_names_real_fetchers_within_the_baseline() -> None: """Pending entries are real fetchers and only ever shrink from the frozen baseline.""" - baseline = set((Path(__file__).parent / "text_format_baseline.txt").read_text().split()) + baseline_file = Path(__file__).parent / "text_format_baseline.txt" + assert hashlib.sha256(baseline_file.read_bytes()).hexdigest() == BASELINE_SHA256, "text_format_baseline.txt is frozen" + baseline = set(baseline_file.read_text().split()) assert len(set(TEXT_FORMAT_PENDING)) == len(TEXT_FORMAT_PENDING), "TEXT_FORMAT_PENDING lists a fetcher twice" assert not set(TEXT_FORMAT_PENDING) - baseline, f"not in the baseline, fix the text instead: {sorted(set(TEXT_FORMAT_PENDING) - baseline)}" missing = set(TEXT_FORMAT_PENDING) - {_short(y) for y in FETCHER_YAMLS} From 6f3771887bd0de08904b4a1b8f457e9a74e72a58 Mon Sep 17 00:00:00 2001 From: Connor Dalton Date: Fri, 13 Nov 2026 11:28:35 -0700 Subject: [PATCH 3/4] Say a deployment with no publisher is not a third-party one Co-Authored-By: Claude Sonnet 5.5 --- fetchers/azure/ai_services_configuration/fetcher.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fetchers/azure/ai_services_configuration/fetcher.yaml b/fetchers/azure/ai_services_configuration/fetcher.yaml index 42298cb11..07502acd9 100644 --- a/fetchers/azure/ai_services_configuration/fetcher.yaml +++ b/fetchers/azure/ai_services_configuration/fetcher.yaml @@ -131,7 +131,7 @@ evidence_set: - deployments_data_zone_processing — Number of deployments that process data within one geography (DataZone deployment types); null when the account list could not be read - deployments_regional_processing — Number of deployments that process data in the account's own region (Standard, ProvisionedManaged or Batch types); null when the account list could not be read - deployments_processing_scope_unknown — Number of deployments whose deployment type is not recognised; null when the account list could not be read - - deployments_third_party_publisher — Number of deployments whose model publisher is neither OpenAI nor Microsoft; null when the account list could not be read + - deployments_third_party_publisher — Number of deployments whose model publisher is neither OpenAI nor Microsoft; a deployment with no publisher reported is not counted; null when the account list could not be read - deployments_by_model_publisher — Number of deployments for each model publisher (OpenAI, Anthropic and so on), with one entry per publisher found; a deployment with no publisher reported is counted under unknown; null when the account list could not be read - deployments_with_automatic_version_upgrade — Number of deployments whose model is upgraded automatically when a new default version is available or the current one expires; null when the account list could not be read From 11fd306352048b9f26fcb8443dfeb94576d3de0b Mon Sep 17 00:00:00 2001 From: Bruce Weygandt Date: Tue, 17 Nov 2026 13:33:12 -0700 Subject: [PATCH 4/4] docs(azure_ai_services_configuration): restore the per-account unread-list rule and the default-policy note The rewrite dropped two sentences. A per-account list that could not be read is null on that account and sets metadata.partial_failure, and the summary counts built from such lists then cover only the accounts whose list was read (with two accounts and one deployment list failing, total_deployments is 1). Microsoft's default policy blocks jailbreaks but does not enable indirect-attack detection. Co-Authored-By: Claude Opus 5.5 --- fetchers/azure/ai_services_configuration/fetcher.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fetchers/azure/ai_services_configuration/fetcher.yaml b/fetchers/azure/ai_services_configuration/fetcher.yaml index 07502acd9..694d805e7 100644 --- a/fetchers/azure/ai_services_configuration/fetcher.yaml +++ b/fetchers/azure/ai_services_configuration/fetcher.yaml @@ -81,7 +81,7 @@ evidence_set: - GET /subscriptions/{id}/providers/Microsoft.Search/searchServices (ARM REST) — every AI Search service with its network, key and encryption settings - diagnostic_settings.list() (per account and search service) — whether audit logs are exported - Output: azure_ai_services_configuration_.json — each AI services account, search service, model deployment, content filter policy, project and connection with the verdicts below. Resources the resource_exclusions patterns take out are listed under excluded instead. + Output: azure_ai_services_configuration_.json — each AI services account, search service, model deployment, content filter policy, project and connection with the verdicts below. Resources the resource_exclusions patterns take out are listed under excluded instead. A per-account list (diagnostic settings, deployments, content filter policies, projects or connections) that could not be read is null on that account and sets metadata.partial_failure; the summary counts built from such lists then cover only the accounts whose list was read. Summary fields: @@ -142,7 +142,7 @@ evidence_set: - deployments_content_filter_missing — Number of deployments that name a policy that is not on the account; null when the account list could not be read - deployments_content_filter_unknown — Number of deployments whose account's policies could not be listed; null when the account list could not be read - deployments_jailbreak_protection — Number of deployments whose policy blocks jailbreak attempts (Prompt Shields); null when the account list could not be read - - deployments_indirect_attack_protection — Number of deployments whose policy blocks indirect attacks hidden in documents or tool output (Prompt Shields); null when the account list could not be read + - deployments_indirect_attack_protection — Number of deployments whose policy blocks indirect attacks hidden in documents or tool output (Prompt Shields); Microsoft's default policy blocks jailbreaks but does not enable indirect-attack detection; null when the account list could not be read - deployments_asynchronous_filtering — Number of deployments whose policy lets output stream before filtering completes; null when the account list could not be read Are custom content filter policies weaker than Microsoft's default?