chore(release): v3.7.0

.gitattributes

@@ -0,0 +1,3 @@
+contracts/*.json text eol=lf
+contracts/*.sha256 text eol=lf
+config/search/tests/test_foundry_iq_templates.py -text whitespace=cr-at-eol

CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## [v3.7.0] - 2026-07-21
+
+### User and operator impact
+
+GPT-RAG `v3.7.0` delivers the governance baseline tracked in [issue #571](https://github.com/Azure/GPT-RAG/issues/571) through [implementation PR #573](https://github.com/Azure/GPT-RAG/pull/573) and [documentation PR #574](https://github.com/Azure/GPT-RAG/pull/574), with [orchestrator `v3.8.0`](https://github.com/Azure/gpt-rag-orchestrator/releases/tag/v3.8.0) and [ingestion `v2.5.0`](https://github.com/Azure/gpt-rag-ingestion/releases/tag/v2.5.0) deployed as one pinned combination. Metadata audit events and ingestion provenance remain disabled by default, sensitive-content capture remains independently disabled, and existing deployments do not change behavior until an operator opts in.
+
+The shared `audit-event-v1` contract is pinned to logical hash `825db8ef40a81e2c19e5d80d37c565b6b47fc9a6540e9881d35cc12b8fde5aab` and Application Insights wire hash `066c8f5408610ab839d5121d06ca5bc59e8797e551d5c47c875c5ba52f7e0588`. These audit events provide technical evidence for adopter governance and incident response; they do not establish legal or regulatory compliance.
+
+### Added
+
+- **Safe governance defaults in App Configuration.** Fresh and upgraded deployments seed missing keys as `AUDIT_EVENTS_ENABLED=false`, `AUDIT_SENSITIVE_CONTENT_ENABLED=false`, `AUDIT_SENSITIVE_CONTENT_FIELDS=''`, `AUDIT_ACTOR_PSEUDONYM_ENABLED=false`, `AUDIT_SOURCE_EVENT_LIMIT=25`, `AUDIT_HMAC_KEY_ID=v1`, `AUDIT_ADDITIONAL_REDACTED_KEYS=''`, `INGESTION_PROVENANCE_ENABLED=false`, `INGESTION_REQUIRE_GOVERNANCE_METADATA=false`, `INGESTION_DEFAULT_CLASSIFICATION=unclassified`, and `INGESTION_DEFAULT_RIGHT_TO_USE=not_asserted`. Reprovisioning preserves operator-managed values unless the corresponding AZD environment value explicitly overrides them.
+- **Key Vault-backed audit pseudonymization key.** When `AUDIT_HMAC_KEY` is absent, post-provisioning creates `AUDIT-HMAC-KEY` from 256 cryptographically random bits and registers only its Key Vault reference in App Configuration. Later runs reuse it; an existing operator-managed Key Vault reference is preserved, and a valid legacy plaintext key is migrated without changing its value. The value is not placed in plaintext deployment configuration, output, or logs. Rotation is explicit: create a new Key Vault secret version and advance `AUDIT_HMAC_KEY_ID`.
+- **Additive provenance fields in the RAG Search index.** The index adds `provenance_id`, `source_uri_id`, `source_version_id`, `content_checksum_sha256`, `ingested_at`, `ingest_run_id`, `data_classification`, `right_to_use`, `retention_class`, and `delete_after`. Identifier/checksum fields are retrievable and filterable; guaranteed UTC `ingested_at` values use sortable `Edm.DateTimeOffset`; `delete_after` remains a sortable string because ingestion v2.5.0 intentionally passes through operator policy values without imposing a date format; governance classification fields are facetable.
+- **Shared contract source.** `contracts/` now carries the byte-pinned logical and Application Insights wire schemas consumed by orchestrator v3.8.0 and ingestion v2.5.0.
+
+### Changed
+
+- **Release manifest.** `manifest.json` identifies the umbrella `v3.7.0` combination and pins orchestrator `v3.8.0` plus ingestion `v2.5.0`; UI `v2.3.13` and infra `v2.3.0` remain unchanged.
+- **Non-destructive existing-index migration.** Search post-provisioning no longer deletes an index before updating it. Missing fields are merged into the current definition and submitted with an in-place `PUT`; operator-owned fields and existing documents are preserved. An incompatible same-name field fails before any update, and there is no delete/recreate fallback.
+
+### Security and operational guidance
+
+Keep `AUDIT_SENSITIVE_CONTENT_ENABLED=false` unless an explicit, reviewed capture policy is in place. Tokens, credentials, authorization material, cookies, connection strings, and configured redacted keys remain prohibited even when sensitive capture is enabled. The audit HMAC secret is intended for orchestrator pseudonymization only; ingestion does not consume it.
+
+`delete_after` records retention policy intent only. GPT-RAG does not schedule or perform automatic deletion from this field. Operators remain responsible for retention enforcement, confirmed deletion, telemetry access, and export controls.
+
+Deployments that explicitly disable Key Vault still receive the disabled governance defaults, but no audit HMAC key/reference is created. Audit events must remain disabled in that deployment mode.
+
+### Migration and rollback
+
+Upgrade the umbrella configuration and both pinned runtime components together, run post-provisioning to register the Key Vault reference and additive Search fields, verify metadata-only events in a non-production environment, and only then enable `AUDIT_EVENTS_ENABLED` or `INGESTION_PROVENANCE_ENABLED`.
+
+Rollback by setting both feature gates to `false` and redeploying prior component tags if needed. The additive Search fields can remain in place because older components ignore unknown fields. Post-provisioning never recreates an existing index for this migration.
+
+### Validation
+
+| Component | Version |
+| --- | --- |
+| gpt-rag-ui | v2.3.13 |
+| gpt-rag-orchestrator | v3.8.0 |
+| gpt-rag-ingestion | v2.5.0 |
+| infra / AI Landing Zone | v2.3.0 |
+
+- GPT-RAG governance and Search suites: 131 tests passed.
+- Bicep compilation, template rendering, static RBAC checks, manifest and component-release checks, and create-only Azure previews passed.
+- Runtime provisioning could not complete. Azure platform capacity blocked Azure AI Search in `eastus2` and the Container Apps managed environment in `swedencentral`; the Sweden Central attempt also exposed an ARM nested deployment name over the 64-character limit. No runtime endpoints, application health checks, live governance defaults, Search migration result, or Application Insights/KQL evidence were obtained.
+
 ## [v3.6.0] - 2026-07-19
 
 ### User and operator impact

config/governance/__init__.py

@@ -0,0 +1 @@
+"""Governance and audit deployment configuration."""

config/governance/setup.py

@@ -0,0 +1,261 @@
+#!/usr/bin/env python3
+"""Seed safe governance defaults and the audit pseudonymization key reference."""
+
+from __future__ import annotations
+
+import base64
+import hmac
+import json
+import logging
+import os
+import secrets
+from collections.abc import Mapping
+from typing import Any
+
+from azure.appconfiguration import AzureAppConfigurationClient, ConfigurationSetting
+from azure.core.exceptions import ResourceNotFoundError
+from azure.identity import (
+    AzureCliCredential,
+    ChainedTokenCredential,
+    ManagedIdentityCredential,
+)
+from azure.keyvault.secrets import SecretClient
+
+
+LABEL = "gpt-rag"
+AUDIT_HMAC_CONFIG_KEY = "AUDIT_HMAC_KEY"
+AUDIT_HMAC_SECRET_NAME = "AUDIT-HMAC-KEY"
+KEY_VAULT_REFERENCE_CONTENT_TYPE = (
+    "application/vnd.microsoft.appconfig.keyvaultref+json;charset=utf-8"
+)
+
+DEFAULT_SETTINGS: Mapping[str, str] = {
+    "AUDIT_EVENTS_ENABLED": "false",
+    "AUDIT_SENSITIVE_CONTENT_ENABLED": "false",
+    "AUDIT_SENSITIVE_CONTENT_FIELDS": "",
+    "AUDIT_ACTOR_PSEUDONYM_ENABLED": "false",
+    "AUDIT_SOURCE_EVENT_LIMIT": "25",
+    "AUDIT_HMAC_KEY_ID": "v1",
+    "AUDIT_ADDITIONAL_REDACTED_KEYS": "",
+    "INGESTION_PROVENANCE_ENABLED": "false",
+    "INGESTION_REQUIRE_GOVERNANCE_METADATA": "false",
+    "INGESTION_DEFAULT_CLASSIFICATION": "unclassified",
+    "INGESTION_DEFAULT_RIGHT_TO_USE": "not_asserted",
+}
+
+
+def create_credential() -> ChainedTokenCredential:
+    return ChainedTokenCredential(
+        AzureCliCredential(process_timeout=30),
+        ManagedIdentityCredential(process_timeout=30),
+    )
+
+
+def generate_audit_hmac_key() -> str:
+    """Return a Base64URL encoding of exactly 256 random bits."""
+    return base64.urlsafe_b64encode(secrets.token_bytes(32)).decode("ascii").rstrip("=")
+
+
+def _decode_audit_hmac_key(value: str) -> bytes:
+    candidate = value.strip()
+    if len(candidate) == 64:
+        try:
+            decoded = bytes.fromhex(candidate)
+        except ValueError:
+            decoded = b""
+        if len(decoded) == 32:
+            return decoded
+
+    padded = candidate + ("=" * (-len(candidate) % 4))
+    try:
+        decoded = base64.urlsafe_b64decode(padded.encode("ascii"))
+    except (ValueError, UnicodeEncodeError):
+        decoded = b""
+    if len(decoded) != 32:
+        raise ValueError(
+            "The existing AUDIT-HMAC-KEY secret is not an encoding of exactly "
+            "32 bytes. Rotate it explicitly before enabling audit events."
+        )
+    return decoded
+
+
+def ensure_audit_hmac_secret(
+    secret_client: Any,
+    initial_value: str | None = None,
+) -> bool:
+    """Create the key once and return whether a new secret version was written."""
+    try:
+        current = secret_client.get_secret(AUDIT_HMAC_SECRET_NAME)
+    except ResourceNotFoundError:
+        value = initial_value or generate_audit_hmac_key()
+        _decode_audit_hmac_key(value)
+        secret_client.set_secret(
+            AUDIT_HMAC_SECRET_NAME,
+            value,
+            content_type="application/octet-stream",
+        )
+        return True
+
+    current_key = _decode_audit_hmac_key(current.value)
+    if initial_value is not None:
+        initial_key = _decode_audit_hmac_key(initial_value)
+        if not hmac.compare_digest(current_key, initial_key):
+            raise ValueError(
+                "The existing AUDIT-HMAC-KEY does not match the operator-managed "
+                "AUDIT_HMAC_KEY value. Resolve the conflict explicitly."
+            )
+    return False
+
+
+def key_vault_reference(vault_uri: str) -> str:
+    uri = f"{vault_uri.rstrip('/')}/secrets/{AUDIT_HMAC_SECRET_NAME}"
+    return json.dumps({"uri": uri}, separators=(",", ":"))
+
+
+def resolve_vault_uri(environ: Mapping[str, str] = os.environ) -> str:
+    configured_uri = environ.get("KEY_VAULT_URI", "").strip()
+    if configured_uri:
+        return configured_uri.rstrip("/") + "/"
+
+    vault_name = environ.get("KEY_VAULT_NAME", "").strip()
+    if vault_name:
+        return f"https://{vault_name}.vault.azure.net/"
+
+    raise RuntimeError(
+        "KEY_VAULT_URI or KEY_VAULT_NAME is required to provision the audit "
+        "HMAC key reference."
+    )
+
+
+def seed_governance_settings(
+    app_config_client: Any,
+    environ: Mapping[str, str] = os.environ,
+) -> None:
+    """Seed missing defaults while preserving operator-managed values."""
+    for key, default in DEFAULT_SETTINGS.items():
+        if key in environ:
+            value = environ[key]
+        else:
+            try:
+                app_config_client.get_configuration_setting(key=key, label=LABEL)
+            except ResourceNotFoundError:
+                value = default
+            else:
+                continue
+
+        app_config_client.set_configuration_setting(
+            ConfigurationSetting(
+                key=key,
+                label=LABEL,
+                value=value,
+                content_type="text/plain",
+            )
+        )
+
+
+def get_configuration_setting_or_none(app_config_client: Any, key: str) -> Any:
+    try:
+        return app_config_client.get_configuration_setting(key=key, label=LABEL)
+    except ResourceNotFoundError:
+        return None
+
+
+def effective_setting(
+    app_config_client: Any,
+    key: str,
+    default: str,
+    environ: Mapping[str, str] = os.environ,
+) -> str:
+    if key in environ:
+        return environ[key]
+    setting = get_configuration_setting_or_none(app_config_client, key)
+    return default if setting is None else str(setting.value)
+
+
+def is_truthy(value: str) -> bool:
+    return value.strip().lower() in {"1", "true", "yes", "on"}
+
+
+def apply_governance_configuration(
+    app_config_client: Any,
+    secret_client: Any,
+    vault_uri: str,
+    environ: Mapping[str, str] = os.environ,
+) -> bool:
+    """Apply plaintext defaults and a Key Vault reference without exposing the key."""
+    seed_governance_settings(app_config_client, environ)
+    current_setting = get_configuration_setting_or_none(
+        app_config_client,
+        AUDIT_HMAC_CONFIG_KEY,
+    )
+    desired_reference = key_vault_reference(vault_uri)
+
+    if current_setting is not None and (
+        current_setting.content_type or ""
+    ).startswith("application/vnd.microsoft.appconfig.keyvaultref+json"):
+        if current_setting.value == desired_reference:
+            return ensure_audit_hmac_secret(secret_client)
+        return False
+
+    initial_value = None
+    if current_setting is not None and current_setting.value:
+        initial_value = str(current_setting.value)
+        _decode_audit_hmac_key(initial_value)
+
+    created = ensure_audit_hmac_secret(secret_client, initial_value)
+
+    app_config_client.set_configuration_setting(
+        ConfigurationSetting(
+            key=AUDIT_HMAC_CONFIG_KEY,
+            label=LABEL,
+            value=desired_reference,
+            content_type=KEY_VAULT_REFERENCE_CONTENT_TYPE,
+        )
+    )
+    return created
+
+
+def main() -> None:
+    app_config_endpoint = os.environ.get("APP_CONFIG_ENDPOINT", "").strip()
+    if not app_config_endpoint:
+        raise RuntimeError("APP_CONFIG_ENDPOINT is required for governance setup.")
+
+    credential = create_credential()
+    app_config_client = AzureAppConfigurationClient(app_config_endpoint, credential)
+    deploy_key_vault = os.environ.get("DEPLOY_KEY_VAULT", "true").strip().lower()
+    if deploy_key_vault in {"0", "false", "no", "off"}:
+        seed_governance_settings(app_config_client)
+        audit_enabled = effective_setting(
+            app_config_client,
+            "AUDIT_EVENTS_ENABLED",
+            "false",
+        )
+        if is_truthy(audit_enabled):
+            raise RuntimeError(
+                "AUDIT_EVENTS_ENABLED cannot be true when Key Vault is disabled."
+            )
+        logging.warning(
+            "Governance defaults were seeded, but AUDIT_HMAC_KEY was not "
+            "provisioned because Key Vault is disabled. Keep audit events disabled."
+        )
+        return
+
+    vault_uri = resolve_vault_uri()
+    secret_client = SecretClient(vault_url=vault_uri, credential=credential)
+    created = apply_governance_configuration(
+        app_config_client,
+        secret_client,
+        vault_uri,
+    )
+
+    action = "created" if created else "reused"
+    logging.info(
+        "Governance configuration applied: safe defaults written, audit HMAC "
+        "secret %s, and Key Vault reference registered.",
+        action,
+    )
+
+
+if __name__ == "__main__":
+    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
+    main()

config/governance/tests/__init__.py

@@ -0,0 +1 @@
+"""Tests for governance deployment configuration."""