diff --git a/reference/implementation-specifications.md b/reference/implementation-specifications.md new file mode 100644 index 0000000..6979c7a --- /dev/null +++ b/reference/implementation-specifications.md @@ -0,0 +1,728 @@ +# DCM Data Model โ€” Implementation Specifications + +**Document Status:** ๐Ÿ“‹ Draft โ€” Ready for Implementation Feedback +**Document Type:** Implementation Reference +**Related Documents:** [Control Plane Components](../architecture/control-plane/components.md) | [data stores](https://github.com/croadfeldt/udlm/blob/main/contracts/storage-providers.md) | [Universal Audit](https://github.com/croadfeldt/udlm/blob/main/observability/universal-audit.md) | [credential management service Model](https://github.com/croadfeldt/udlm/blob/main/governance/credentials.md) | [Deployment Redundancy](../architecture/runtime-features/deployment-redundancy.md) | [Session Revocation](../architecture/control-plane/session-revocation.md) + +> **AEP Alignment:** API endpoints follow [AEP](https://aep.dev) conventions. +> See `schemas/openapi/dcm-admin-api.yaml` and `dcm-consumer-api.yaml` for normative specs. + +--- + +## 1. Purpose + +This document specifies the implementation mechanics for capabilities that are architecturally defined elsewhere but whose runtime behavior โ€” enforcement location, algorithm, data structure โ€” has not been fully specified. It closes implementation gaps identified in the architecture gap analysis. + +--- + +## 2. Rate Limiting โ€” Enforcement Implementation + +Rate limiting is defined at the interface level in the Consumer API Specification (ยง1.6) and the Admin API. This section specifies *how* it is enforced. + +### 2.1 Enforcement Location + +Rate limiting is enforced by the **API Gateway** component โ€” the single ingress point for all consumer and admin API traffic. It is enforced before the request reaches any pipeline component. The Request Orchestrator never sees rate-limited requests. + +Rate limiting is **not** enforced at the network layer (load balancer) or application layer (Request Payload Processor). A single enforcement point at the API Gateway ensures: +- Consistent limits across all consumer paths (Web UI, direct API, CI/CD) +- No rate limit bypass via internal component calls +- Single source of rate limit state for accurate tracking + +### 2.2 Token Bucket Algorithm + +DCM uses the **token bucket** algorithm with a per-actor bucket: + +``` +Actor makes request: + โ”‚ + โ–ผ API Gateway looks up actor_uuid in rate limit store + โ”‚ (in-memory cache backed by a fast PostgreSQL store contract) + โ”‚ + โ–ผ Current bucket state: + โ”‚ tokens_remaining: + โ”‚ last_refill_at: + โ”‚ + โ–ผ Refill calculation: + โ”‚ elapsed_seconds = now - last_refill_at + โ”‚ tokens_to_add = elapsed_seconds ร— (rate_limit / 60) + โ”‚ tokens_remaining = min(bucket_max, tokens_remaining + tokens_to_add) + โ”‚ last_refill_at = now + โ”‚ + โ–ผ Token check: + โ”œโ”€โ”€ tokens_remaining >= 1: + โ”‚ tokens_remaining -= 1 + โ”‚ Request proceeds + โ”‚ Response headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset + โ”‚ + โ””โ”€โ”€ tokens_remaining < 1: + Request rejected: 429 Too Many Requests + Response headers: Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset + Audit record written: rate_limit_exceeded +``` + +### 2.3 Bucket Parameters by Profile + +| Profile | Rate (req/min) | Burst Max | Bucket Max | +|---------|---------------|-----------|------------| +| `minimal` | 60 | 20 | 80 | +| `dev` | 120 | 40 | 160 | +| `standard` | 300 | 100 | 400 | +| `prod` | 600 | 200 | 800 | +| `fsi` | 600 | 200 | 800 | +| `sovereign` | 600 | 200 | 800 | + +System components (service accounts, provider callbacks) use `prod` bucket parameters regardless of profile. + +### 2.4 Rate Limit State Store + +The rate limit state is stored in a **dedicated in-memory cache** backed by a fast data store: +- Cache TTL: 2ร— the rate limit window (120 seconds for 60 req/min rate) +- PostgreSQL store contract: key-value (Redis or equivalent) +- Consistency: eventual โ€” brief over-counting tolerated to avoid distributed lock overhead +- Cross-replica sharing: rate limit state is shared across all API Gateway replicas via the backing store + +### 2.5 Exemptions + +The following are exempt from consumer rate limits: +- Admin API calls (separate rate limit bucket, 3ร— profile limit) +- Provider callback endpoints (authenticated via provider callback credential; separate per-provider bucket) +- Internal DCM component calls (authenticated via mTLS + interaction credential; not rate limited) +- Health check endpoints (`/livez`, `/readyz`, `/metrics`) + +### 2.6 System Policies + +| Policy | Rule | +|--------|------| +| `RLM-001` | Rate limiting is enforced at the API Gateway. No other component enforces rate limits. | +| `RLM-002` | Rate limit buckets are per authenticated actor (actor_uuid). Unauthenticated requests are rejected at auth before reaching the rate limiter. | +| `RLM-003` | All rate limit rejections produce an audit record with actor_uuid, endpoint, and timestamp. | +| `RLM-004` | Rate limit parameters are governed by the active Profile. Operators may increase but not decrease profile-defined limits. | +| `RLM-005` | Rate limit state is not persisted across API Gateway restarts. Buckets refill from empty after restart โ€” brief over-serving is acceptable. | + +--- + +## 3. Audit Log Hash Chain โ€” Verification Schedule and Implementation + +The hash chain structure is defined in [Universal Audit](https://github.com/croadfeldt/udlm/blob/main/observability/universal-audit.md) ยง8. This section specifies the verification schedule, triggering component, and response protocol. + +### 3.1 Hash Computation + +Each audit record's `record_hash` is computed as: + +``` +record_hash = SHA-256( + record_uuid || + record_timestamp || + entity_uuid || + action || + actor.immediate.uuid || + subject_handle || + chain_sequence || + previous_record_hash +) + +Where || denotes canonical concatenation with a field separator (0x1F โ€” ASCII unit separator). +The hash is stored as a lowercase hex string. +``` + +The `previous_record_hash` for the first record in an entity's chain is `SHA-256("GENESIS")` โ€” a known constant, not null. + +### 3.2 Verification Schedule + +Hash chain verification runs on two schedules: + +**Continuous verification (per-write):** +Every audit record write triggers an immediate verification of that record against its predecessor. This catches chain breaks at write time โ€” before the record is committed. A write that would break the chain is rejected and triggers `audit.chain_integrity_alert`. + +**Periodic batch verification (scheduled):** +The Audit component runs a full-chain verification sweep on a profile-governed schedule: + +| Profile | Sweep interval | Scope per sweep | +|---------|---------------|-----------------| +| `dev` | P7D | All entities modified in the last 7 days | +| `standard` | P1D | All entities modified in the last 24 hours | +| `prod` | PT12H | All entities modified in the last 12 hours | +| `fsi` | PT6H | All entities + random 5% sample of all-time records | +| `sovereign` | PT1H | All entities + random 10% sample of all-time records | + +### 3.3 Owning Component + +Hash chain verification is owned by the **Audit component** โ€” the same component that writes audit records. It is not a separate service. The Audit component runs verification as a background goroutine with no external trigger required. + +For data store implementations: the Audit Store must support ordered range queries by `(entity_uuid, chain_sequence)` to enable efficient sweep verification. + +### 3.4 Breach Response Protocol + +``` +Chain break detected (during write-time or sweep verification): + โ”‚ + โ–ผ Affected records flagged: integrity_status = chain_break + โ”‚ Break point: chain_sequence N where hash mismatch occurs + โ”‚ All records with chain_sequence > N for this entity: integrity_status = unverified + โ”‚ + โ–ผ audit.chain_integrity_alert event fired (urgency: critical, non-suppressable) + โ”‚ payload: {entity_uuid, entity_type, break_at_sequence, break_detected_at, sweep_type} + โ”‚ + โ–ผ Notifications dispatched: + โ”‚ โ†’ Platform Admin (urgency: critical) + โ”‚ โ†’ Security team (if configured in notification routing) + โ”‚ + โ–ผ Affected entity flagged in audit dashboard + โ”‚ Consumer-visible: "Audit integrity alert โ€” contact platform admin" + โ”‚ + โ””โ”€โ”€ Human investigation required: + Normal resolution paths: + - store failure caused write corruption โ†’ data store replacement + - Clock skew between replicas caused ordering issue โ†’ Non-malicious; document and reseal + - Administrative error (direct DB edit) โ†’ Incident report, access review + - Malicious tampering โ†’ Security incident declared +``` + +### 3.5 Chain Resealing + +After a chain break is investigated and root cause documented, a platform admin may reseal the chain: + +``` +POST /api/v1/admin/audit/entities/{entity_uuid}:reseal-chain + { + "investigation_reference": "INC-2026-042", + "root_cause": "storage_failure", + "resolution_notes": "PostgreSQL WAL corruption during storage migration" + } + +Response: + { + "entity_uuid": "", + "chain_resealed_at": "", + "records_affected": 7, + "new_chain_anchor": "", + "audit_record_uuid": "" + } +``` + +The reseal itself produces an audit record that references the investigation. Chain integrity is restored from the reseal point forward. + +--- + +## 4. Multi-Tenancy at the Storage Layer + +Tenant isolation in DCM is enforced at the data model level (every entity carries `tenant_uuid`) and at the API level (all consumer endpoints are scoped to the authenticated actor's tenant). This section specifies the storage-layer enforcement mechanisms. + +### 4.1 Isolation Strategy by Store Type + +| Store Type | Isolation Strategy | Implementation Notes | +|-----------|-------------------|---------------------| +| **DCM database** (Intent/Requested State) | Directory namespace per tenant | `/tenants/{tenant_uuid}/intents/`, `/tenants/{tenant_uuid}/requests/` โ€” Git ACLs enforce read/write scope | +| **pipeline_events table** (Audit) | Separate stream per tenant | `dcm.audit.{tenant_uuid}` stream; Kafka topic ACLs restrict producer/consumer access | +| **realized data domain** (Realized State) | Row-level filter + column-level encryption | `tenant_uuid` column indexed; all queries mandatory-include `WHERE tenant_uuid = ?`; tenant-scoped encryption key | +| **Search Index** | Index namespace per tenant | Separate index prefix `tenant_{uuid}_*`; query routing enforces tenant scope | +| **Rate Limit Cache** | Key-namespaced per actor (actor carries tenant context) | `rl:{tenant_uuid}:{actor_uuid}` key structure | + +### 4.2 realized data domain โ€” Row-Level Security Implementation + +The realized data domain (Realized State) uses row-level security as the primary isolation mechanism: + +```sql +-- PostgreSQL row-level security policy +CREATE POLICY tenant_isolation ON realized_state_records + USING (tenant_uuid = current_setting('dcm.current_tenant_uuid')::uuid); + +-- Every connection sets tenant context before any query: +SET LOCAL dcm.current_tenant_uuid = ''; + +-- This makes it impossible to query across tenant boundaries, +-- even with direct database access using the application credential. +-- Platform admin access uses a separate role without the RLS policy. +``` + +### 4.3 Tenant-Scoped Encryption + +For `fsi` and `sovereign` profiles, realized state records are encrypted at rest using a per-tenant encryption key: + +``` +Tenant provisioned: + โ”‚ + โ–ผ credential management service generates tenant encryption key (AES-256-GCM) + โ”‚ Key stored in: credential management service (e.g., Vault) + โ”‚ Key reference stored in: Tenant record as tenant_encryption_key_ref + โ”‚ + โ–ผ On write to realized data domain: + โ”‚ API Gateway fetches tenant encryption key + โ”‚ Payload encrypted with tenant key before storage + โ”‚ data store stores ciphertext only + โ”‚ + โ–ผ On read from realized data domain: + โ”‚ API Gateway fetches tenant encryption key + โ”‚ Decrypts payload in memory + โ”‚ Plaintext never written to data store logs + โ”‚ + โ–ผ Tenant decommission: + Tenant encryption key revoked in credential management service + All tenant data becomes unreadable without external recovery + This is the cryptographic equivalent of data deletion +``` + +### 4.4 Cross-Tenant Query Prevention + +Platform admin endpoints that query across tenants use a separate database role with explicit permission grants โ€” they do not bypass RLS, they use a role that has cross-tenant read permission with full audit logging. The principle is: cross-tenant reads are possible only through intentional, audited, privileged operations. + +### 4.5 System Policies + +| Policy | Rule | +|--------|------| +| `STI-001` | Every query to a tenant-scoped store must include `tenant_uuid` as a mandatory predicate. Queries without tenant scope are rejected by the storage layer. | +| `STI-002` | Row-level security is enabled on all relational snapshot stores. Disabling RLS requires platform admin action and produces an audit record. | +| `STI-003` | For `fsi` and `sovereign` profiles, tenant-scoped encryption is mandatory. Key rotation is performed on a profile-governed schedule (P90D for fsi; P30D for sovereign). | +| `STI-004` | data store implementations must declare their tenant isolation strategy at registration. DCM validates the declared strategy against the active profile's isolation requirements during the registration approval pipeline. | + +--- + +## 5. Cross-Region Data Replication + +DCM's multi-region deployment model is specified in [Deployment Redundancy](../architecture/runtime-features/deployment-redundancy.md). This section specifies the replication mechanics and sovereignty enforcement at the replication layer. + +### 5.1 What Replicates Where + +| Store Type | Replication Model | Sovereignty Constraint | +|-----------|------------------|----------------------| +| **DCM database** (Intent State) | Git push/pull โ€” upstream-downstream replication | Intent records tagged with `sovereignty_zone`; replicated only to stores within the declared zone | +| **pipeline_events table** (Audit) | Stream mirroring with lag monitoring | Audit records replicated to all authorized regions; cross-sovereignty replication requires explicit consent | +| **realized data domain** (Realized State) | Synchronous within-zone; async cross-zone with consent | `sovereignty_zone` on entity governs which regions may hold a copy | +| **Search Index** | Async replication; eventual consistency acceptable | Same sovereignty rules as realized data domain | + +### 5.2 Sovereignty-Aware Replication + +Every entity carries `sovereignty_zone` declarations that constrain which data store instances may hold copies: + +```yaml +entity: + entity_uuid: + sovereignty_zones: + - zone_id: EU-WEST + data_classifications: [restricted, phi] # these classifications must stay in EU-WEST + - zone_id: "*" + data_classifications: [internal] # internal data may replicate anywhere +``` + +The replication controller evaluates `sovereignty_zones` before routing any replication event. Replication to a non-authorized region for a given data classification is blocked at the replication layer โ€” the storage provider receives a `SOVEREIGNTY_VIOLATION` rejection. + +### 5.3 Replication Lag Monitoring + +``` +data store declares: max_replication_lag: PT30S + +DCM monitoring: + Every PT10S: measure replication lag across all replica pairs + + If lag > max_replication_lag: + storage.replication_lag_exceeded event (urgency: medium) + + If lag > 5 ร— max_replication_lag: + storage.replication_degraded event (urgency: high) + Affected region marked: capacity_status = degraded + New requests avoid degraded region for placement + + If replica unreachable: + storage.replica_unavailable event (urgency: critical) + Affected region marked: capacity_status = unavailable + Requests that require this region: held pending recovery +``` + +### 5.4 Conflict Resolution + +DCM uses a **last-write-wins with causality tracking** model for cross-region conflicts: + +- All writes carry a vector clock `{region_id: sequence_number}` +- Concurrent writes (same entity, different regions) are detected by vector clock comparison +- Resolution: the write with higher aggregate sequence number wins +- Losing write: preserved as a `conflict_record` in the Audit Store (never silently dropped) +- Platform admin notified of conflicts above a configurable threshold + +--- + +## 6. Secret Zero โ€” Initial Credential Bootstrap + +The bootstrap sequence is specified in [Deployment Redundancy](../architecture/runtime-features/deployment-redundancy.md) ยง6. This section specifies the credential bootstrap specifically โ€” how DCM components authenticate to each other before the credential management service is running. + +### 6.1 The Bootstrap Credential Problem + +At day-0, no credential management service exists. DCM components need credentials to communicate. The resolution is a **declarative bootstrap manifest** that contains one-time bootstrap credentials, plus a mandatory rotation on first successful startup. + +### 6.2 Bootstrap Sequence โ€” Credential Perspective + +``` +1. Bootstrap manifest contains: + bootstrap_credentials: + internal_ca: + cert_pem: + key_pem: # sealed with bootstrap passphrase + + bootstrap_admin: + username: bootstrap-admin + password_hash: # operator sets this + + component_credentials: + # Pre-shared credentials for component-to-component auth + # until mTLS internal CA is operational + api_gateway: {shared_secret: } + orchestrator: {shared_secret: } + policy_engine: {shared_secret: } + audit: {shared_secret: } + +2. Bootstrap DCM starts: + - Internal CA initialized from bootstrap_credentials.internal_ca + - Components issued mTLS certificates from Internal CA + - Pre-shared secrets replaced by mTLS certificates on first successful CA handshake + - Pre-shared secrets deleted from memory and manifest after replacement + +3. credential management service starts: + - Bootstrapped with Internal CA certificate (trusts DCM's CA) + - Registered as the primary credential management service via bootstrap admin credential + - Takes ownership of internal CA key management + - Internal CA private key: transferred to credential management service, deleted from bootstrap manifest + +4. Bootstrap admin credential rotation (BOOT-002 โ€” mandatory): + - Bootstrap admin password must be rotated on first login + - New credential issued by credential management service (not the bootstrap manifest) + - Old password hash deleted from manifest + - Manifest sealed: no more secrets, only configuration + +5. Bootstrap manifest after completion: + - Contains only: DCM deployment configuration, Git remote, profile + - No secrets remain in the manifest + - Manifest committed to Git (now safe, secret-free) +``` + +### 6.3 Air-Gapped Bootstrap + +For sovereign/air-gapped deployments where the credential management service requires network access to an external vault: + +``` +Option A โ€” Embedded credential management service: + Use a locally-running credential management service (e.g., HashiCorp Vault in dev mode) + bootstrapped from the bootstrap manifest. + Upgrade to production Vault config post-bootstrap. + +Option B โ€” Operator-held keys: + Bootstrap manifest contains encrypted key material. + Operator provides passphrase at bootstrap time via stdin or hardware token. + Keys are never stored unencrypted at rest. + +Option C โ€” HSM-backed bootstrap: + Internal CA private key is generated inside an HSM. + Bootstrap manifest contains only the HSM endpoint and slot reference. + Requires HSM to be available before DCM bootstrap begins. +``` + +### 6.4 System Policies + +| Policy | Rule | +|--------|------| +| `BOOT-001` | The bootstrap manifest must not contain secrets after bootstrap completion. Any secret that persists in the manifest after first successful startup is a security violation. | +| `BOOT-002` | The bootstrap admin credential must be rotated on first login. DCM enforces this โ€” the bootstrap admin account is locked from normal use until rotation is complete. | +| `BOOT-003` | Pre-shared component credentials must be replaced by mTLS certificates within PT5M of Internal CA startup. Any component still using pre-shared secrets after this window generates a security alert. | +| `BOOT-004` | The Internal CA private key must be transferred to the credential management service on credential management service registration. The key must not remain in any component's memory or storage after transfer is confirmed. | + +--- + +## 7. Ownership Ambiguities โ€” Resolved + +### 7.1 Who Issues `operation_uuid`? + +**Decision: The API Gateway issues `operation_uuid` at request ingress.** + +Rationale: The API Gateway is the component that receives the POST request and must return the Operation response immediately. It assigns the UUID synchronously before any pipeline processing begins. The `operation_uuid` equals the `request_uuid` โ€” they are the same UUID, assigned at ingress. + +``` +Consumer: POST /api/v1/requests {...} + +API Gateway: + 1. Authenticates consumer (checks session token) + 2. Assigns request_uuid = operation_uuid = UUID4() โ† here + 3. Writes initial request record to Intent Store (status: INITIATED) + 4. Publishes request.initiated event to Request Orchestrator (with request_uuid) + 5. Returns Operation{name: /api/v1/operations/{request_uuid}, done: false} + +Request Orchestrator: + Receives request.initiated event with request_uuid + Uses the already-assigned request_uuid throughout the pipeline + Never assigns a new UUID +``` + +The Operation resource lives in a fast-queryable store owned by the API Gateway. When the pipeline progresses, the Request Orchestrator updates the Operation status by writing to this store (it has write access; the API Gateway reads from it for GET /api/v1/operations/{uuid} responses). + +### 7.2 Who Owns the Credential Revocation Registry? + +**Decision: The credential management service owns the Credential Revocation Registry.** + +Rationale: The credential management service is the authoritative source of credential lifecycle state. It issues credentials, rotates them, and revokes them. The revocation registry is a projection of that lifecycle state optimized for fast lookup. + +``` +Credential Revocation Registry: + Owner: credential management service + Storage: dedicated fast cache (Redis or equivalent) + Key structure: credential_uuid โ†’ {revoked_at, revocation_reason, effective_at} + TTL: max(credential_ttl, P90D) โ€” persists at minimum 90 days after revocation + +Access model: + Write: credential management service (on revocation event) + Read: All DCM components (via credential management service query API) + OR via local cache synced from credential management service push events + +Cache sync protocol: + credential management service publishes: credential.revoked event (Message Bus) + All subscribed components update local revocation cache + Cache TTL: PT1M standard; PT30S fsi/sovereign + On cache miss: component queries credential management service directly (not the cache) + +Session Revocation Registry: separate, owned by the Auth component + (Session revocation is distinct from credential revocation) +``` + +--- + +## 8. Security Posture Specifications + +### 8.1 Threat Model โ€” Attack Surface Summary + +DCM's attack surface has five distinct boundaries. Each boundary has a specific trust model and mitigation set. + +**Boundary 1 โ€” Consumer Ingress (Web UI, Consumer API)** +- Threat: Credential theft / session hijacking +- Mitigations: mTLS optional at consumer boundary; bearer tokens with short TTL (PT1H standard); session revocation registry checked on every request; rate limiting at API Gateway +- Threat: Tenant escape (accessing another tenant's data) +- Mitigations: All queries mandatory-include tenant_uuid; row-level security on storage; Governance Matrix enforced before any read + +**Boundary 2 โ€” Provider Interface (Operator Interface, Callback API)** +- Threat: Provider impersonation (malicious actor claims to be a legitimate provider) +- Mitigations: mTLS required at provider boundary; provider callback credential required for callbacks; API Gateway validates dcm_entity_uuid in every callback against provider's registered entity scope +- Threat: Malicious provider payload (provider sends crafted Realized State payload) +- Mitigations: Realized State payloads validated against Resource Type Specification schema on receipt; GateKeeper policies evaluate provider-supplied data before it enters DCM state + +**Boundary 3 โ€” Admin Interface (Admin API)** +- Threat: Unauthorized platform admin action +- Mitigations: Authority Tier model enforces multi-tier approval for high-impact actions; all admin actions produce non-suppressable audit records; emergency admin access (break-glass) triggers immediate security notification +- Threat: Configuration injection via GitOps +- Mitigations: All GitOps PRs require domain-appropriate review before merge; policy contributions enter shadow mode before activation; GateKeeper policies validate all contributions at submission + +**Boundary 4 โ€” Internal Component Communication** +- Threat: Component impersonation (compromised component issues requests as another) +- Mitigations: mTLS with Internal CA for all component-to-component calls; interaction credentials checked on every call; Credential Revocation Registry queried on credential use +- Threat: Lateral movement after component compromise +- Mitigations: Each component holds minimum-scope interaction credentials; no component has write access to stores it does not own; audit records cannot be deleted by any component + +**Boundary 5 โ€” Storage Layer** +- Threat: Direct database access bypassing application controls +- Mitigations: Row-level security enforces tenant isolation even with direct DB access using application credentials; platform admin credentials are separate, audited, and require MFA; data store provenance emission means all direct writes are detectable + +**Highest-risk paths (not mitigated by single control):** +1. credential management service compromise โ†’ cascading trust failure. Mitigation: credential management service is air-gapped from consumer traffic; HSM-backed key storage for sovereign profiles; separate backup credential authority. +2. Internal CA compromise โ†’ all component trust fails. Mitigation: CA private key held only in credential management service (HSM-backed for fsi/sovereign); CA certificate rotation procedure documented. + +### 8.2 Supply Chain Security + +**Provider OpenAPI Spec Signing:** +All Service Provider OpenAPI specifications submitted at registration must be signed using the provider's private key (corresponding to the public key in their mTLS certificate). DCM verifies the signature before the spec is processed. Unsigned specs are rejected with `SPEC_UNSIGNED` at GATE-SP-01. + +**Operator Container Image Provenance:** +The DCM reference implementation containers are signed using Sigstore (Cosign). Deployment manifests declare the expected image digest. Any container running a different digest triggers drift detection on DCM's own deployment. + +**DCM database Secrets Scanning:** +All content committed to DCM's GitOps stores passes through a secrets scanner before being accepted. The scanner checks for: +- High-entropy strings matching known secret patterns (API keys, tokens, private keys) +- Known credential formats (AWS access keys, GitHub PATs, JWT secrets) +- PEM-encoded private key blocks + +A commit containing detected secrets is rejected with `SECRETS_DETECTED` and an audit record is written. The committing actor is notified. + +**SBOM Declaration:** +Service Providers must declare a Software Bill of Materials reference at registration (optional for Tier 1 `dev` profiles; required for `fsi` and `sovereign`). The SBOM reference is stored in the provider record and included in accreditation evidence. + +### 8.3 System Policies + +| Policy | Rule | +|--------|------| +| `SEC-001` | All provider OpenAPI specs submitted at registration must be signed. Signature verification is performed at GATE-SP-01. | +| `SEC-002` | DCM GitOps stores enforce secrets scanning on all commits. Commits with detected secrets are rejected. | +| `SEC-003` | For `fsi` and `sovereign` profiles, SBOM declaration is mandatory for all Service Providers before activation. | +| `SEC-004` | The Internal CA private key must be stored in an HSM for `sovereign` profile deployments. Software-only key storage is not permitted at sovereign profile. | +| `SEC-005` | Any direct database write to a DCM store that bypasses the application layer is detectable via data store provenance emission. Detection triggers `audit.chain_integrity_alert` for affected records. | + +--- + +## 9. Experience Gap Specifications + +### 9.1 New Tenant Onboarding Flow + +``` +Platform Admin initiates tenant creation: + POST /api/v1/admin/tenants + { + "display_name": "Payments Platform", + "handle": "payments-platform", + "group_class": "tenant_boundary", + "initial_quota_profile": "standard", + "billing_contact": "payments-ops@corp.example.com", + "data_classifications_permitted": ["internal", "restricted"], + "sovereignty_zones": ["EU-WEST"] + } + +DCM auto-provisions: + 1. Tenant entity created (tenant_uuid assigned) + 2. Default resource groups created: + - payments-platform/default (general resources) + - payments-platform/admins (tenant admin group) + 3. Initial quota applied per quota_profile declaration + 4. Tenant admin actor created (if initial_admin_email provided): + - Actor record created + - Welcome notification dispatched with first-login credential + 5. Tenant Git namespace provisioned in GitOps store: + - /tenants/payments-platform/ directory created + - Initial tenant-scope policy stubs committed (shadow mode) + 6. Search index namespace initialized + 7. Audit stream created: dcm.audit.{tenant_uuid} + +Tenant admin completes setup: + 1. First login โ†’ mandatory credential rotation (BOOT-002 equivalent) + 2. Configure Auth Provider (or inherit platform default) + 3. Add tenant members (invite by email or LDAP group mapping) + 4. Review and activate initial policy stubs + 5. Submit first service request (onboarding validation complete) + +Onboarding event sequence: + tenant.created โ†’ Platform Admin + tenant.member_added ร— N โ†’ new members (welcome email) + tenant.quota_configured โ†’ Platform Admin + tenant.onboarding_complete โ†’ Platform Admin + Tenant Admin + (fired when first OPERATIONAL entity exists in the tenant) +``` + +### 9.2 Pre-Request Cost Estimation UX + +The consumer experience for cost estimation before committing a request: + +``` +Step 1: Consumer browses catalog + GET /api/v1/catalog/{catalog_item_uuid} + Response includes: cost_estimate: {monthly_usd: 45.00, basis: "declared_static"} + +Step 2: Consumer configures request fields (e.g., selects VM size) + POST /api/v1/cost/estimate + { + "catalog_item_uuid": "", + "fields": {"cpu": 8, "ram_gb": 32, "storage_gb": 200, "environment": "prod"} + } + Response: + { + "estimated_monthly_usd": 187.50, + "cost_basis": "dynamic", + "cost_breakdown": [ + {"component": "compute", "monthly_usd": 120.00}, + {"component": "storage", "monthly_usd": 40.00}, + {"component": "network_egress", "monthly_usd": 27.50} + ], + "disclaimer": "Estimate based on declared provider rates. Actual costs may vary.", + "provider_uuid": null // not yet placed; estimate is across eligible providers + } + +Step 3: Consumer submits request with dry_run: true (optional pre-flight) + POST /api/v1/requests + { + "catalog_item_uuid": "", + "fields": {...}, + "dry_run": true // evaluate policy and placement; do not dispatch + } + Response: Operation with metadata.dry_run_result: + { + "policy_result": "PASS", + "placement_result": { + "selected_provider": "eu-west-prod-1", + "cost_at_selected_provider": 182.00 + }, + "gatekeeper_gates": [], + "warnings": ["Storage class 'premium' requested; 'standard' also eligible at $35.00/mo"] + } + +Step 4: Consumer submits without dry_run โ†’ actual request +``` + +### 9.3 Provider Sandbox / Test Mode + +Providers can register in sandbox mode for development and certification testing without affecting production routing: + +```yaml +provider_registration: + # ...standard registration fields... + sandbox_mode: true # this provider never receives production requests + sandbox_profile: dev # sandbox providers only activated under dev profile + + # Sandbox providers: + # - Appear in the provider registry with status: sandbox + # - Can be explicitly targeted by test requests (fields.target_provider_uuid) + # - Never appear in placement engine candidate selection for non-test requests + # - Produce full audit records (useful for certification evidence) + # - Subject to same API validation as production providers + # - Can graduate to production via standard registration approval flow +``` + +Test request targeting a sandbox provider: +``` +POST /api/v1/requests +{ + "catalog_item_uuid": "", + "fields": { ... }, + "_test_context": { + "target_provider_uuid": "", + "suppress_billing": true, + "test_label": "certification-run-2026-04-01" + } +} +``` + +### 9.4 SLA/SLO Tracking + +DCM tracks service delivery against declared SLOs at the Resource Type level: + +```yaml +# Declared in Resource Type Specification (doc 05) +resource_type_slo: + resource_type: Compute.VirtualMachine + + slos: + - metric: time_to_operational + target_percentile: p95 + target_value: PT30M # 95% of VMs should be OPERATIONAL within 30 minutes + measurement_window: P7D # measured over trailing 7 days + + - metric: uptime + target_percentile: p99 + target_value: "99.5%" # 99.5% uptime over trailing 30 days + measurement_window: P30D + + - metric: drift_detection_latency + target_percentile: p90 + target_value: PT1H # drift detected within 1 hour of occurrence + measurement_window: P7D +``` + +SLO breach detection: +``` +DCM computes SLO metrics continuously from audit records and entity lifecycle events. + +When a metric crosses a threshold: + slo.breach_approaching (urgency: medium) โ€” at 90% of SLO budget consumed + slo.breach_detected (urgency: high) โ€” SLO violated + payload: {resource_type, slo_metric, target, actual, measurement_window} + +Consumer-facing: + GET /api/v1/resources/{entity_uuid}/slo-status + Returns: current SLO metrics for the entity's resource type + +Platform admin: + GET /api/v1/admin/slo/report?resource_type=Compute.VirtualMachine&window=P7D + Returns: aggregate SLO performance across all entities of this type +``` + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* diff --git a/reference/implementation-standards.md b/reference/implementation-standards.md new file mode 100644 index 0000000..b35137f --- /dev/null +++ b/reference/implementation-standards.md @@ -0,0 +1,447 @@ +--- +Document Status: โœ… Stable โ€” DCM implementation +Document Type: Reference โ€” Implementation Standards +Established: 2026-05-26 +Maps to: udlm/reference/standards-catalog.md +--- + +# DCM Implementation Standards + +> **Selects implementations of the standards listed in UDLM**: +> [udlm/reference/standards-catalog.md](https://github.com/croadfeldt/udlm/blob/main/reference/standards-catalog.md). +> UDLM lists the normative external standards that any DCM-conformant +> realization must consume. This document records the specific +> implementation choices DCM makes: which algorithms, which protocol +> implementations, which OpenAPI conventions, which observability stack, +> which K8s integration, and which compliance configurations per profile. + +--- + +## 1. Cryptographic implementation details + +DCM selects specific algorithms from the UDLM-approved set, profile-governed. + +### 1.1 Algorithm choices by credential type + +| Credential type | Algorithm | Key size | Profile | +|---|---|---|---| +| `api_key` | Cryptographically random | 256 bits min | All | +| `x509_certificate` | Ed25519 (preferred) | 256-bit | standard/prod | +| `x509_certificate` | ECDSA P-384 | 384-bit | fsi/sovereign (FIPS) | +| `x509_certificate` | RSA-4096 | 4096-bit | Permitted for compatibility | +| `ssh_key` | Ed25519 (preferred) | 256-bit | standard/prod | +| `ssh_key` | ECDSA P-384 | 384-bit | fsi (FIPS) | +| `service_account_token` | ES256 (preferred) | P-256 | standard/prod | +| `service_account_token` | RS256 | RSA 4096 | Compatibility | +| `database_password` | Cryptographically random | 128-bit printable | standard | +| `database_password` | Cryptographically random | 256-bit | fsi/sovereign | +| `dcm_interaction` | HS256 or ES256 | 256-bit | All | +| `hsm_backed_key` | ECDSA P-384 | HSM-generated | fsi/sovereign | + +### 1.2 Forbidden algorithms (all profiles) + +DCM rejects credentials using these at issuance regardless of profile: + +- MD5, SHA-1 +- DES, 3DES +- RC4 +- RSA < 2048 +- ECDSA curves weaker than P-256 +- DSA-1024 + +The negative list applies even in `minimal`/`dev` โ€” real attacks hit all +deployments. Homelab is not exempt. + +### 1.3 Hash and integrity + +| Use | Algorithm | Profile | +|---|---|---| +| Audit hash chain | SHA-256 | All (minimum) | +| Audit hash chain | SHA-384 / SHA-512 | fsi/sovereign (preferred) | +| TLS cipher suites | per RFC 8446 (TLS 1.3 mandatory) | All | +| Audit signing | Ed25519 or ECDSA P-384 | All | + +### 1.4 Symmetric encryption + +| Use | Algorithm | Profile | +|---|---|---| +| Credential at-rest | AES-256-GCM | standard+ | +| Credential at-rest | AES-128-GCM permitted | minimal/dev (performance trade) | +| Pipeline payload encryption | AES-256-GCM (envelope) | All | +| Audit record encryption | AES-256-GCM (sovereign) | sovereign | + +### 1.5 FIPS level per profile + +| Profile | FIPS 140 level | +|---|---| +| `minimal` | None required | +| `dev` | None required | +| `standard` | None required | +| `prod` | Level 1 (software-only acceptable) | +| `fsi` | Level 2 (role-based authentication) | +| `sovereign` | Level 3 (physical tamper evidence + identity-based auth) | + +--- + +## 2. Certificate and key management procedures + +### 2.1 Internal CA (built-in) + +DCM ships with an internal CA that issues mTLS certificates to control plane +components. Default configuration: + +```yaml +internal_ca: + algorithm: ECDSA-P-384 + certificate_lifetime: P90D # rotates quarterly + ca_root_lifetime: P10Y # long-lived; offline backup + ocsp_endpoint: /api/v1/internal/ocsp + crl_endpoint: /api/v1/internal/crl + bootstrap_token_lifetime: PT24H # for new component enrollment +``` + +The CA root is stored in DCM's internal secrets table with HSM-backed +encryption in `fsi`/`sovereign` profiles. + +### 2.2 External CA integration (recommended for fsi/sovereign) + +DCM supports replacing the internal CA with an external enterprise CA via +the Credential Provider model. Supported protocols: + +| Protocol | RFC | Implementations | +|---|---|---| +| ACME | RFC 8555 | Let's Encrypt, cert-manager, Venafi, DigiCert | +| EST | RFC 7030 | Cisco CA, Microsoft NDES, Venafi | +| SCEP | RFC 8894 | Microsoft NDES, Cisco iOS CA | +| CMP | RFC 4210 | EJBCA, OpenXPKI | +| Native API | โ€” | HashiCorp Vault PKI, AWS ACM PCA, Azure Key Vault | + +This makes DCM's internal mTLS fully auditable through existing enterprise +PKI infrastructure โ€” a key requirement for fsi/sovereign profiles. + +### 2.3 Certificate rotation + +- Internal CA-issued certs rotate at P90D (default); P30D in fsi/sovereign +- External CA-issued certs rotate per CA's protocol (ACME orders are + short-lived; EST/SCEP follow enterprise schedule) +- Rotation overlap window: P7D โ€” both old and new certs accepted during + the window +- Pre-expiry warning: P14D before expiry + +### 2.4 OCSP and CRL + +- Internal CA exposes OCSP per RFC 6960 with `ocsp_stapling: true` +- CRL refresh: PT5M cache (standard); PT1M (fsi); PT30S (sovereign) +- External CA OCSP/CRL endpoints consumed via standard PKI library + +--- + +## 3. Authentication protocol integration + +### 3.1 OIDC implementation + +DCM uses standard OIDC discovery via the `/.well-known/openid-configuration` +endpoint. ID tokens validated per RFC 7519. Claims mapping configurable per +Auth Provider registration: + +```yaml +claims_mapping: + username: preferred_username + email: email + display_name: name + groups: groups # mapped to DCM roles per group_role_map + department: department # custom claim + cost_center: cost_center # custom claim +``` + +### 3.2 LDAP/AD implementation + +- RFC 4511 LDAP v3 +- StartTLS (RFC 4513) or LDAPS (port 636) โ€” mandatory in standard+ +- Bind operation for authentication +- Group membership via `member` attribute or RFC 4510 `LDAP_MATCHING_RULE_IN_CHAIN` + (1.2.840.113556.1.4.1941) for AD nested group resolution +- Connection pooling; idle timeout PT5M; max idle conns: 10 per pool + +### 3.3 SCIM 2.0 implementation + +- RFC 7643 (Core Schema) + RFC 7644 (Protocol) +- Endpoint: `/scim/v2/Users` and `/scim/v2/Groups` +- Bearer-token authenticated; token rotation per AUTH-014 +- Deprovision (DELETE /Users/{id}) triggers AUTH-016 (session revoke) + + CPX-006 (credential revoke) in parallel +- Roles NOT SCIM-provisioned โ€” explicit DCM policy authorization required + +### 3.4 SAML 2.0 (optional) + +- OASIS SAML 2.0 assertion parsing +- Signing cert + encryption cert via Credential Management Service +- AttributeStatement โ†’ DCM claims mapping + +### 3.5 OAuth 2.0 (RFC 6749) + +- Authorization Code flow for OIDC Auth Providers +- Client Credentials flow for service account API keys +- Token introspection per RFC 7662 at `POST /api/v1/auth:introspect` + +--- + +## 4. OpenAPI implementation specifics + +DCM uses OpenAPI 3.1 (OpenAPI Initiative 3.1) for all REST APIs. + +### 4.1 API specification locations + +| API | Spec file | +|---|---| +| Consumer API | `schemas/openapi/dcm-consumer-api.yaml` | +| Admin API | `schemas/openapi/dcm-admin-api.yaml` | +| Provider Callback API | `schemas/openapi/dcm-provider-callback-api.yaml` | +| Operator Interface | `schemas/openapi/dcm-operator-api.yaml` | + +### 4.2 Endpoint conventions + +DCM follows [AEP (API Enhancement Proposals)](https://aep.dev) conventions: + +- Resource-oriented URLs: `/api/v1/{resource_collection}/{resource_id}` +- Custom methods use colon syntax: `/api/v1/requests/{uuid}:cancel` +- Async operations return Operation resources for long-running work +- Pagination: `page_size` + `page_token` (cursor-based) +- Filtering: `filter` query param with simple expression language +- Field masks: `fields` query param for partial response + +### 4.3 Versioning headers + +- `Sunset` header (RFC 8594) on deprecated endpoints +- `Deprecation` header (RFC 9745) paired with Sunset +- `Link: rel="successor-version"` on deprecated responses +- Version negotiation via `Accept: application/vnd.dcm.v2+json` + +### 4.4 Error format + +```json +{ + "error": { + "code": "VALIDATION_FAILED", + "message": "Resource type Compute.VirtualMachine version 2.0.0 not active", + "details": { + "resource_type": "Compute.VirtualMachine", + "version": "2.0.0", + "current_version": "1.5.3" + }, + "request_id": "", + "documentation_url": "https://docs.dcm-project/errors/VALIDATION_FAILED" + } +} +``` + +--- + +## 5. Observability implementation + +### 5.1 Metrics (Prometheus / OpenMetrics) + +DCM exposes Prometheus exposition format at `GET /metrics` on each control +plane component. Standard metric families: + +| Family | Description | +|---|---| +| `dcm_requests_total` | Counter โ€” requests by tenant, resource_type, status | +| `dcm_requests_duration_seconds` | Histogram โ€” request pipeline duration | +| `dcm_provider_health_status` | Gauge โ€” 1=healthy, 0=unhealthy per provider | +| `dcm_policy_evaluation_seconds` | Histogram โ€” policy evaluation duration | +| `dcm_placement_decisions_total` | Counter โ€” placement outcomes by provider | +| `dcm_audit_records_total` | Counter โ€” audit volume | +| `dcm_credential_operations_total` | Counter โ€” by operation type, profile | +| `dcm_realized_entities` | Gauge โ€” current entities by resource_type, tenant | + +Metrics are scraped by Prometheus; profile-governed visibility (some sensitive +metrics hidden in `sovereign` per HLT-005). + +### 5.2 Distributed tracing (OpenTelemetry) + +DCM emits OTel spans for the request pipeline: + +| Span | Parent | +|---|---| +| `dcm.request.submit` | โ€” (root) | +| `dcm.request.assemble` | dcm.request.submit | +| `dcm.policy.evaluate` | dcm.request.assemble | +| `dcm.scoring.compute` | dcm.policy.evaluate | +| `dcm.placement.decide` | dcm.request.assemble | +| `dcm.provider.dispatch` | dcm.request.assemble | +| `dcm.provider.callback` | โ€” (linked via X-DCM-Correlation-ID) | +| `dcm.realized.persist` | dcm.provider.callback | + +`X-DCM-Correlation-ID` header propagated across all service boundaries; +maps to the OTel trace ID for cross-service tracing. + +### 5.3 Logging + +- Structured JSON logs at all log levels +- `correlation_id`, `request_id`, `actor_uuid`, `tenant_uuid` in every log + entry +- Log levels: `error`, `warn`, `info`, `debug` +- Profile defaults: `info` for standard+, `debug` for dev/minimal +- Log aggregation: not prescribed (Loki, Elasticsearch, Splunk all supported) + +### 5.4 Health endpoints + +- `/livez` โ€” process liveness; unauthenticated +- `/readyz` โ€” ready to serve traffic; unauthenticated +- `/api/v1/admin/health` โ€” detailed component health; authenticated +- `/api/v1/admin/health/dependencies` โ€” external dependency status + +Format: IANA `application/health+json` per RFC 8615. + +--- + +## 6. Kubernetes integration + +DCM is designed for K8s deployment but does not require it. K8s-specific +features when deployed on K8s: + +### 6.1 K8s manifests + +- Deployment / StatefulSet per control plane component +- Service for internal service discovery +- HorizontalPodAutoscaler for stateless services +- PodDisruptionBudget for stateful components +- NetworkPolicy for component-to-component allowed paths +- ServiceAccount per component with minimum-necessary RBAC + +### 6.2 CRD-based DCM Operator + +Optional DCM Operator translates DCM artifacts to/from K8s CRDs: + +| CRD | DCM artifact | +|---|---| +| `DCMRequest` | Resource request (consumer-friendly K8s-native intent) | +| `DCMPolicy` | Policy artifact | +| `DCMProvider` | Provider registration | +| `DCMTenant` | Tenant configuration | + +CRDs enable `kubectl apply -f` workflow as an ingress alongside the API. + +### 6.3 Probes + +- Liveness probe: `GET /livez` every PT10S; failureThreshold 3 +- Readiness probe: `GET /readyz` every PT5S; failureThreshold 1 +- Startup probe: `GET /livez` every PT5S; failureThreshold 12 (PT60S total) + +### 6.4 Service mesh integration + +Optional service mesh (Istio, OpenShift Service Mesh, Linkerd) replaces +application-level TLS for internal component communication. DCM's +implementation-specifications.md documents the recommended mesh config. + +--- + +## 7. Compliance configuration + +DCM ships profile-specific compliance overlays. Per profile, the following +standards are enforced: + +### 7.1 Profile โ†’ compliance mapping + +| Profile | Standards | +|---|---| +| `minimal` | None enforced | +| `dev` | None enforced | +| `standard` | ISO 27001 (advisory); SOC 2 (advisory) | +| `prod` | ISO 27001; SOC 2 Type II; NIST 800-53 (Moderate baseline advisory) | +| `fsi` | All of prod + PCI DSS overlay + HIPAA overlay (when active) + FedRAMP Moderate + DoD IL4 + FIPS 140-2 Level 2 | +| `sovereign` | All of fsi + sovereign-specific overlay + FIPS 140-3 Level 3 + air-gapped enforcement | + +### 7.2 Compliance overlay activation + +Compliance domains are activated additively: + +```yaml +active_profile: prod +active_compliance_domains: [hipaa, pci_dss] +``` + +DCM applies the union of profile defaults and all active compliance domain +overlays. Compliance overlays always tighten (never relax) the base profile +(`CPX-011`). + +### 7.3 Standard-to-DCM-feature mapping + +| Standard | DCM features | +|---|---| +| NIST SP 800-53 | Policy domains map to control families; access control, audit, configuration management | +| FedRAMP Moderate | Profile `fedramp_moderate`; NIST baseline; FIPS 140-2 Level 1+ | +| FedRAMP High | Profile `fedramp_high`; NIST baseline; FIPS 140-2 Level 2+; enhanced audit retention | +| DoD IL4 | Profile `dod_il4`; FIPS 140-2 Level 2; hardware attestation; enhanced logging | +| PCI DSS v4 | Req 8.3.9: P90D max credential rotation enforced; segmentation via sovereignty; cardholder data logging; 12-month audit retention | +| HIPAA | Profile `fsi` or `hipaa` overlay; PHI access logging; minimum necessary RBAC; transmission security TLS 1.2+; workforce MFA | +| SOC 2 | profile standard+; Type II audit trail; availability/security/confidentiality; GitOps change management | +| ISO 27001 | All profiles; risk-based approach; asset management; access control; cryptography | +| GDPR | Sovereignty constraints; data classification; right to erasure (entity decommission + audit retention) | +| Schrems II | Sovereignty + federation boundaries; data transfer restrictions | + +### 7.4 Authentication Assurance Levels (NIST SP 800-63B) + +DCM maps profiles to AAL: + +| Profile | AAL | Requirements | +|---|---|---| +| `minimal` | AAL1 | Single-factor; password or API key | +| `dev` | AAL1 | Single-factor | +| `standard` | AAL2 | MFA required for actor sessions | +| `prod` | AAL2 | MFA required; TOTP, FIDO2, or hardware token | +| `fsi` | AAL2+ | MFA required; phishing-resistant (FIDO2/hardware) | +| `sovereign` | AAL3 | Hardware-based authenticator; verifier impersonation resistance | + +--- + +## 8. ITSM integration standards + +| Standard | Use | Obligation | +|---|---|---| +| ServiceNow REST Table API | Primary ServiceNow integration | Normative for ServiceNow provider | +| Jira REST API v3 | Primary Jira Service Management | Normative for Jira provider | +| BMC AR REST API v1 | BMC Remedy/Helix ITSM | Normative for BMC provider | +| PagerDuty Events API v2 | Incident creation for alert-type integrations | Normative for PagerDuty | +| HMAC-SHA256 | Inbound webhook signature verification | Normative | +| ITIL v4 Change Management | DCM change record lifecycle mapping | Informative | +| JSON:API | Several ITSM REST APIs response formatting | Informative | +| JSONPath | Template expression for `generic_rest` action payloads | Normative for generic_rest | + +See [`../architecture/integrations/itsm.md`](../architecture/integrations/itsm.md) +for the ITSM integration implementation. + +--- + +## 9. CNCF ecosystem + +DCM is designed for CNCF ecosystem compatibility: + +| Project | CNCF Status | DCM Use | +|---|---|---| +| Kubernetes | Graduated | Deployment target; DCM Operator | +| Open Policy Agent (OPA) | Graduated | Policy evaluation engine; Rego for GateKeeper, Validation | +| Prometheus | Graduated | Metrics exposition | +| OpenTelemetry | Graduated | Distributed tracing; correlation ID propagation | +| Istio | Graduated | Optional service mesh for internal mTLS | +| Argo CD / Flux | Graduated | GitOps delivery for DCM layers/policies | + +--- + +## 10. Policy-family-to-standard mapping (DCM realization) + +| Policy family | Standards basis | +|---|---| +| `AUTH-001-DCM` through `AUTH-015-DCM` | RFC 6749, RFC 7519, OIDC Core, NIST SP 800-63B, RFC 7643/7644 | +| `SES-001-DCM` through `SES-005-DCM` | RFC 7662, RFC 7009, OAuth 2.0 best practices | +| `CPX-001-DCM` through `CPX-012-DCM` | FIPS 140-2/3, RFC 5280, RFC 8555/7030/8894/4210, NIST SP 800-57 | +| `ATM-001-DCM` through `ATM-012-DCM` | ISO 27001 change management | +| `EVT-001-DCM` through `EVT-007-DCM` | OpenTelemetry, CNCF event-driven best practices | +| `VER-001-DCM` through `VER-009-DCM` | RFC 8594, RFC 9745, industry API lifecycle | +| `ICOM-001-DCM` through `ICOM-009-DCM` | RFC 8446, RFC 5280, SPIFFE conceptual, FIPS 140 | +| `HLT-001-DCM` through `HLT-006-DCM` | RFC 8615, K8s probe conventions, Prometheus OpenMetrics | +| `ZTS-001-DCM` through `ZTS-005-DCM` | Zero Trust Architecture (NIST SP 800-207), NIST SP 800-63B | + +The full list of policy IDs lives in each subsystem document under +`../architecture/`. diff --git a/reference/operational-reference.md b/reference/operational-reference.md new file mode 100644 index 0000000..f2ca614 --- /dev/null +++ b/reference/operational-reference.md @@ -0,0 +1,521 @@ +# DCM Data Model โ€” Operational Reference + +**Document Status:** ๐Ÿ”„ In Progress +**Document Type:** SRE Reference โ€” GitOps Scale, Store Migration, Disaster Recovery +**Related Documents:** [data stores](https://github.com/croadfeldt/udlm/blob/main/contracts/storage-providers.md) | [Deployment and Redundancy](../architecture/runtime-features/deployment-redundancy.md) | [Four States Model](https://github.com/croadfeldt/udlm/blob/main/foundations/four-states.md) | [Internal Component Authentication](../architecture/control-plane/internal-component-auth.md) | [DCM Self-Health Endpoints](../architecture/control-plane/self-health.md) + +> **Audience:** Platform engineers and SREs operating DCM in production. This document covers three operational concerns that require guidance beyond the architectural specifications: GitOps store partitioning at large scale, migrating between store implementations, and recovering from failure scenarios. + +--- + +## 1. DCM database Scale and Partitioning + +### 1.1 When a Repo Becomes Too Large + +The default DCM GitOps layout uses one repository per store type (Intent, Requested, Layer, Policy). For most deployments this is correct. At large scale โ€” tens of thousands of active entities or hundreds of active tenants โ€” a single repository can exhibit: + +- Git operation latency (clone, fetch, log) growing beyond SLA +- CI/CD pipeline fan-out delays as every write triggers the full repository +- Access control granularity limits (all tenants share one repo) +- Search index sync lag from large diffs + +**Thresholds that suggest partitioning:** + +| Signal | Threshold | Recommended action | +|--------|-----------|-------------------| +| Entities in Intent/Requested store | > 50,000 active | Consider tenant-shard partitioning | +| Git clone time | > PT30S | Add shallow-clone depth; consider partitioning | +| PR merge latency | > PT5M | Partition or add write buffer | +| Tenant count | > 500 | Consider per-tenant repositories | +| Repository size on disk | > 10 GB | Partition | + +These are guidelines, not hard limits. Profile, hardware, and Git host performance all affect the actual inflection point. + +### 1.2 Partitioning Strategies + +DCM supports three partitioning strategies. All are compatible with the storage contract โ€” partitioning changes how stores are organized, not what the store contract requires. + +#### Strategy A โ€” Tenant Shard Partitioning (recommended for most) + +Split each store type into N shard repositories, with tenants assigned to shards by a deterministic hash of `tenant_uuid`: + +``` +shard = hash(tenant_uuid) % N + +dcm-intent-shard-0/ โ† tenants whose hash(uuid) % N == 0 +dcm-intent-shard-1/ โ† tenants whose hash(uuid) % N == 1 + tenants/ + {tenant-uuid}/ + requests/ + {request-uuid}/ + intent.yaml +``` + +**DCM configuration:** +```yaml +gitops_store: + intent_store: + partitioning: tenant_shard + shard_count: 8 + shard_routing: hash_mod # deterministic; no routing table needed + repositories: + - shard: 0 + url: https://git.corp/dcm/dcm-intent-shard-0 + - shard: 1 + url: https://git.corp/dcm/dcm-intent-shard-1 + # ... +``` + +**Operational implications:** Adding shards requires re-hashing. Plan shard counts for 3โ€“5 years of expected growth; use a power of 2 to simplify future doubling. + +#### Strategy B โ€” Per-Tenant Repositories (for strict isolation) + +Each tenant has its own set of store repositories. Used when: +- Tenants are separate organizations (MSP model) +- Compliance requires complete data isolation per tenant +- Different retention policies per tenant + +``` +dcm-intent-{tenant-uuid}/ โ† one repository per tenant + requests/ + {request-uuid}/ + intent.yaml +``` + +**Operational implications:** Repository count scales with tenant count. Requires automation for tenant onboarding (repository creation, access provisioning). Git host must support large numbers of repositories. + +#### Strategy C โ€” Time-Based Archiving (for retention management) + +Active entities stay in the primary repository. Entities past a declared age threshold are archived to read-only archive repositories: + +``` +dcm-intent-active/ โ† current entities (hot) +dcm-intent-archive-2025/ โ† entities from 2025 (cold, read-only) +dcm-intent-archive-2024/ โ† entities from 2024 (cold, read-only) +``` + +DCM's audit and search components are configured with both active and archive repository lists. Strategy C is typically combined with Strategy A or B. + +### 1.3 Large-Scale Layer Store Partitioning + +The Layer Store grows more slowly than the Intent/Requested stores (layers are reused across requests). Layer Store partitioning is by domain rather than by tenant: + +``` +dcm-layers-compute/ โ† Compute.* resource type layers +dcm-layers-network/ โ† Network.* resource type layers +dcm-layers-storage/ โ† Storage.* resource type layers +dcm-layers-platform/ โ† Platform.* and cross-cutting layers +dcm-policies-core/ โ† System and core policies +dcm-policies-tenant/ โ† Tenant-contributed policies (per tenant or sharded) +``` + +Domain-partitioned Layer stores are configured in DCM's layer assembly engine: + +```yaml +layer_store: + repositories: + - domain: Compute.* + url: https://git.corp/dcm/dcm-layers-compute + priority: provider_contribution # provider contributions go here + - domain: Network.* + url: https://git.corp/dcm/dcm-layers-network + - domain: "*" # catch-all for uncategorized + url: https://git.corp/dcm/dcm-layers-platform +``` + +### 1.4 Shallow Clones and Read-Only Mirrors + +For read-heavy operations (audit, search index rebuild, drift reconciliation) that do not need full Git history: + +```yaml +gitops_store: + read_operations: + clone_depth: 1 # shallow clone for read-only consumers + use_mirror: true # read from read-only mirror; writes go to primary + mirror_url: https://git-mirror.corp/dcm/ + mirror_sync_lag_max: PT5M # alert if mirror is more than 5 minutes behind +``` + +--- + +## 2. Store Migration + +### 2.1 Migration Principles + +DCM store migrations follow three invariants: + +1. **No data loss** โ€” every record in the source store must exist in the target store after migration +2. **Audit chain continuity** โ€” the audit hash chain must be unbroken across the migration; audit records written before and after must chain correctly +3. **Read availability during migration** โ€” DCM continues serving read requests throughout; write availability may be briefly paused during cutover + +### 2.2 Migration Playbook Structure + +Every store migration follows this pattern regardless of source or target implementation: + +``` +Phase 1 โ€” Prepare + โ”‚ Provision target store alongside source + โ”‚ Validate target store meets storage contract (health check, write test, read test) + โ”‚ Configure DCM to write to BOTH source and target (dual-write mode) + โ”‚ +Phase 2 โ€” Backfill + โ”‚ Export all existing records from source + โ”‚ Import records to target in chronological order (preserving provenance timestamps) + โ”‚ Verify record counts match; spot-check content hashes + โ”‚ +Phase 3 โ€” Validate + โ”‚ Run DCM's store validation suite against target + โ”‚ Verify audit chain integrity on target store + โ”‚ Verify search index can be rebuilt from target store + โ”‚ +Phase 4 โ€” Cutover + โ”‚ Brief write pause (PT30Sโ€“PT5M depending on profile) + โ”‚ Disable dual-write; switch DCM to target as primary + โ”‚ Verify /readyz returns healthy + โ”‚ Resume writes to target only + โ”‚ +Phase 5 โ€” Decommission source (after burn-in period) + Default burn-in: P30D (standard), P90D (fsi/sovereign) + Keep source in read-only mode during burn-in for rollback +``` + +### 2.3 Common Migration Paths + +#### SQLite โ†’ PostgreSQL (minimal/dev โ†’ standard) + +Typical trigger: scaling beyond single-node evaluation environment. + +```bash +# Step 1: Export from SQLite +dcm-admin store export \ + --store realized \ + --format jsonl \ + --output realized-export.jsonl + +# Step 2: Import to PostgreSQL +dcm-admin store import \ + --store realized \ + --source realized-export.jsonl \ + --target postgres://pg-host:5432/dcm_realized \ + --validate-chain + +# Step 3: Enable dual-write +dcm-admin store dual-write enable \ + --store realized \ + --primary sqlite://dcm-realized.db \ + --secondary postgres://pg-host:5432/dcm_realized + +# Step 4: Validate +dcm-admin store validate \ + --store realized \ + --target postgres://pg-host:5432/dcm_realized \ + --check-count --check-chain --check-spot-sample 0.05 + +# Step 5: Cutover +dcm-admin store cutover \ + --store realized \ + --target postgres://pg-host:5432/dcm_realized +``` + +#### PostgreSQL single-instance โ†’ CockroachDB / PostgreSQL HA + +Typical trigger: HA requirement for production; multi-region deployment. + +**Key difference from SQLite โ†’ PostgreSQL:** CockroachDB uses serializable isolation and distributed transactions. Test write throughput under realistic load before cutover โ€” CockroachDB's latency profile differs from single-node PostgreSQL. + +```yaml +# Pre-migration checklist +migration_checklist: + - Load test target under realistic DCM write volume (PT4H minimum) + - Verify CockroachDB schema compatibility (DCM uses standard PostgreSQL wire protocol) + - Configure connection pooler (PgBouncer or similar) โ€” CockroachDB default connection count + - Verify time synchronization (CockroachDB requires NTP within PT500MS across nodes) + - Test audit chain write under partition scenario +``` + +#### DCM database โ€” Repo Restructuring + +Restructuring a GitOps repository (e.g. monorepo to sharded) requires special handling because Git history must be preserved. + +``` +Step 1: Enable write buffer โ€” all new writes queue while migration proceeds +Step 2: git filter-repo or git subtree to extract tenant directories to shard repos +Step 3: Validate file counts and content hashes in each shard +Step 4: Update DCM gitops_store configuration to point to shards +Step 5: Drain write buffer โ€” queued writes replay to new shard repos +Step 6: Verify search index rebuild from shards +Step 7: Archive or delete monorepo after burn-in period +``` + +### 2.4 Rollback Procedure + +If migration fails before cutover: disable dual-write, discard target, no impact to production. + +If migration fails after cutover (during burn-in): + +``` +1. Alert: /readyz reports degraded or source store discrepancy detected +2. dcm-admin store rollback --store --to source + (requires source still in read-only mode โ€” NOT decommissioned) +3. DCM restarts reads/writes from source +4. Export any writes that reached target but not source (if any, during dual-write gap) +5. Import gap records to source +6. Re-enable source as primary +``` + +**This is why burn-in period exists.** Do not decommission source stores until burn-in completes. + +### 2.5 Profile-Governed Migration Constraints + +| Profile | Min dual-write duration | Max cutover pause | Burn-in period | +|---------|------------------------|-------------------|----------------| +| `minimal` | P1D | PT5M | P7D | +| `dev` | P3D | PT5M | P14D | +| `standard` | P7D | PT2M | P30D | +| `prod` | P14D | PT1M | P30D | +| `fsi` | P30D | PT30S | P90D | +| `sovereign` | P60D | PT30S | P90D | + +--- + +## 3. Disaster Recovery Runbook + +### 3.1 DCM Recovery Architecture + +DCM's recovery model is built on a key property: **all durable state is in the stores, not in the control plane.** Control plane components (Policy Engine, Request Orchestrator, etc.) are stateless and can be restarted without data loss. Recovery from most failures is component restart, not data restoration. + +The five DCM stores and their recovery characteristics: + +| Store | Implementation | Data durability | Recovery method | +|-------|---------------|----------------|-----------------| +| Intent Store | GitOps (Git) | Git replication + remote | Re-clone from remote | +| Requested Store | GitOps or write-once | Git replication / DB replication | Re-clone or DB restore | +| Layer Store | GitOps (Git) | Git replication + remote | Re-clone from remote | +| Realized Store | Write-once (PostgreSQL/CockroachDB) | DB replication / WAL | DB failover or restore | +| Audit Store | Append-only (Kafka/PostgreSQL) | Replication / WAL | Kafka failover or restore | + +### 3.2 Recovery Scenarios and Procedures + +#### Scenario 1: Single Component Failure (Most Common) + +**Symptoms:** One DCM component (e.g. Policy Engine) is unhealthy. `/readyz` shows degraded. Requests may be delayed but not lost. + +**RTO:** PT5M +**RPO:** 0 (no data loss โ€” components are stateless) + +``` +1. Identify failing component via GET /api/v1/admin/health +2. Check component logs for panic/OOM/deadlock +3. Kubernetes: pod restart is automatic (liveness probe) + Manual: kubectl rollout restart deployment/dcm-policy-engine +4. Monitor /readyz โ€” should recover within PT2M of pod restart +5. If component repeatedly fails: check Internal CA cert expiry (ICOM-006) + dcm-admin component cert-status --component policy-engine +6. Write post-incident note to DCM audit store +``` + +#### Scenario 2: Store Failure (Database / Kafka) + +**Symptoms:** `/readyz` fails specific store check. Requests queue or fail depending on which store. + +**Realized Store failure (highest severity โ€” blocks realization):** + +``` +RTO target: PT30M (standard), PT15M (prod), PT5M (fsi/sovereign) +RPO: 0 for PostgreSQL HA (synchronous replication); near-zero for async + +1. Confirm store failure: GET /api/v1/admin/health โ†’ realized_store: fail +2. If HA: check if automatic failover occurred + kubectl get pods -n dcm-stores | grep postgres + Check PostgreSQL replication lag / CockroachDB node status +3. Manual failover if automatic did not trigger: + dcm-admin store failover --store realized --target replica-2 +4. Verify replication caught up: dcm-admin store lag --store realized +5. Verify /readyz recovers +6. Root cause analysis: WAL lag, disk full, network partition +``` + +**Audit Store failure:** + +``` +RTO: PT1H acceptable (audit trail can tolerate temporary buffering) +RPO: profile-governed โ€” see Audit Store write buffer policy + +1. DCM buffers audit records locally (Commit Log) during store outage + Write buffer capacity: profile-governed (PT1H standard, PT15M sovereign) +2. Restore Kafka cluster from replica or snapshot +3. DCM drains buffer to restored store automatically on reconnection +4. Verify chain integrity: dcm-admin audit chain-verify --since +``` + +**DCM database failure (Intent/Requested/Layer):** + +``` +RTO: PT30M (stores are remountable from Git remote) +RPO: 0 (all writes go to Git remote; loss only if remote is also lost) + +1. Git remote unreachable: check network connectivity +2. If Git host is down: DCM switches to cached/buffered mode + New requests queue in write buffer; reads served from local clone +3. Write buffer capacity: PT4H (standard) โ€” configure per deployment +4. When Git host recovers: buffer drains automatically +5. Force drain: dcm-admin store drain-buffer --store intent +``` + +#### Scenario 3: Full Control Plane Loss + +**Symptoms:** All DCM pods down. Stores intact. Users cannot submit requests. + +**RTO:** PT15M (kubernetes deployment restart) +**RPO:** 0 (stores are external โ€” no data in pods) + +``` +1. Verify stores are healthy (connect directly): + dcm-admin store health-check --all --direct + +2. Verify Internal CA is available: + curl -k https://dcm-internal-ca.dcm-system.svc.cluster.local/health + +3. Restart DCM deployment (Kubernetes): + kubectl rollout restart deployment -n dcm-system + +4. Monitor /readyz โ€” startup sequence should complete within PT3M: + watch -n 5 kubectl get pods -n dcm-system + +5. Verify session store recovers: + GET /api/v1/admin/health โ†’ session_store: pass + +6. Alert consumers: any in-flight requests at time of failure + are in ACKNOWLEDGED/DISPATCHED state and may need status check + dcm-admin requests find --status in-flight --since +``` + +#### Scenario 4: Partial Region Loss (Multi-Region Deployments) + +**Symptoms:** One region's DCM instance degraded. Other regions operational. + +``` +1. DCM federation routes requests away from degraded region (automatic) + Verify: GET /api/v1/admin/health โ†’ federation peer status + +2. If region is sovereign-scoped (data must not leave): + Alert: sovereignty.migration_required event fires + Consumers in that region may be blocked until region recovers + +3. For non-sovereign regions: traffic reroutes automatically + Monitor: dcm_requests_total{region} for traffic shift + +4. Region recovery: standard Scenario 3 procedure + After recovery: drift detection validates recovered state +``` + +#### Scenario 5: Complete Loss (Repave) + +The nuclear scenario: entire DCM installation destroyed. Git remote intact. + +**RTO:** PT4Hโ€“PT24H (depends on infrastructure provisioning speed) +**RPO:** 0 for GitOps stores; near-zero for Realized/Audit stores + +``` +1. Provision new Kubernetes cluster (or equivalent) + +2. Deploy DCM bootstrap installer: + helm install dcm-bootstrap dcm/dcm-bootstrap \ + --set gitops.manifest_url=https://git.corp/dcm/dcm-deployment \ + --set gitops.manifest_ref= + +3. DCM bootstrap reads dcm_deployment manifest from Git + Provisions itself: control plane, Internal CA, stores + +4. Restore Realized Store from backup: + dcm-admin store restore --store realized \ + --from s3://dcm-backups/realized/latest \ + --validate-chain + +5. Restore Audit Store from backup or Kafka snapshot: + dcm-admin store restore --store audit \ + --from s3://dcm-backups/audit/latest \ + --chain-verify + +6. Intent/Requested/Layer stores: re-clone from Git remote (already current) + +7. DCM rehydrates managed resources in dependency order: + dcm-admin rehydrate --all-tenants --dry-run # verify plan first + dcm-admin rehydrate --all-tenants + +8. Drift detection validates recovered state matches declared state: + dcm-admin drift scan --all --post-recovery + +9. Re-issue Internal CA certificates for all components: + (handled automatically by bootstrap โ€” components acquire new certs) + +10. Notify consumers: recovery complete; request status available +``` + +### 3.3 Recovery Time Objectives by Profile + +| Profile | Scenario 1 (component) | Scenario 2 (store) | Scenario 3 (full CP) | Scenario 5 (repave) | +|---------|------------------------|--------------------|-----------------------|---------------------| +| `minimal` | PT15M | PT2H | PT30M | PT24H | +| `standard` | PT5M | PT30M | PT15M | PT8H | +| `prod` | PT2M | PT15M | PT10M | PT4H | +| `fsi` | PT2M | PT5M | PT5M | PT2H | +| `sovereign` | PT1M | PT5M | PT5M | PT2H | + +### 3.4 Recovery Point Objectives + +| Store | Standard RPO | fsi/sovereign RPO | Notes | +|-------|-------------|------------------|-------| +| Intent Store | 0 | 0 | Git remote is source of truth | +| Requested Store | 0 | 0 | Write-once; replicated | +| Layer Store | 0 | 0 | Git remote is source of truth | +| Realized Store | PT5M | PT1M | Async replication lag | +| Audit Store | PT15M | PT1M | Kafka replication + write buffer | + +### 3.5 Backup Schedule + +DCM does not manage backups of infrastructure stores directly โ€” that responsibility belongs to the storage platform. Recommended schedules by store: + +| Store | Backup method | Frequency | Retention | +|-------|--------------|-----------|-----------| +| Intent / Requested / Layer | Git push to offsite remote | Continuous | Per Git host policy | +| Realized Store | PostgreSQL PITR + daily snapshot | Continuous WAL + P1D snapshot | P90D (standard), P365D (fsi/sovereign) | +| Audit Store | Kafka topic snapshot | P4H | P365D (all profiles โ€” regulatory minimum) | +| Internal CA | Key material backup to HSM/Vault | On change | P7Y (key material outlives certs) | + +### 3.6 Post-Recovery Validation Checklist + +Run after any Scenario 3+ recovery: + +``` +โ–ก /livez returns pass on all control plane pods +โ–ก /readyz returns pass (all 5 core dependencies green) +โ–ก GET /api/v1/admin/health shows all components pass +โ–ก dcm-admin audit chain-verify --full returns no broken links +โ–ก dcm-admin store validate --all returns no discrepancies +โ–ก dcm-admin drift scan --all returns no unexpected drift +โ–ก Internal CA certificates valid for all components (ICOM-006) +โ–ก At least one Auth Provider healthy (GET /api/v1/admin/health โ†’ auth_providers) +โ–ก Search index rebuild complete (if search index store was affected) +โ–ก Session Store empty (expected โ€” all sessions expired during outage; users re-authenticate) +โ–ก Write post-incident note to audit store with recovery timeline +โ–ก Notify consumers of recovery completion +``` + +--- + +## 4. System Policies + +| Policy | Rule | +|--------|------| +| `OPS-001` | GitOps store partitioning strategy must be declared in the DCM deployment manifest. Changes to partitioning strategy require dual-write migration procedure (Section 2). | +| `OPS-002` | Store migrations must maintain audit chain continuity across cutover. Audit records written to the source store before cutover and to the target store after cutover must form an unbroken chain. | +| `OPS-003` | Source stores must remain accessible in read-only mode for the profile-governed burn-in period after cutover. Source stores must not be decommissioned until the burn-in period completes and rollback is confirmed unnecessary. | +| `OPS-004` | Recovery from Scenario 3 (full control plane loss) must complete within the profile-governed RTO. If RTO cannot be met, the incident must be escalated and root cause must address the recovery path. | +| `OPS-005` | The post-recovery validation checklist (Section 3.6) must be completed and its results written to the audit store before declaring an incident resolved. | +| `OPS-006` | Audit Store backups must be retained for a minimum of P365D in all profiles, regardless of other data retention policies, to satisfy regulatory audit trail requirements. | +| `OPS-007` | Git remote repositories serving as GitOps stores must be configured with push access from at least two geographically separated locations to prevent single-point-of-failure data loss. | + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).*