diff --git a/architecture/runtime-features/deployment-redundancy.md b/architecture/runtime-features/deployment-redundancy.md new file mode 100644 index 0000000..86fa235 --- /dev/null +++ b/architecture/runtime-features/deployment-redundancy.md @@ -0,0 +1,797 @@ +--- +Document Status: ✅ Complete +Document Type: Architecture Reference — Deployment and Redundancy +--- + +# DCM Data Model — Deployment and Redundancy Model + +> **DCM-native runtime feature; no single UDLM contract counterpart.** +> Deployment topology and redundancy are realization-layer concerns. UDLM +> does not mandate a deployment model — a peer DCM realization could choose a +> different redundancy strategy and still satisfy every UDLM contract. This +> document specifies DCM's own deployment and redundancy approach. + + +**Document Status:** ✅ Complete +**Related Documents:** [Context and Purpose](https://github.com/croadfeldt/udlm/blob/main/foundations/context-and-purpose.md) | [data stores](https://github.com/croadfeldt/udlm/blob/main/contracts/storage-providers.md) | [Universal Audit Model](https://github.com/croadfeldt/udlm/blob/main/observability/universal-audit.md) | [Policy Organization](../governance-enforcement/policy-profiles.md) + +> **Foundation Document Reference** +> +> This document is a detailed reference for a specific domain of the DCM architecture. +> The three foundational abstractions — Data, Provider, and Policy — are defined in +> [udlm/foundations/foundations.md](https://github.com/croadfeldt/udlm/blob/main/foundations/foundations.md). All concepts in this document map to one or +> more of those three abstractions. +> See also: [Provider Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/provider-contract.md) | [Policy Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/policy-contract.md) +> +> **This document maps to: DATA + PROVIDER** +> +> Data: deployment specification. Provider: data store redundancy + + + +--- + +> **Operational guidance:** GitOps the disaster recovery runbook and RTO/RPO tables are in [Operational Reference](../../reference/operational-reference.md) Section 3. + +## 1. Purpose + +Every DCM component, every data store, and every capability is designed for redundancy by default. Redundancy is not an add-on or an advanced configuration — it is the baseline operational posture for all profiles above `minimal`. + +**The minimal profile** provides single-instance deployment for home lab and evaluation simplicity. All other profiles assume redundancy as the floor. The transition from `minimal` to `dev` is the transition from "works on my laptop" to "survives a node failure." + +**Everything in DCM runs as a container in a pod.** No bare-metal DCM components. No special-case deployment paths. Every DCM component follows the same container lifecycle, the same health check model, the same rolling update pattern. DCM runs on Kubernetes — and manages Kubernetes. + +**DCM is self-hosting.** DCM's own deployment is expressible as DCM resources. DCM can manage its own lifecycle, detect drift in its own components, and rehydrate its own deployment from Git. This is the ultimate expression of the data center repave use case — DCM restoring itself. + +--- + +## 2. Design Principles + +**Redundant by default.** Every component, store, and capability has a redundancy model. The `minimal` profile sets `replicas: 1`. Every other profile sets `replicas: >= 3` with quorum writes and anti-affinity scheduling. + +**Everything containerized.** All DCM components run as containers in Kubernetes pods. No exceptions. This gives a consistent deployment model, rolling updates, health checks, and self-healing as first-class properties. + +**Profile-governed redundancy.** Replica counts, quorum thresholds, geo-replication, and anti-affinity requirements are declared by the active Profile — not by per-component configuration. Activating a Profile configures redundancy for the entire deployment. + +**Self-hosting.** DCM's own deployment is a DCM resource. DCM manages itself through the same model it uses to manage customer infrastructure. + +**Stateless control plane.** All DCM control plane components are stateless — all state lives in external stores. Any component instance can fail and be replaced without data loss. State recovery means restarting a pod — not restoring a database. + +**Quorum writes for durability.** All durable stores use quorum writes — a write is confirmed only when a majority of replicas acknowledge it. This ensures durability even if a minority of replicas fail simultaneously. + +--- + +## 3. Component Redundancy Model + +### 3.1 Control Plane Components + +All control plane components are stateless, horizontally scalable, and deployed as Kubernetes Deployments with configurable replica counts. + +```yaml +component_redundancy: + component: request_payload_processor # same model for all components + deployment_type: kubernetes_deployment + replicas: 3 # set by active Profile + affinity: + anti_affinity: required # pods spread across nodes + zone_spread: preferred # prefer spreading across availability zones + disruption_budget: + min_available: 2 # always keep 2 running during rolling updates + health_check: + liveness: + path: /healthz + interval_seconds: 10 + failure_threshold: 3 + readiness: + path: /readyz + interval_seconds: 5 + failure_threshold: 2 + rolling_update: + strategy: RollingUpdate + max_unavailable: 0 # never take a pod down before replacement is ready + max_surge: 1 +``` + +**Control plane components:** + +| Component | Stateless? | Replica Model | +|-----------|-----------|--------------| +| API Gateway | Yes | Deployment + HorizontalPodAutoscaler | +| Request Payload Processor | Yes | Deployment | +| Policy Engine (OPA) | Yes | Deployment + PolicyBundle sidecar | +| Placement Engine | Yes | Deployment | +| Service Catalog | Yes | Deployment | +| IDM / IAM | Yes | Deployment (external IdP recommended) | +| Audit Forward Service | Yes | Deployment (1 active + 1 standby) | +| Lifecycle Constraint Enforcer | Yes | Deployment (leader election) | +| Drift Detection | Yes | Deployment (leader election for scheduling) | +| Resource Discovery | Yes | Deployment (leader election) | +| Message Bus Router | Yes | Deployment | + +**Leader election** for scheduler-type components (Lifecycle Constraint Enforcer, Drift Detection, Resource Discovery): multiple replicas run but only one holds the leader lease at a time. On leader failure, a replica acquires the lease within seconds. + +### 3.2 Data Store Redundancy + +All DCM data stores run as containers. Each store type has a declared replication and quorum model. + +#### Commit Log + +```yaml +commit_log: + replicas: 3 + write_quorum: 2 # confirmed durable when 2/3 replicas acknowledge + read_quorum: 1 # any replica can serve reads + affinity: + zone_spread: required # replicas MUST span availability zones + implementation: etcd # or equivalent consensus store + # etcd is purpose-built for this pattern: Raft consensus, quorum writes, + # sub-millisecond local writes, proven in Kubernetes itself +``` + +The Commit Log uses consensus protocol (Raft/equivalent). A write is confirmed when the quorum acknowledges — ensuring durability even if a minority of replicas fail simultaneously. + +#### DCM database (Intent, Requested, Layers, Policies) + +```yaml +gitops_store: + implementation: gitea # or equivalent self-hosted Git + replicas: 3 + replication_mode: active_active # any node can accept writes + write_quorum: 2 + backup: + enabled: true + schedule: "0 */6 * * *" # every 6 hours + retention: 30d +``` + +#### pipeline_events table (Realized, Discovered, Audit Events) + +```yaml +event_stream: + implementation: kafka # or equivalent + brokers: 3 + replication_factor: 3 + min_insync_replicas: 2 # minimum replicas that must acknowledge a write + partitions: 12 # enables parallel consumption + retention: + bytes: -1 # unlimited — retention governed by policy + ms: -1 # unlimited +``` + +#### Audit Store + +```yaml +audit_store: + implementation: elasticsearch # or equivalent — optimized for queryable retention + replicas: 3 + primary_shards: 5 + replica_shards: 1 # each shard has 1 replica = 2 copies total + geo_replicated: true # in prod/fsi/sovereign profiles + append_only_enforced: true # storage layer enforces immutability +``` + +#### Search Index (Non-Authoritative) + +```yaml +search_index: + implementation: elasticsearch + replicas: 2 # lower redundancy — can rebuild from Git + rebuild_from_git: true # on data loss, rebuild from authoritative stores +``` + +### 3.3 Container Specification + +Every DCM component pod follows a common security and resource model: + +```yaml +pod_spec: + security_context: + run_as_non_root: true + run_as_user: 65534 # nobody + run_as_group: 65534 + fs_group: 65534 + seccomp_profile: + type: RuntimeDefault + capabilities: + drop: [ALL] + + containers: + - name: + image: ghcr.io/dcm-project/: + image_pull_policy: IfNotPresent + security_context: + read_only_root_filesystem: true + allow_privilege_escalation: false + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + cpu: 2000m + memory: 2Gi + liveness_probe: + readiness_probe: + volume_mounts: + - name: tmp + mount_path: /tmp # writable temp — no other writable paths + + volumes: + - name: tmp + empty_dir: {} +``` + +--- + +## 4. Redundancy by Profile + +Profile activation configures redundancy for the entire deployment. Organizations do not configure replica counts individually — they activate a profile. + +```yaml +# Redundancy matrix per profile +redundancy_matrix: + minimal: + control_plane_replicas: 1 + store_replicas: 1 + write_quorum: false + zone_spread: false + geo_replication: false + anti_affinity: false + note: "Single-instance. No redundancy. Home lab and evaluation only." + + dev: + control_plane_replicas: 1 + store_replicas: 1 + write_quorum: false + zone_spread: false + geo_replication: false + anti_affinity: false + note: "Single-instance with backup. Basic resilience for dev environments." + + standard: + control_plane_replicas: 3 + store_replicas: 3 + write_quorum: 2 # 2 of 3 + zone_spread: preferred + geo_replication: false + anti_affinity: required + note: "Production baseline. Survives single node or zone failure." + + prod: + control_plane_replicas: 3 + store_replicas: 3 + write_quorum: 2 + zone_spread: required + geo_replication: true + anti_affinity: required + sla_disruption_budget: "always 2 replicas available" + note: "Production with SLA. Geo-replicated stores." + + fsi: + control_plane_replicas: 5 + store_replicas: 5 + write_quorum: 3 # 3 of 5 + zone_spread: required + geo_replication: true + anti_affinity: required + audit_store_replicas: 5 + note: "FSI-grade. Higher quorum threshold. Compliance-grade audit." + + sovereign: + control_plane_replicas: 5 + store_replicas: 5 + write_quorum: 3 + zone_spread: required + geo_replication: true # within sovereignty boundary only + anti_affinity: required + air_gap_backup: true + sovereignty_boundary_enforced: true + note: "Maximum. All geo-replication within sovereignty boundary." +``` + +--- + +## 5. The DCM Deployment Specification + +The DCM deployment itself is a DCM resource — declared in YAML, stored in Git, governed by Policy, subject to the same four-state lifecycle as any other resource. + +```yaml +dcm_deployment: + artifact_metadata: + uuid: + handle: "deployments/primary/dcm-control-plane" + version: "1.0.0" + status: active + + profile: system/profile/standard + kubernetes_namespace: dcm-system + + # Redundancy — set by active Profile, overridable per component + redundancy: + control_plane: + replicas: 3 + affinity: + anti_affinity: required + zone_spread: preferred + disruption_budget: + min_available: 2 + + stores: + commit_log: + replicas: 3 + write_quorum: 2 + implementation: etcd + gitops_store: + replicas: 3 + write_quorum: 2 + implementation: gitea + event_stream: + brokers: 3 + replication_factor: 3 + min_insync_replicas: 2 + implementation: kafka + audit_store: + replicas: 3 + geo_replicated: false # standard profile default + implementation: elasticsearch + search_index: + replicas: 2 + implementation: elasticsearch + + # Provider health configuration + providers: + health_check_interval_seconds: 30 + unhealthy_threshold: 3 + automatic_failover: true + + # Self-hosting: DCM manages its own deployment + self_managed: true + drift_detection_enabled: true + rehydration_enabled: true + # DCM detects if its own components drift from declared spec + # and can rehydrate (redeploy) from this declaration +``` + +--- + +## 6. Self-Hosting — DCM Managing Itself + +DCM's own deployment is managed through the same model it uses to manage customer infrastructure. This is the **self-hosting principle** — DCM eats its own cooking. + +### 6.1 What Self-Hosting Means + +- DCM control plane components are defined as Resource Entities in DCM +- DCM data stores are defined as data store resources in DCM +- DCM's own Policy Groups govern DCM's own deployment constraints +- DCM runs drift detection on its own components — a component running the wrong image version is drift +- DCM can rehydrate its own deployment from the `dcm_deployment` declaration in Git + +### 6.2 The Bootstrap Problem + +DCM cannot manage itself before it exists. The bootstrap sequence: + +``` +1. Bootstrap installer deploys minimal DCM (single instance, no redundancy) + from a declarative bootstrap manifest + │ +2. Bootstrap DCM reads the target dcm_deployment declaration from Git + │ +3. Bootstrap DCM provisions itself to the target state: + - Scales from 1 to N replicas + - Provisions redundant stores + - Configures quorum + │ +4. Bootstrap instance hands off to the now-redundant DCM + │ +5. DCM manages its own lifecycle from this point forward +``` + +The bootstrap manifest is the only thing that exists outside DCM's management scope. It is minimal by design — just enough to get DCM running. + +### 6.3 Self-Hosted Drift Detection + +DCM continuously compares its own running state against the `dcm_deployment` declaration: + +| Drift Type | Example | Response | +|-----------|---------|---------| +| Wrong image version | Component running v1.1.0, declared v1.2.0 | Rolling update triggered | +| Wrong replica count | 2 replicas running, declared 3 | Scale-up triggered | +| Wrong resource limits | Component using more than declared limits | Alert + potential eviction | +| Store replication mismatch | Store has 2 replicas, declared 3 | Replication repair triggered | + +### 6.4 The Repave Scenario + +The ultimate test of self-hosting: DCM is lost entirely (ransomware, catastrophic failure). Recovery: + +``` +1. Deploy bootstrap installer to new Kubernetes cluster + │ +2. Bootstrap DCM reads dcm_deployment declaration from Git backup + │ +3. DCM provisions itself — full redundant deployment + │ +4. DCM reads all resource declarations from Git + │ +5. DCM rehydrates customer workloads in dependency order + │ +6. Drift detection validates recovered state matches declared state +``` + +The recovery time is bounded by infrastructure provisioning speed — not by backup restoration or manual configuration. Everything is code. Everything is declarative. Everything is in Git. + +--- + +## 7. Commit Log Redundancy — Two-Stage Audit Integration + +The Commit Log is the synchronous component of the two-stage audit model. In a distributed deployment, "synchronous durable write" means quorum acknowledgment: + +``` +DCM component initiates change + │ + ▼ +Write to Commit Log (Raft consensus) + │ Propose to leader + │ Leader replicates to followers + │ Write confirmed when quorum (2/3 or 3/5) acknowledge + │ + ├── Replica 1 (local node) → ACK ─┐ + ├── Replica 2 (different node) → ACK ─┤ quorum reached + └── Replica 3 (different zone) → ACK ─┘ + │ + ▼ Operation returns success (< 1ms typical with NVMe) + │ + ▼ [async — Audit Forward Service] +Read from any surviving Commit Log replica + │ Enrich → write to Audit Store + └── Clear Commit Log entry after Audit Store confirms +``` + +**Failure scenarios and recovery:** + +| Failure | During Stage 1 | Effect | +|---------|---------------|--------| +| Single replica fails before quorum | Quorum still achievable | No impact | +| Majority fail before quorum | Commit Log unavailable | Operation aborted — no silent change | +| Leader fails after quorum | New leader elected (seconds) | In-flight writes complete on new leader | +| All replicas fail after quorum | Audit Forward Service reads from backup | Full recovery on restart | +| Audit Store unavailable | Commit Log accumulates | Forward resumes when Audit Store recovers | + +--- + +## 8. Network Architecture + +> **Full internal auth specification:** See [Internal Component Authentication](../control-plane/internal-component-auth.md) for component identity model, Internal CA, bootstrap protocol, and ICOM-001–ICOM-009 system policies. + +### 8.1 Service Mesh + +All DCM component-to-component communication uses a service mesh (Istio or equivalent): +- mTLS everywhere (RFC 8446 TLS 1.3 + RFC 5280 X.509) — no plaintext internal communication +- Traffic policies enforced at mesh level +- Observability: traces, metrics, logs for all inter-component calls +- Circuit breaking: prevent cascade failures + +### 8.2 Ingress + +External traffic enters through a redundant Ingress layer: + +``` +External clients + │ + ▼ +Load Balancer (external — cloud or on-premises) + │ + ▼ +Ingress Controller (replicated — 2+ instances) + │ + ▼ +API Gateway pods (3+ instances, anti-affinity) + │ + ▼ +Internal service mesh +``` + +### 8.3 DNS and Service Discovery + +All DCM components address each other via Kubernetes Service DNS. No hardcoded IPs. Service discovery is automatic — a new pod replica is immediately addressable. + +--- + +## 9. DCM System Policies — Redundancy + +| Policy | Rule | +|--------|------| +| `RED-001` | All DCM control plane components must run as containers in Kubernetes pods | +| `RED-002` | All control plane components must be stateless — all persistent state in external stores | +| `RED-003` | In profiles above `minimal`, all control plane components must have `replicas >= 3` with anti-affinity | +| `RED-004` | All durable stores in profiles above `minimal` must use quorum writes with `write_quorum >= 2` | +| `RED-005` | The Commit Log must use consensus protocol (Raft or equivalent) with quorum writes | +| `RED-006` | The DCM deployment must be declared as a DCM resource in Git — self-hosting required | +| `RED-007` | DCM must run drift detection on its own components — version drift is treated as resource drift | +| `RED-008` | A rolling update of any DCM component must not reduce available replicas below `min_available` | +| `RED-009` | All DCM component communication must use mTLS — no plaintext internal communication | +| `RED-010` | The bootstrap manifest is the only DCM configuration outside DCM's management scope | + +--- + +## 10. Open Questions + +| # | Question | Impact | Status | +|---|----------|--------|--------| +| 1 | Should the bootstrap manifest be version-controlled and verifiable? | Bootstrap integrity | ✅ Resolved — GitOps store + hash verification at every startup; tampering prevents start; operator-signed (RED-011) | +| 2 | How does DCM handle Kubernetes cluster upgrades in sovereign deployments? | Operational | ✅ Resolved — pre-staged images via signed bundles; maintenance mode during upgrade; startup verification before resume (RED-012) | +| 3 | Should DCM support non-Kubernetes container runtimes? | Portability | ✅ Resolved — Kubernetes primary/required for production; Podman/Docker Compose for dev/community only (RED-013) | +| 4 | What is the minimum hardware specification per profile? | Implementation | ✅ Resolved — declared as DCM Resource definitions; enforced by placement engine; table documented (RED-014) | +| 5 | How does DCM's self-hosted drift detection handle DCM drifting from its own state? | Self-hosting | ✅ Resolved — DCM is DCM-managed resource; Operator reconciles; bootstrap hash provides independent check; audit hash chain externally verifiable (RED-015) | + +--- + +## 11. Related Concepts + +- **Universal Audit Model** (doc 16) — two-stage audit; Commit Log quorum model +- **Policy Organization** (doc 14) — Profile-governed redundancy configuration +- **data stores** (doc 11) — Store contracts include replication requirements +- **Four States** (doc 02) — all state stores are redundant per this model +- **Ingestion Model** (doc 13) — DCM's own deployment recovery uses the repave/rehydration pattern + + +## 8. Deployment Redundancy Gap Resolutions + +### 8.1 Bootstrap Manifest Verification (Q1) + +The bootstrap manifest is stored in the GitOps store, hash-verified at DCM startup and every restart. Tampering prevents DCM from starting. + +```yaml +bootstrap_manifest: + version: "1.0.0" + manifest_uuid: + manifest_hash: # computed at creation; verified at every startup + signed_by: # signed by deploying operator +``` + +DCM startup sequence: verify manifest hash → verify manifest signature → proceed with initialization. On failure: refuse to start; emit security alert; notify platform admin. + +### 8.2 Kubernetes Cluster Upgrades in Sovereign Deployments (Q2) + +Sovereign DCM (air-gapped) uses pre-staged container images and the maintenance mode pattern: + +``` +Pre-upgrade: + 1. Pre-stage all DCM container images in local registry (signed bundles) + 2. DCM enters maintenance mode — new requests queued; in-flight complete + +During upgrade: + 3. Kubernetes upgrade proceeds using pre-staged images + 4. DCM components restart against new cluster version + +Post-upgrade: + 5. Startup verification (bootstrap manifest hash check) + 6. DCM exits maintenance mode — queued requests resume +``` + +Same signed bundle model as registry updates — no new pattern needed. + +### 8.3 Non-Kubernetes Container Runtime Support (Q3) + +Kubernetes is DCM's primary and recommended container runtime. Non-Kubernetes runtimes (Podman, Docker Compose) are supported for development and community extension only. The DCM Operator is Kubernetes-native and not supported on other runtimes. Production deployments must use Kubernetes. + +### 8.4 Minimum Hardware Specifications (Q4) + +Expressed as DCM Resource definitions per profile — machine-readable and enforced by the placement engine during self-deployment. + +| Profile | CPU | Memory | Storage | Replicas | +|---------|-----|--------|---------|---------| +| minimal | 2 cores | 4 Gi | 20 Gi | 1 | +| dev | 4 cores | 8 Gi | 50 Gi | 1 | +| standard | 8 cores | 16 Gi | 100 Gi | 3 | +| prod | 16 cores | 32 Gi | 200 Gi | 3 | +| fsi | 32 cores | 64 Gi | 500 Gi | 5 | +| sovereign | 32 cores | 64 Gi | 500 Gi | 5 | + +These are minimums. Production workloads may require significantly more based on managed resource count. + +### 8.5 DCM Self-Hosted Drift Detection (Q5) + +DCM's own deployment is a DCM-managed resource subject to the same drift detection as any other resource. The DCM Operator continuously reconciles running components against the declared deployment manifest. + +**The bootstrap paradox — who watches the watchman:** +- DCM Operator drifts → bootstrap manifest hash verification (RED-011) detects independently +- Audit component compromised → Audit Store hash chain break is detectable by external verification +- Full DCM compromise → signed bundle verification at import time provides external trust anchor + +### 8.6 System Policies — Deployment Redundancy Gaps + +| Policy | Rule | +|--------|------| +| `RED-011` | The bootstrap manifest is version-controlled in the GitOps store, hash-verified at DCM startup and every restart. Bootstrap manifest tampering prevents DCM from starting and triggers a security alert. | +| `RED-012` | Kubernetes cluster upgrades in Sovereign DCM deployments use pre-staged container images from the local signed bundle registry. DCM enters maintenance mode during upgrade. In-flight operations complete before upgrade. DCM exits maintenance mode on successful startup verification. | +| `RED-013` | Kubernetes is DCM's primary and recommended container runtime. Non-Kubernetes runtimes are supported for development and community extension purposes only. Production deployments must use Kubernetes. | +| `RED-014` | Minimum hardware specifications are expressed as DCM Resource definitions per profile and enforced by the placement engine during DCM self-deployment. | +| `RED-015` | DCM's own deployment is a DCM-managed resource subject to the same drift detection as any other resource. Bootstrap manifest hash verification (RED-011) provides independent verification. Audit Store hash chain breaks are detectable externally. | + + + +--- + +## 9. Bootstrap Tenant Creation Sequence + +### 9.1 The Bootstrap Problem + +DCM requires every entity to belong to exactly one Tenant. But during initial deployment, no Tenants exist. The bootstrap sequence resolves this by creating the foundational Tenants as part of DCM startup, declared in the bootstrap manifest. + +### 9.2 The Three Foundation Tenants + +The bootstrap manifest declares three system Tenants that are created before any consumer can submit requests: + +```yaml +bootstrap_tenants: + - handle: "__platform__" + display_name: "DCM Platform" + purpose: "Owns DCM's own control plane resources (components, stores, providers)" + automatically_created: true + cannot_be_decommissioned: true + + - handle: "__transitional__" + display_name: "Transitional" + purpose: "Holds brownfield entities during ingestion before promotion to a real Tenant" + automatically_created: true + cannot_be_decommissioned: true + + - handle: "__system__" + display_name: "System" + purpose: "Owns system-level artifacts (system layers, system policies, system workflows)" + automatically_created: true + cannot_be_decommissioned: true +``` + +### 9.3 Bootstrap Startup Sequence + +``` +DCM starts + │ + ▼ Step 1: Verify bootstrap manifest hash and signature + │ + ▼ Step 2: Initialize storage providers + │ GitOps stores initialized + │ Audit Store initialized + │ Commit Log initialized + │ + ▼ Step 3: Create foundation Tenants (if not already existing) + │ __platform__, __transitional__, __system__ + │ + ▼ Step 4: Create initial Platform Admin actor + │ Declared in bootstrap manifest + │ Assigned to __platform__ Tenant + │ Given platform_admin role + │ + ▼ Step 5: Activate system domain layers and policies + │ System layers loaded from GitOps store + │ System policies activated + │ Built-in recovery profiles activated + │ + ▼ Step 6: Register built-in providers + │ Built-in Auth Provider + │ Search Index data store + │ Audit Store data store + │ (All owned by __platform__ Tenant) + │ + ▼ Step 7: DCM ready + Consumer API, Provider API, Admin API accepting requests + Platform Admin can now create organization Tenants + Organization Tenants can request resources +``` + +### 9.4 System Policy + +| Policy | Rule | +|--------|------| +| `RED-016` | The three foundation Tenants (__platform__, __transitional__, __system__) are created during bootstrap and cannot be decommissioned. All DCM control plane resources are owned by __platform__. All brownfield ingested entities enter __transitional__ before promotion. | + + + +--- + +## 9. Bootstrap Sequence and Initial Tenant Creation + +### 9.1 The Bootstrap Problem + +The DCM data model requires every entity to be owned by a Tenant. But Tenants are themselves DCM entities. The bootstrap sequence defines how the initial Tenants and platform admin actor are created before DCM can accept external requests. + +### 9.2 Bootstrap Manifest + +The bootstrap manifest (RED-011) declares the initial state required for DCM to start. It includes: + +```yaml +bootstrap_manifest: + version: "1.0.0" + signed_by: + + # Initial system Tenants (created before any external requests) + system_tenants: + - uuid: + handle: "__platform__" + display_name: "DCM Platform" + description: "System Tenant owning DCM's own control plane resources" + immutable: true # cannot be decommissioned or modified by regular operators + + - uuid: + handle: "__transitional__" + display_name: "Transitional" + description: "Holding Tenant for brownfield ingestion (INGEST phase)" + immutable: true + + # Initial platform admin actor + bootstrap_admin: + uuid: + username: "dcm-bootstrap-admin" + auth_provider: builtin + roles: [platform_admin] + credential_ref: + # This credential is rotated on first login + + # Active profile for initial deployment + initial_profile: + deployment_posture: minimal # or as declared; can be changed post-bootstrap + compliance_domains: [] + + # Bootstrap admin's initial Tenant + initial_tenant: + uuid: + handle: "org-default" + display_name: "Default Organization Tenant" + owned_by: bootstrap_admin +``` + +### 9.3 Bootstrap Sequence + +``` +DCM starts → bootstrap manifest hash verified (RED-011) + │ + ▼ System Tenants created (before Policy Engine active): + │ __platform__ Tenant — owns DCM control plane resources + │ __transitional__ Tenant — brownfield ingestion holding + │ These are created by the bootstrap process itself, not through the request pipeline + │ + ▼ Bootstrap admin actor created + │ Auth Provider initialized with bootstrap credential + │ Platform Admin role assigned + │ + ▼ Initial profile activated + │ Deployment posture policies loaded + │ Compliance domain policies loaded (if declared) + │ + ▼ Policy Engine comes online + │ All subsequent operations go through the standard request pipeline + │ + ▼ Bootstrap admin creates the initial organization Tenant (optional) + │ First real request through the pipeline + │ Creates the initial production Tenant for organizational resources + │ + ▼ Bootstrap admin credential rotation notification sent + │ Bootstrap credential must be rotated on first login + │ After rotation, bootstrap_admin becomes a standard platform admin actor + │ + ▼ DCM accepts external requests +``` + +### 9.4 System Tenants + +The `__platform__` and `__transitional__` Tenants are created by the bootstrap process and are immutable: + +| System Tenant | Purpose | Who can modify | +|--------------|---------|---------------| +| `__platform__` | Owns DCM's own control plane resources | Platform Admin (restricted operations only) | +| `__transitional__` | Brownfield ingestion holding area | Ingestion pipeline only | + +These Tenants are exempt from the normal Tenant decommission workflow — they cannot be decommissioned while DCM is operational. + +### 9.5 System Policy + +| Policy | Rule | +|--------|------| +| `BOOT-001` | The __platform__ and __transitional__ system Tenants are created by the bootstrap process before the Policy Engine comes online. They are immutable and cannot be decommissioned while DCM is running. | +| `BOOT-002` | The bootstrap admin credential must be rotated on first login. The bootstrap manifest declares the initial credential reference only; the credential itself is managed by the credential management service. | +| `BOOT-003` | After bootstrap, all Tenant creation and modification goes through the standard request pipeline. The bootstrap process is a one-time operation. | + + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* diff --git a/architecture/runtime-features/federation-runtime.md b/architecture/runtime-features/federation-runtime.md new file mode 100644 index 0000000..0940276 --- /dev/null +++ b/architecture/runtime-features/federation-runtime.md @@ -0,0 +1,758 @@ +--- +Document Status: ✅ Complete +Document Type: Architecture Reference — Federation Runtime +Maps to: udlm/governance/federated-contribution-model.md +--- + +# DCM Data Model — DCM Federation, Peering, and Cross-Instance Coordination + +> **Implements contracts defined in UDLM**: +> [udlm/governance/federated-contribution-model.md](https://github.com/croadfeldt/udlm/blob/main/governance/federated-contribution-model.md). +> UDLM defines the federated contribution model — how peer DCM instances +> contribute, share, and govern data across instance boundaries, including +> the contribution authority and cross-instance trust contracts. DCM +> operationalizes peer DCMs as typed Providers, the peering and tunnel +> runtime, and the cross-instance coordination of contributed data. + + +**Document Status:** ✅ Complete +**Related Documents:** [Federated Contribution Model](https://github.com/croadfeldt/udlm/blob/main/governance/federated-contribution-model.md) | [Universal Group Model](https://github.com/croadfeldt/udlm/blob/main/observability/universal-groups.md) | [data stores](https://github.com/croadfeldt/udlm/blob/main/contracts/storage-providers.md) | [Auth Providers](https://github.com/croadfeldt/udlm/blob/main/governance/auth-providers.md) | [Information Providers Advanced](https://github.com/croadfeldt/udlm/blob/main/contracts/information-providers-advanced.md) + +> **Foundation Document Reference** +> +> This document is a detailed reference for a specific domain of the DCM architecture. +> The three foundational abstractions — Data, Provider, and Policy — are defined in +> [udlm/foundations/foundations.md](https://github.com/croadfeldt/udlm/blob/main/foundations/foundations.md). All concepts in this document map to one or +> more of those three abstractions. +> See also: [Provider Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/provider-contract.md) | [Policy Contract](https://github.com/croadfeldt/udlm/blob/main/contracts/policy-contract.md) +> +> **This document maps to: PROVIDER + POLICY** +> +> Provider: Peer DCM as typed Provider. Policy: federation governance rules + + + +--- + +> **Federated Contribution:** Federation contribution follows the [Federated Contribution Model](https://github.com/croadfeldt/udlm/blob/main/governance/federated-contribution-model.md) — peer DCMs are contributors to each other's artifact stores, scoped by their federation trust posture. + +## 1. Purpose + +DCM instances do not operate in isolation. Organizations with multiple data centers, regions, or organizational boundaries may run multiple DCM instances that need to coordinate, share resources, and maintain consistent governance. This document defines how DCM instances relate to each other — as peers, in parent-child hierarchies, or as hub-and-spoke configurations — and the mechanisms for cross-instance resource sharing, data export/import, and provider federation eligibility. + +--- + +## 2. DCM-to-DCM Relationship Types + +### 2.1 Three Relationship Types + +**Peer DCM** — two DCM instances at the same organizational level that share resources or information. Regional DCM instances sharing VLAN allocations. Campus DCM instances sharing compute capacity across departments. + +**Parent-Child DCM** — hierarchical relationship where the parent has governance overlay authority over child instances. Corporate DCM with regional children. Service provider DCM with customer children. Parent does not own child resources — it has governance visibility and policy overlay authority (same model as nested Tenants in the Universal Group Model). + +**Hub DCM** — specialized Parent DCM acting as a clearinghouse for resource allocation across multiple children. The Hub holds the master resource inventory; children request allocations from the Hub. + +### 2.2 Relationship Mapping to Universal Group Model + +DCM-to-DCM relationships use the existing Universal Group Model: + +```yaml +# Peer relationship — federation group +dcm_group: + group_class: federation + name: "EU-US Regional Federation" + members: + - member_uuid: + member_type: dcm_peer + member_role: eu_region_dcm + - member_uuid: + member_type: dcm_peer + member_role: us_region_dcm + federation_config: + shared_policy_inheritance: opt_in + cross_member_visibility: audit_only + +# Parent-Child relationship — nesting +dcm_group: + group_class: tenant_boundary + name: "Corporate DCM" + child_groups: + - + - + policy_inheritance: opt_out # parent policies cascade unless child excludes +``` + +--- + +## 3. Provider Federation Eligibility + +### 3.1 Concept + +Every provider registration carries a `federation_eligibility` declaration — whether the provider can participate in cross-DCM federation, with whom, and under what conditions. This is **layer-defined** (static organizational knowledge) and **policy-enforced** (runtime governance). + +### 3.2 Federation Eligibility on Provider Registration + +```yaml +provider_registration: + handle: "providers/service/eu-compute-primary" + + federation_eligibility: + mode: + # none: Provider cannot participate in any DCM federation + # (sovereign, classified, or compliance-restricted providers) + # selective: Federation permitted only with explicitly declared partners + # open: Federation permitted with any trusted DCM peer + # (sovereignty checks always apply regardless) + + permitted_partners: + - partner_type: + dcm_instance_uuids: [] # specific instances + dcm_instance_tags: [region-eu, internal] # tag-based matching + dcm_certification_required: [ISO-27001, GDPR-compliant] + relationship_requires_approval: true # bilateral approval required + + federation_scope: + permitted_resource_types: + - resource_type: Compute.VirtualMachine + operations: [allocate, query_capacity] + # NOT: decommission — remote DCMs cannot decommission local resources + - resource_type: Network.VLAN + operations: [allocate, query_capacity, release] + data_sharing: + capacity_data: true + realized_state: false # do not share realized state details + pricing_data: true + sovereignty_declaration: true # always share — required for federation + max_allocations_per_partner: 100 + max_concurrent_allocations: 500 + + override_reason: null # populated when overriding layer default +``` + +### 3.3 Layer-Defined Federation Defaults + +Federation eligibility defaults live in a `platform` domain layer — static organizational knowledge inherited by all providers unless overridden. Individual provider registrations may be **more restrictive** than the layer default (always permitted); **less restrictive** requires GateKeeper policy approval. + +```yaml +layer: + handle: "platform/federation/provider-federation-defaults" + domain: platform + priority: 600.0.0 + fields: + provider_federation_defaults: + compute_providers: + federation_eligibility: + mode: selective + permitted_partners: + - partner_type: dcm_peer + dcm_instance_tags: [internal, eu-region] + dcm_certification_required: [ISO-27001] + federation_scope: + permitted_resource_types: + - resource_type: Compute.VirtualMachine + operations: [allocate, query_capacity] + (prescribed infrastructure)s: + federation_eligibility: + mode: none # storage never federated — data sovereignty + network_providers: + federation_eligibility: + mode: selective + permitted_partners: + - partner_type: dcm_peer + dcm_instance_tags: [internal] + information_providers: + federation_eligibility: + mode: selective + data_sharing_restrictions: + max_classification: internal # never share confidential/restricted +``` + +### 3.4 Policy Enforcement on Federation + +Policies act on federation eligibility at three enforcement points: + +**At tunnel establishment:** +```yaml +policy: + type: gatekeeper + target: dcm_tunnel_establishment + rule: > + If provider.federation_eligibility.mode == none + THEN gatekeep: "Provider is not eligible for federation" + +policy: + type: gatekeeper + target: dcm_tunnel_establishment + rule: > + If remote_dcm.sovereignty_zone NOT IN permitted_sovereignty_zones + THEN gatekeep: "Remote DCM sovereignty zone incompatible with local requirements" + +policy: + type: gatekeeper + target: dcm_tunnel_establishment + rule: > + If remote_dcm.certifications NOT CONTAINS + provider.federation_eligibility.permitted_partners.dcm_certification_required + THEN gatekeep: "Remote DCM does not hold required certifications" +``` + +**At allocation time:** +```yaml +policy: + type: gatekeeper + target: cross_dcm_allocation + rule: > + If resource_type NOT IN provider.federation_eligibility.federation_scope.permitted_resource_types + THEN gatekeep: "Resource type not permitted through this federation tunnel" + +policy: + type: gatekeeper + target: cross_dcm_allocation + rule: > + If cross_dcm_allocations_active > provider.federation_eligibility.max_concurrent_allocations + THEN gatekeep: "Maximum concurrent federation allocations exceeded" +``` + +**At data egress:** +```yaml +policy: + type: gatekeeper + target: dcm_tunnel_data_egress + rule: > + If data.classification > remote_dcm.max_data_classification_receivable + THEN gatekeep: "Data classification exceeds remote DCM authorization" +``` + +--- + +## 4. The DCM Provider — Cross-Instance Tunneling + +### 4.1 Concept + +A **DCM Provider** is a ninth provider type that wraps another DCM instance's API, enabling one DCM to consume resources managed by another DCM as if they were local providers. + +| # | Type | Purpose | +|---|------|---------| +| 1 | Service Provider | Realizes resources | +| 2 | Information Provider | Serves authoritative external data | +| 3 | composite service definition | Composes multiple providers | +| 4 | data store | Persists DCM state | +| 5 | External Policy Evaluator | Supplies and evaluates policies | +| 6 | event routing service | Bridges internal/external event streams | +| 7 | credential management service | Resolves secrets | +| 8 | Auth Provider | Authenticates identities | +| 9 | **DCM Provider** | Wraps another DCM instance's API | + +### 4.2 DCM Provider Registration + +```yaml +dcm_provider_registration: + artifact_metadata: + uuid: + handle: "providers/dcm/region-eu-dcm" + status: active + + provider_type: dcm_provider + relationship_type: + + remote_dcm: + instance_uuid: + endpoint: https://dcm.region-eu.corp.example.com + dcm_version: "2.1.0" # minimum compatible version + sovereignty_declaration_ref: # verified at registration time + + # Authentication — always mTLS for DCM-to-DCM + auth: + mode: mtls + client_cert_ref: + service_provider_uuid: + path: "dcm/dcm-providers/region-eu/client-cert" + server_ca_ref: + service_provider_uuid: + path: "dcm/dcm-providers/region-eu/server-ca" + + # Tunnel configuration + tunnel_config: + encrypted: true # always — not configurable + sovereignty_boundary_check: true # always — not configurable + permitted_resource_types: [Compute.VirtualMachine, Network.VLAN] + max_allocation_per_request: 10 + audit_forwarding: true # forward audit records to local Audit Store + observability_forwarding: true + + # Sovereignty — must be compatible with local requirements + sovereignty_declaration: + remote_jurisdiction: eu-west + data_residency_guarantee: true + certifications: [ISO-27001, GDPR-compliant] + + # Health check + health_check: + endpoint: /api/v1/health + interval_seconds: 60 + on_unhealthy: suspend_allocations +``` + +### 4.3 Primary Concerns on All DCM Tunnels + +These are non-negotiable on every DCM-to-DCM connection: + +| Concern | Enforcement | +|---------|-----------| +| **Sovereignty** | Sovereignty_declaration verified before tunnel establishment; data classification checked before every egress | +| **Authentication** | Always mTLS — no API key, no bearer token — mutual certificate authentication between DCM instances | +| **Authorization** | Local DCM policies govern ALL resources obtained through a tunnel; remote DCM's policies do not override local | +| **Audit** | All cross-DCM operations produce audit records in BOTH DCM instances; shared `correlation_id` links the two trails | +| **Observability** | Cross-DCM resource allocation visible in both instances' observability stores | +| **Governance** | Local GateKeeper policies apply to all resources from any tunnel source | + +--- + +## 5. Cross-DCM Confidence Scoring + +Resources obtained through a DCM tunnel carry compound confidence scores — the resource's confidence in its source DCM, degraded by the tunnel trust score: + +``` +cross_dcm_confidence = source_resource_confidence × (tunnel_trust_score / 100) +``` + +A resource with confidence 90 in source DCM, through a tunnel with trust score 85: `90 × 85/100 = 76.5` → **77** + +### 5.1 Federation Trust Score + +```yaml +dcm_federation_trust_score: + remote_dcm_uuid: + score: 84 # 0-100 + scored_at: + decay_rate: per_30_days + factors: + identity_verified: true # mTLS certificate chain verified + sovereignty_compatible: true + certifications_current: true + audit_trail_integrity: true # audit hash chains verified on sample + uptime_score: 0.98 + compliance_score: 0.90 # policy compliance in recent operations + data_completeness: 0.92 + action_on_score_below: + threshold: 60 + action: +``` + +--- + +## 6. DCM Export and Import + +### 6.1 Export Package + +DCM state is fully exportable as a signed package — for disaster recovery, migration, cross-DCM sharing, and Hub DCM onboarding. + +```yaml +dcm_export_package: + package_uuid: + exported_at: + exported_by: + dcm_version: + signed_by: + + scope: + tenants: [, ...] + resource_types: all + layers: [platform, tenant] + policies: [platform, tenant] + providers: registrations_only # not credentials — never export credentials + entities: [intent_state, requested_state] # not realized (that's provider state) + groups: all + audit_records: + date_range: [, ] + include_hash_chain: true # for audit trail verification on import + + sovereignty: + classification: internal + permitted_import_jurisdictions: [eu-west] + signed: true + encryption: aes256_gcm +``` + +### 6.2 Import Trust Score + +When importing from another DCM instance, each imported resource carries a trust score: + +```yaml +import_trust_score: + score: 78 # 0-100 + factors: + source_dcm_verified: true # source DCM identity verified + sovereignty_compatible: true + data_completeness: 0.92 + schema_compatibility: 1.00 # source schema matches current version + audit_trail_complete: true # audit records included and hash chain valid + certifications_current: true + action_on_low_score: + threshold: 70 # reject if below +``` + +### 6.3 Scoring in Resource Definition and Allocation + +Resources imported from or allocated through peer DCMs carry their compound confidence score throughout their lifecycle in the importing DCM. The score is visible to the placement engine, Cost Analysis, and the Policy Engine — enabling policies that prefer locally-managed resources over federated resources when scores are comparable. + +--- + +## 7. Profile-Appropriate Federation Policy Groups + +DCM ships built-in federation policy groups activated by default per profile: + +| Group | Profile | Behavior | +|-------|---------|---------| +| `system/group/federation-minimal` | minimal | No federation — single instance only | +| `system/group/federation-dev` | dev | Peer federation permitted; advisory only | +| `system/group/federation-standard` | standard | Peer federation with certification requirements | +| `system/group/federation-prod` | prod | Selective federation; bilateral approval; audit forwarding | +| `system/group/federation-fsi` | fsi | Strict federation; within-jurisdiction only; full audit; no storage federation | +| `system/group/federation-sovereign` | sovereign | No external federation; internal peer federation within sovereignty boundary only | + +--- + +## 8. DCM System Policies — Federation + +| Policy | Rule | +|--------|------| +| `DCM-001` | DCM instances may establish peer, parent-child, or hub relationships using the Universal Group Model federation and nesting constructs. | +| `DCM-002` | All DCM-to-DCM communication uses mTLS. No API key or bearer token. Sovereignty checks are mandatory before tunnel establishment. These requirements are non-configurable. | +| `DCM-003` | Local DCM policies govern all resources obtained through DCM tunnels. Cross-DCM operations produce audit records in both DCM instances with a shared correlation_id. | +| `DCM-004` | DCM state is exportable as a signed package. Imported packages carry a trust score (0-100) computed from source verification, sovereignty compatibility, data completeness, schema compatibility, and audit trail integrity. | +| `DCM-005` | Resources obtained through DCM tunnels carry compound confidence scores: source_resource_confidence × (tunnel_trust_score / 100). | +| `DCM-006` | Every provider registration must declare federation_eligibility (mode: none, selective, or open). Federation eligibility defaults are declared in platform domain layers. Individual provider registrations may be more restrictive — never more permissive without GateKeeper policy approval. | +| `DCM-007` | Provider federation scope declares: permitted resource types, permitted operations per type, data sharing permissions, and allocation limits. Remote DCMs cannot decommission local resources through a federation tunnel. | +| `DCM-008` | Storage providers default to federation_eligibility.mode: none. Data sovereignty constraints prohibit storage federation unless explicitly authorized by sovereign policy with full justification. | + +--- + +## 9. Open Questions + +| # | Question | Impact | Status | +|---|----------|--------|--------| +| 1 | How are DCM-to-DCM certificate rotation and renewal handled — coordinated or independent? | Operations | ✅ Resolved — independent rotation with P30D overlap; peer notification 60 days before expiry via Message Bus; auto-renewal at 90 days (DCM-009) | +| 2 | Should Hub DCM relationships support automatic load balancing across child DCMs? | Architecture | ✅ Resolved — full placement engine logic at DCM instance level; sovereignty as hard pre-filter; tie-breaking hierarchy same as provider selection; sub-regional routing recursive (DCM-010) | +| 3 | How does drift detection work for resources allocated from a peer DCM — who is responsible for discovery? | Operational | ✅ Resolved — provider-side DCM discovers; consumer-side DCM compares; events via federation Message Bus; peer unavailable = alert-and-hold (DCM-011) | +| 4 | Should cross-DCM audit records be synchronized — so each DCM has the other's audit records? | Compliance | ✅ Resolved — correlation_id reference model; no full sync; on-demand pull with platform admin auth + sovereignty check (DCM-012) | +| 5 | What is the maximum supported federation depth (peer of peer of peer)? | Architecture | ✅ Resolved — profile-governed max depth: minimal/dev=5, standard/prod=3, fsi/sovereign=2; measured as hops from deepest to Hub (DCM-013) | + + +## 11. Federation Gap Resolutions + +### 11.1 Certificate Rotation and Renewal (Q1) + +DCM-to-DCM mTLS certificates rotate independently per instance with a coordinated notification model. Coordinated simultaneous rotation would create a single point of failure. + +```yaml +dcm_federation_cert_rotation: + rotation_model: independent_with_overlap + overlap_period: P30D # old cert valid for 30 days after new cert issued + notification: + notify_peers_at: P60D_before_expiry + notification_channel: message_bus + notification_payload: + new_cert_public_key: + new_cert_valid_from: + old_cert_expires_at: + automatic_renewal: + trigger_at: P90D_before_expiry + requires_approval: false # renewal is automatic; elevation requires approval +``` + +The P30D overlap allows peers to update their trust stores at their own pace without service interruption. + +### 11.2 Federation Routing — Placement Engine at the DCM Level (Q2) + +**Hub DCM federation routing follows the same placement engine logic as provider selection.** Regional DCMs are treated as DCM Provider instances in the placement engine. + +**Sovereignty is a hard pre-filter — not a preference:** + +``` +Before placement loop: + Filter eligible Regional DCMs where: + sovereignty_declaration satisfies request constraints + operating_jurisdictions includes required jurisdictions + sovereignty_zone matches tenant.sovereignty_zone + + If no eligible Regional DCMs → Reject with clear error + Only eligible DCMs enter the placement loop +``` + +**The full federation routing flow:** + +``` +Request arrives at Hub DCM + │ + ▼ Steps 1-5: Standard nine-step assembly (layers, policies, placement constraints) + │ Pre-placement policies may declare federation routing constraints: + │ "This resource must be in a Regional DCM with EU sovereignty" + │ "This Tenant's resources must stay in Regional DCM-EU-West" + │ + ▼ Step 6: Placement loop — at the DCM instance level + │ Reserve query to eligible Regional DCMs: + │ capacity available? sovereignty compatible? trust score adequate? + │ + ▼ Tie-breaking (same hierarchy as provider selection): + │ 1. Policy preference (policy declares preferred Regional DCM) + │ 2. Federation priority (numeric priority on DCM Provider registration) + │ 3. Tenant affinity (Tenant's resources prefer a specific Regional DCM) + │ 4. Sovereignty match quality (exact match over partial match) + │ 5. Geographic affinity (closest regional to consumer) + │ 6. Least loaded (capacity utilization across instances) + │ 7. Consistent hash (deterministic tiebreaker) + │ + ▼ Selected Regional DCM receives assembled request payload + │ Runs its own local assembly and placement (regional layers, regional providers) + │ Returns realization result to Hub DCM → forwarded to consumer + │ + ▼ Sub-regional routing: Regional DCM acts as Hub for its children + Same logic applies recursively within federation depth limit (DCM-013) +``` + +**Load balancing is the least-loaded step in the hierarchy** — not a primary strategy. Sovereignty, policy, and tenant affinity all take precedence. Optional `hub_dcm_load_balancing` configuration: + +```yaml +hub_dcm_load_balancing: + enabled: true # default: true + sovereignty_override: true # always — sovereignty is a hard pre-filter + fallback_on_regional_unavailable: route_to_next_eligible +``` + +### 11.3 Federated Drift Detection Ownership (Q3) + +Provider-side DCM is responsible for discovery; consumer-side DCM is responsible for drift comparison. + +```yaml +federated_drift_detection: + discovery_responsibility: provider_side_dcm + comparison_responsibility: consumer_side_dcm + mechanism: + provider_dcm: + - Run standard discovery against its providers + - Publish Discovered State events to federation Message Bus + - Tagged with: entity_uuid + consumer_dcm_uuid + correlation_id + consumer_dcm: + - Subscribe to Discovered State events for its federated entities + - Compare against its Requested State + - Trigger drift response policy if drift detected + on_peer_dcm_unavailable: + action: alert_and_hold # not assumed drift + max_hold_period: PT24H + on_hold_exceeded: escalate_to_platform_admin +``` + +### 11.4 Cross-DCM Audit Record Correlation (Q4) + +No full synchronization. Each DCM keeps its own authoritative audit trail. Cross-DCM correlation uses correlation_id references and on-demand pull. + +```yaml +cross_dcm_audit_correlation: + model: correlation_id_reference + local_audit_record: + action: ALLOCATE_FROM_PEER + correlation_id: + peer_dcm_uuid: + peer_audit_record_uuid: # reference — not a copy + on_demand_pull: + endpoint: GET /api/v1/audit/cross-dcm/{correlation_id} + requires: platform_admin + peer_dcm_authorization + sovereignty_check +``` + +Full synchronization is not required — auditors follow correlation_id to the peer DCM on demand. + +### 11.5 Maximum Federation Depth (Q5) + +```yaml +federation_depth_policy: + max_depth: 3 # profile-governed + on_max_exceeded: reject_federation_establishment + profile_defaults: + minimal: 5 + dev: 5 + standard: 3 + prod: 3 + fsi: 2 + sovereign: 2 +``` + +Depth is measured as hops from the deepest instance to the Hub DCM. Depth 3 covers Hub → Regional → Sub-Regional → Edge — sufficient for most real-world architectures. + +### 11.6 System Policies — Federation Gaps + +| Policy | Rule | +|--------|------| +| `DCM-009` | DCM-to-DCM mTLS certificates rotate independently per instance with a P30D overlap period. Peers are notified 60 days before expiry via Message Bus. Automatic renewal triggers 90 days before expiry. The overlap period allows peers to update trust stores without coordinated downtime. | +| `DCM-010` | Hub DCM federation routing follows the same placement engine logic as provider selection. Sovereignty is a hard pre-filter — only Regional DCMs satisfying all sovereignty constraints enter the placement loop. The tie-breaking hierarchy applies at the DCM instance level: policy preference → federation priority → tenant affinity → sovereignty match quality → geographic affinity → least loaded → consistent hash. Regional DCMs are treated as DCM Provider instances. Sub-regional routing applies the same logic recursively within the federation depth limit. | +| `DCM-011` | For resources allocated from peer DCMs, the provider-side DCM is responsible for discovery. The consumer-side DCM is responsible for drift comparison. Discovered State events are published via federation Message Bus with correlation_id. Peer DCM unavailability triggers alert-and-hold — not assumed drift. | +| `DCM-012` | Cross-DCM audit records are referenced via correlation_id — not fully synchronized. Each DCM keeps its own authoritative audit trail. Cross-DCM correlation uses on-demand pull with platform admin authorization and sovereignty check. | +| `DCM-013` | Federation depth is limited to a profile-governed maximum (default: 3 for standard/prod; 2 for fsi/sovereign; 5 for minimal/dev). Requests to establish federation beyond the maximum depth are rejected. Depth is measured as hops from the deepest instance to the Hub DCM. | + + +--- + +## 10. Related Concepts + +- **Universal Group Model** (doc 15) — federation and nesting group classes +- **data stores** (doc 11) — storage never federated by default +- **Auth Providers** (doc 19) — mTLS for DCM-to-DCM authentication +- **Universal Audit Model** (doc 16) — audit records in both DCM instances; correlation_id +- **Registry Governance** (doc 20) — signed bundles for air-gapped registry updates +- **Information Providers Advanced** (doc 21) — confidence scoring used in cross-DCM context + +--- + +*Document maintained by the DCM Project. For questions or contributions see [GitHub](https://github.com/dcm-project).* + +--- + +## 12. Federation Tunnel Establishment and Maintenance + +> **Implements contracts defined in UDLM**: +> [udlm/governance/accreditation-and-authorization-matrix.md](https://github.com/croadfeldt/udlm/blob/main/governance/accreditation-and-authorization-matrix.md). +> UDLM defines the Federation Tunnel Model as the contract for secure +> inter-DCM channels. This section operationalizes the establishment and +> maintenance of those tunnels. + +### 12.1 Tunnel as zero-trust boundary + +A federation tunnel is a mutually authenticated, encrypted, scoped channel — +**not a VPN**. It establishes secure transport, not perimeter trust. Every +message crossing the tunnel is authenticated, authorized, and subject to the +five-check boundary model defined in +[udlm/governance/accreditation-and-authorization-matrix.md](https://github.com/croadfeldt/udlm/blob/main/governance/accreditation-and-authorization-matrix.md). + +### 12.2 Tunnel structure + +```yaml +federation_tunnel: + uuid: + local_dcm_uuid: + remote_dcm_uuid: + tunnel_type: peer | parent_child | hub_spoke + trust_model: zero_trust # always; non-negotiable + + # Mutual authentication + authentication: + protocol: mtls + local_certificate_ref: + remote_certificate_pin: # pinned; not just chain-valid + trust_anchor: + certificate_rotation_interval: P90D + revocation_check: ocsp_stapling + + # Per-message signing + message_integrity: + signing_algorithm: ed25519 + local_signing_key_ref: + remote_verification_key_ref: + replay_protection: true # nonce + timestamp window PT5M + + # Inbound authorization — what remote may request from this DCM + inbound_authorization: + - operation: catalog_query + permitted_resource_types: [Compute.VirtualMachine, Network.VLAN] + requires_cross_tenant_authorization: true + - operation: allocation_request + permitted_resource_types: [Network.IPAddress] + max_allocations_per_request: 10 + requires_cross_tenant_authorization: true + + # Outbound authorization — what this DCM may request from remote + outbound_authorization: + - operation: placement_query + permitted_resource_types: [Compute.VirtualMachine] + - operation: realized_state_query + permitted_entity_uuids: [] # scoped to specific entities + + # Data classification boundary (hard constraints) + data_boundary: + max_outbound_classification: restricted # never send sovereign/classified + max_inbound_classification: restricted + # sovereign profile: max_*_classification: internal + # classified profile: no federation permitted + + # Sovereignty scope + sovereignty_scope: + local_jurisdiction: EU + remote_jurisdiction: EU + cross_jurisdiction_permitted: false # fsi/sovereign: always false +``` + +### 12.3 Federation credential scoping + +Federation credentials are scoped to specific tunnel operations: + +```yaml +federation_credential: + credential_uuid: + issued_by_dcm_uuid: + issued_to_dcm_uuid: + expires_at: # PT15M for fsi/sovereign + operation_scope: catalog_query + scoped_resource_types: [Compute.VirtualMachine] + non_transferable: true + tunnel_uuid: # bound to specific tunnel +``` + +A federation credential issued for `catalog_query` cannot be used for +`allocation_request`. + +### 12.4 Establishment flow + +``` +Local DCM initiates establishment with Remote DCM: + ▼ Mutual mTLS handshake + │ Validate remote certificate against trust_anchor and pin + │ OCSP stapling for real-time revocation check + ▼ Sovereignty compatibility check (hard pre-filter) + │ Local + remote sovereignty zones must satisfy declared requirements + │ Cross-jurisdiction blocked in fsi/sovereign + ▼ Accreditation verification + │ Remote DCM's accreditations checked via Accreditation Monitor + │ Required accreditations per profile + ▼ Tunnel record created (status: establishing) + ▼ Initial federation credential issued (scoped to limited ops) + ▼ Health probe exchange to verify bidirectional connectivity + ▼ Tunnel transitions to active + ▼ Audit record written: federation.tunnel_established +``` + +### 12.5 Tunnel maintenance + +DCM maintains tunnel health via: + +- **Health probes** every PT60S (configurable) +- **Certificate rotation** independent per side with P30D overlap (DCM-009) +- **Federation trust score** updated continuously per + [udlm/contracts/information-providers-advanced.md](https://github.com/croadfeldt/udlm/blob/main/contracts/information-providers-advanced.md) +- **Sovereignty re-verification** on configurable cadence; tunnel suspended + if sovereignty becomes incompatible +- **Federation depth enforcement** per DCM-013 (profile-governed max depth) + +On tunnel degradation: +- Trust score drops below 60 → action per `action_on_score_below` config +- Sovereignty incompatibility → tunnel suspended immediately; admin notified +- Certificate revocation → tunnel suspended; new credential required + +### 12.6 Hub-spoke zero trust + +In hub-spoke federation, the Hub DCM coordinates Regional DCMs. Zero trust +means: + +- The Hub DCM does not have root-level access to Regional DCMs — only + explicitly scoped federation credentials +- A Regional DCM cannot impersonate the Hub to another Regional DCM +- Cross-Regional-DCM operations route through the Hub with the **Hub's + authorization**, not the originating Regional DCM's authorization +- The Hub DCM's accreditation is visible to Regional DCMs — they can verify + the Hub before accepting federation messages + +``` +RegionalDCM-A → HubDCM: authenticated; scoped to allocation_request +HubDCM → RegionalDCM-B: authenticated; scoped to realization_request + Hub presents its own credential to RegionalDCM-B + Not RegionalDCM-A's credential +RegionalDCM-B verifies: Hub certificate; Hub accreditation; data boundary +``` diff --git a/architecture/runtime-features/scheduling.md b/architecture/runtime-features/scheduling.md new file mode 100644 index 0000000..dc53fb6 --- /dev/null +++ b/architecture/runtime-features/scheduling.md @@ -0,0 +1,337 @@ +--- +Document Status: ✅ Stable — DCM implementation +Document Type: Architecture Reference — Request Scheduling +Established: 2026-05-26 +Maps to: udlm/lifecycle/scheduled-requests.md +--- + +# Request Scheduling + +> **Implements contracts defined in UDLM**: +> [udlm/lifecycle/scheduled-requests.md](https://github.com/croadfeldt/udlm/blob/main/lifecycle/scheduled-requests.md). +> UDLM defines the scheduling model (immediate / at / window / recurring), +> the SCHEDULED request state contract, the maintenance window coordination +> contract, and the deadline enforcement contract. DCM operationalizes the +> Request Scheduler component, deferred request lifecycle management, +> maintenance window scheduling logic, deadline evaluation and timeout +> enforcement, consumer API additions, new events, and profile-governed +> scheduling constraints. + +--- + +## 1. Request Scheduler component + +The Request Scheduler is a DCM control plane component (an internal function +of the Request Orchestrator per +[`../control-plane/components.md`](../control-plane/components.md), Section +5.3) responsible for: + +- Maintaining a priority queue of SCHEDULED requests ordered by `not_before` +- Polling the queue; dispatching requests when `not_before` is reached +- Checking `not_after` deadlines; cancelling expired requests +- Listening for `maintenance_window` events to trigger window-scheduled + requests +- On dispatch: handing off to Request Orchestrator (same path as immediate + requests) +- Writing SCHEDULED status updates to Intent State +- Publishing `request.scheduled` and `request.schedule_cancelled` events + +### 1.1 Implementation choice + +DCM implements the Request Scheduler as a PostgreSQL-backed cron worker per +the +[`../persistence/postgres-implementation.md`](../persistence/postgres-implementation.md) +infrastructure. A `scheduled_requests` table maintains the queue; a worker +polls every PT15S for ripe requests; HA via leader election (PostgreSQL +advisory locks or a simple lease pattern). + +```sql +CREATE TABLE scheduled_requests ( + request_uuid UUID PRIMARY KEY, + entity_uuid UUID NOT NULL, + tenant_uuid UUID NOT NULL REFERENCES tenants(tenant_uuid), + schedule_dispatch VARCHAR(16) NOT NULL, -- at | window | recurring + not_before TIMESTAMPTZ NOT NULL, + not_after TIMESTAMPTZ, + window_uuid UUID, + cron_expression TEXT, + max_occurrences INTEGER, + occurrences_so_far INTEGER DEFAULT 0, + status VARCHAR(16) NOT NULL DEFAULT 'SCHEDULED', + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE INDEX idx_scheduled_ripe ON scheduled_requests(not_before, status) + WHERE status = 'SCHEDULED'; +``` + +--- + +## 2. Deferred request lifecycle management + +A scheduled request moves through the four states with one additional +intermediate status. DCM implements: + +``` +Submit with schedule.dispatch: at + ▼ ACKNOWLEDGED (Intent State created; entity_uuid assigned) + ▼ Policy evaluation at declaration time + │ GateKeeper policies run immediately (fail-fast) + │ If rejected: request fails before entering queue + │ If approved: request enters scheduled queue + ▼ SCHEDULED (new status within Intent State) + │ Stored in Request Scheduler queue + │ Visible via GET /api/v1/requests/{uuid}/status + │ Cancellable: DELETE /api/v1/requests/{uuid} + ▼ [at not_before] → Policy re-evaluation at dispatch + │ Transformation policies re-run (data may have changed) + │ GateKeeper re-evaluation with current data + │ If still approved: proceed to LAYERS_ASSEMBLED → dispatch + │ If rejected at dispatch: FAILED with failure_reason: schedule_policy_rejection + ▼ DISPATCHED → REALIZED (normal pipeline) +``` + +### 2.1 Dual policy evaluation (SCH-001) + +Policies evaluate twice intentionally: + +- **At declaration time** — catches obvious rejections early (fail fast) +- **At dispatch time** — catches changes since declaration (quota exhausted, + new compliance policy activated, actor role changed, etc.) + +Dispatch-time evaluation uses the **current** policy set, not the one in +effect at declaration. SCH-003: requests that fail dispatch-time policy +re-evaluation enter FAILED state with `failure_reason: schedule_policy_rejection`. +The consumer receives a `request.failed` event with the rejection detail. + +--- + +## 3. Maintenance window scheduling logic + +Maintenance Windows are reusable schedule artifacts referenced by scheduled +requests. DCM implements: + +```yaml +maintenance_window: + window_uuid: + window_handle: "weekly-sunday-0200-utc" + description: "Weekly maintenance window — low traffic period" + + cron: "0 2 * * 0" # every Sunday at 02:00 UTC + duration: PT2H # window is 2 hours long + + tenant_uuid: # null = platform-wide + resource_types: [] # empty = all resource types + + status: active | suspended + approved_by: + effective_from: + + created_at: + created_by: +``` + +### 3.1 Window-scheduled request flow + +``` +Consumer submits request with schedule.dispatch: window, window_id: + ▼ Request validates window exists, is active, is in scope (tenant + resource type) + ▼ Request enters SCHEDULED with not_before = next window start + ▼ Window scheduler periodically computes upcoming window starts via cron + ▼ At window start: all queued requests for the window dispatch in batch + ▼ Within the window's duration, requests dispatch normally + ▼ Outside window duration, queued requests wait for next window +``` + +### 3.2 Maintenance Window API + +``` +# Platform admin operations +POST /api/v1/admin/maintenance-windows +GET /api/v1/admin/maintenance-windows +GET /api/v1/admin/maintenance-windows/{window_uuid} +PATCH /api/v1/admin/maintenance-windows/{window_uuid} +DELETE /api/v1/admin/maintenance-windows/{window_uuid} + +# Consumer operations +GET /api/v1/maintenance-windows # list visible to consumer +GET /api/v1/maintenance-windows/{uuid} # describe a specific window +``` + +Window approval tier varies by profile: + +| Profile | Window approval tier | +|---|---| +| minimal | auto | +| dev | auto | +| standard | reviewed | +| prod | reviewed | +| fsi | verified | +| sovereign | authorized | + +--- + +## 4. Deadline evaluation and timeout enforcement + +### 4.1 not_before enforcement + +DCM validates `not_before` is a future timestamp at submission. Past +timestamps are rejected with 422 (SCH-002). + +### 4.2 not_after deadline enforcement + +If `not_after` is set and the deadline passes before dispatch: + +``` +not_after reached without dispatch + ▼ Request status → FAILED + │ failure_reason: schedule_deadline_missed + ▼ request.failed event published (urgency: medium) + ▼ Consumer notified + ▼ Intent State marked terminal — no further retries (SCH-005) +``` + +The deadline worker scans every PT15S: + +```sql +SELECT request_uuid FROM scheduled_requests +WHERE status = 'SCHEDULED' + AND not_after IS NOT NULL + AND not_after < NOW() +``` + +For each match: transition to FAILED, emit `request.failed`, remove from +queue. + +### 4.3 Recurring schedule enforcement + +```yaml +schedule: + dispatch: recurring + cron: "0 2 * * 0" + max_occurrences: 4 + not_after: "2026-12-31T00:00:00Z" +``` + +For recurring schedules, DCM: + +1. Computes next ripe time via cron +2. Dispatches; increments `occurrences_so_far` +3. If `occurrences_so_far == max_occurrences`: marks terminal +4. If `not_after` exceeded: marks terminal +5. Otherwise: re-queues with next ripe time + +--- + +## 5. Consumer API additions + +### 5.1 Submit scheduled request + +Scheduling is an optional `schedule` field on the existing request endpoint: + +``` +POST /api/v1/requests +{ + "catalog_item_uuid": "", + "fields": { ... }, + "schedule": { + "dispatch": "at", + "not_before": "2026-04-01T02:00:00Z", + "not_after": "2026-04-01T06:00:00Z" + } +} + +Response 202: +{ + "request_uuid": "", + "entity_uuid": "", + "status": "SCHEDULED", + "scheduled_dispatch_at": "2026-04-01T02:00:00Z", + "schedule_deadline": "2026-04-01T06:00:00Z" +} +``` + +### 5.2 List scheduled requests + +``` +GET /api/v1/requests?status=SCHEDULED +``` + +### 5.3 Cancel scheduled request + +Existing endpoint — no new endpoint required: + +``` +DELETE /api/v1/requests/{request_uuid} + +# Works on SCHEDULED requests; moves status to CANCELLED +# Returns 409 if request is already dispatched (past SCHEDULED) + +Response 204 No Content +``` + +--- + +## 6. New events + +| Event | Urgency | Trigger | +|---|---|---| +| `request.scheduled` | info | Request entered SCHEDULED queue | +| `request.schedule_cancelled` | low | Scheduled request cancelled before dispatch | +| `request.schedule_deadline_missed` | medium | not_after passed without dispatch | + +These add to the `request.*` domain in +[udlm/contracts/event-catalog.md](https://github.com/croadfeldt/udlm/blob/main/contracts/event-catalog.md). + +--- + +## 7. Profile-governed scheduling constraints + +DCM enforces scheduling constraints per profile to reflect operational risk +tolerance: + +| Profile | Max scheduling horizon | Max concurrent scheduled/actor | Recurring max frequency | Window approval tier | +|---|---|---|---|---| +| `minimal` | P365D | unlimited | PT1H | auto | +| `dev` | P365D | 50 | PT1H | auto | +| `standard` | P90D | 20 | PT4H | reviewed | +| `prod` | P30D | 10 | PT12H | reviewed | +| `fsi` | P14D | 5 | PT24H | verified | +| `sovereign` | P7D | 3 | PT24H | authorized | + +- **Max scheduling horizon:** how far in the future `not_before` may be set; + beyond → reject with 422 +- **Max concurrent scheduled/actor:** how many SCHEDULED requests per actor; + exceeded → 429 +- **Recurring max frequency:** minimum interval between recurring dispatches; + cron more frequent than this → reject +- **Window approval tier:** authority tier required to create or modify a + Maintenance Window + +--- + +## 8. What operations support scheduling + +| Operation | Scheduling supported | Notes | +|---|---|---| +| Resource creation | ✅ | Full scheduling model | +| Resource update (PATCH) | ✅ | Full scheduling model | +| Suspend | ✅ | Full scheduling model | +| Resume | ✅ | Full scheduling model | +| Decommission | ✅ | Full scheduling model; `not_after` recommended | +| Rehydration | ✅ | Full scheduling model | +| TTL extension | ✅ | Full scheduling model | +| Discovery trigger | ❌ | Handled by Discovery Scheduler (see [`../convergence-engine/recovery-and-retry.md` §4](../convergence-engine/recovery-and-retry.md)) | + +--- + +## 9. Policy IDs (DCM realization) + +| Policy | Rule | +|---|---| +| `SCH-001-DCM` | DCM evaluates GateKeeper policies twice on scheduled requests: at declaration (fail-fast) and at dispatch (current state). Both must pass | +| `SCH-002-DCM` | DCM rejects scheduled requests with a past not_before (422) | +| `SCH-003-DCM` | DCM transitions requests failing dispatch-time policy re-evaluation to FAILED with failure_reason: schedule_policy_rejection | +| `SCH-004-DCM` | DCM permits cancellation of SCHEDULED requests at any time before dispatch via DELETE /api/v1/requests/{uuid} | +| `SCH-005-DCM` | DCM transitions requests with exceeded not_after to FAILED with failure_reason: schedule_deadline_missed; no retry | +| `SCH-006-DCM` | DCM treats Maintenance Windows as platform-level or tenant-scoped artifacts subject to standard DCM artifact lifecycle |