diff --git a/architecture/api/APIOverview.md b/architecture/api/APIOverview.md deleted file mode 100644 index 09a3cc4..0000000 --- a/architecture/api/APIOverview.md +++ /dev/null @@ -1,9 +0,0 @@ -## API Overview - -The DCM design has two layers of APIs; the control Plane APIs, where users and other services interact, and Service APIs, that encompass the functions encapsulated within a service boundary. Each handles the API payload differently, due to the declarative nature of DCM. - -The Control Plane APIs are designed to capture the user declarations and add them to the DCM datastore as the “requested state”. These APIs can also read from the “realized state” to return the status of the request, and provide any inventory specific information to the user. - -Service APIs work in reverse; they read from the “requested state” to pass along to the Execution Engine to affect the environment. When the Execution Engine completes the request, the metadata from the realized entity is saved in the “realized state” section. - -The metadata used for both the requested state and the realized state must align to the DCM data model. The Control Plane APIs and Service APIs can pull additional metadata from source controlled repos to fill in any additional information required to fulfill the request. diff --git a/architecture/api/ServiceOverview.md b/architecture/api/ServiceOverview.md deleted file mode 100644 index 6811b72..0000000 --- a/architecture/api/ServiceOverview.md +++ /dev/null @@ -1,11 +0,0 @@ -## Service Definition - -A service, in the context of DCM, is a component that can expose functions via an API to declaratively create, ingest, modify, delete, manage, and decommission a realized entity. That entity can be a component of the Platform to deliver services to development teams or an instantiation of an application and the components necessary to run it. - -It starts at the most atomic layer, the individual hardware items within the data center, and can encompass an orchestrated collection of subservices that manage the workloads running on the platform. For example, a VM service will encompass subservices provided by DNS, networking, and storage to instantiate a virtual machine. It would also make a call to the Operating System service to retrieve the current, approved OS image version to deploy and register it with the Operating System patch management service. - -These APIs are designed with a standard set of data components, and a structure to extend as required to declaratively deliver the defined outcome. It uses a standardized, hierarchical data model to support integration across services, along with additional details, including inventory and ownership information, that make it easier to manage the realized entities. There is also a standardized method to define a business rules and/or logic to ensure idempotency of the API calls and eases the management of the realized entities. These APIs should encompass all aspects of management, from Day 0 processes to stand up new gear, to Day 1 processes to deliver realized entities, followed by Day 2 management of those realized entities. Lastly, the Day N services should cleanly free up the consumed resources and update the inventory state. As higher order services are created, composed of these atomic structures, decommissioning services should be a mirror of the provisioning process. E.g. For every orchestrated creation process, there should be an equally comprehensive and orchestrated decommissioning process to ensure the correct declared state. - -The Business Rules should include references to which category the APIs fall under and which teams are authorized to access each of the functions. The metadata and rules definitions should be stored and managed under version control. - -The API exposed would ideally be a real API that calls automation to execute consistently. However, there is a need to address the existing environment and workloads. To “green the brownfield”, the use of Team APIs can be used to help manage those existing environments and align the organization over time. Team APIs (https://github.com/TeamTopologies/Team-API-template) are a way to standardize how to communicate with a team and describe where that team fits within the organization. This sets the table to eventually replace the Team API with a real Service API that executes automation to accomplish tasks. diff --git a/architecture/components/data_model/DCM-Data_Flow_Model.pdf b/architecture/components/data_model/DCM-Data_Flow_Model.pdf deleted file mode 100644 index ccd77bb..0000000 Binary files a/architecture/components/data_model/DCM-Data_Flow_Model.pdf and /dev/null differ diff --git a/architecture/components/data_model/overview.md b/architecture/components/data_model/overview.md deleted file mode 100644 index 7fced3a..0000000 --- a/architecture/components/data_model/overview.md +++ /dev/null @@ -1,102 +0,0 @@ -# DCM - Data Model - -## Data Model - Objectives - - * Develop a framework for disseminating IaaC / Config data from base standards to consumption pipelines - * Data model should enable consumption for provisioning and configuring of infrastructure and if applicable, software for production use - * Eg; VM provisioning - * Data model should enable for drift management, remediation and monitoring - * Can be applied / consumed via layers / templates / direct usage - * Allows for usage constraints - * Avoid unanticipated data interactions. - * Eg; during consumption, does the model allow for any results that are not easily and readily predictable or observable in advance - * Flexible to contain any entity definition - * Can be transformed to meet the needs of consumption tools - * Terraform - * Ansible - * 3rd party tools - * Must be stored in git repository - * Use / mimic k8s yaml format - -## Data Model - Rules - - * Host entities are the binding entities that realized entities attach to form a single unit - * Everything should start with a base entity - * Changes are implemented via a layer entity - * Widgets come implemented via a realized entity which is a unit scoped layer entity made real - * Layer entities override all previous entities in precedence chain - * Layer entities must contain the chain of parent entities and maintain precedence order - * Layer entities contain changes to parent entities or add / remove data - * No limit on number of parent entities - * All base and layer entities are immutable - * All entities must be versioned - * Changes to an entity result in a new version of the entity - * Changes to a realized entity are done via a layer entity, resulting in a new realized entity - * Versions consist of major, minor and revision - * Major - Compatibility breaking changes - * Minor - Feature additions, retain compatibility within major version - * Revision - config data change, retain compatibility within major version - * Version number components are base 10 numbers without limit - * Eg; 1.100.34 - * Any piece of data in an entity can have metadata subtag of the base tag / data - * Eg; Override preference - * Eg; Basis for config value - * Eg; Baseline for config value - * All entities in a chain should maintain the parent format - * Eg; yaml - * All realized entities should be directly consumable by target operations - * Each entity should be in source control, realized entities should be in CMDB and associated with the host entity - * Each entity should contain a link to it’s parent entity - * Each entity repo should remain online , or automatically redirect to new repo without impacting existing processes. - * Origination Date / Time stamp must be included in each entity - -## Data Model - Example Git Repo layout - - * Parent Repo(s) - * Standards - * Base templates - * Meta data - * Context - * Governance - * Origination date - * Git Repo and / or directory structure should consolidate and isolate on the following - * Standard Layers - * Base - level 1 - * Specific Use Layers - level 1 - * Downstream logical units - * Ships - level 2 - * Subs - level 2 - * Specific Use Layers - level 2 - * Enclaves in a logical unit - * Enclave A - level 3 - * Enclave B - level 3 - * Specific Use Layers - level 4 - * Customizations - * VM A realized config - level 4 - * Specific Use Layers - level 4 - -## Data Model - Example - - * Centralized config and provisioned state - * Layered / templated customization - * Everything as Code using standard Git processes - * Observable data supply chain - * Localization at point of need - * Data versioning enables audit / compatibility - * Ancillary Meta-Data - * JSON / YAML native - * API native consumption - * Flexible repo layout based on need - * Standardized naming schema provides clear reporting - -Diagram: Ship, Enclave, Infra, Shore, Request, Git, Cache - -## DCM - Data Model - Example data structure - -Diagram: Inventory and Configuration - -## Scale example for 40k linux vms - -40,000 = (CIS Benchmark, Baseline, DMZ, Payments) x (Common Linux Config, RHEL, CoreOS, OEL) x (RHEL version specifics, 6, 7, 8, 9) x Request - -40,000 = 3 x 4 x 3 x Request diff --git a/architecture/infrastructure_knowledge.yaml b/architecture/infrastructure_knowledge.yaml deleted file mode 100644 index 2bef800..0000000 --- a/architecture/infrastructure_knowledge.yaml +++ /dev/null @@ -1,125 +0,0 @@ -architecture: - name: Infrastructure Automation Platform - description: A platform for automating the provisioning and management of infrastructure resources. - version: 1.0 - -components: - - name: Orchestration - type: Workflow Engine - description: Manages the flow of infrastructure requests and coordinates the execution of tasks. - language: Python - dependencies: - - Service Catalog - - API Gateway - interfaces: - - HTTP - - Message Queue - properties: - scheduling: "Cron-based and Event-driven" - scalability: "Horizontal" - - - name: Service Catalog - type: API and Data Store - description: Provides a centralized catalog of available infrastructure services and their configurations. - language: Java - dependencies: - - Core Data - interfaces: - - REST API - properties: - data_format: "JSON" - versioning: "Semantic Versioning" - - - name: API Gateway - type: Reverse Proxy and API Manager - description: Provides a single entry point for accessing infrastructure services and manages authentication, authorization, and rate limiting. - language: Go - dependencies: - - Orchestration - interfaces: - - HTTP - - HTTPS - properties: - rate_limiting: "100 requests per minute" - authentication: "OAuth 2.0" - - - name: Core Data - type: Data Store - description: Stores core system data, including configuration settings, user profiles, and audit logs. - language: PostgreSQL - interfaces: - - SQL - properties: - replication: "Synchronous" - backup_frequency: "Daily" - - - name: Create API - type: REST API - description: Responsible for creating new infrastructure resources. - language: Python - dependencies: - - Service Catalog - interfaces: - - REST API - properties: - resource_types: ["VM", "Network", "Storage"] - - - name: Read API - type: REST API - description: Responsible for retrieving information about existing infrastructure resources. - language: Java - dependencies: - - Core Data - interfaces: - - REST API - - - name: Update API - type: REST API - description: Responsible for updating existing infrastructure resources. - language: Python - dependencies: - - Service Catalog - - Core Data - interfaces: - - REST API - - - name: Delete API - type: REST API - description: Responsible for deleting existing infrastructure resources. - language: Go - dependencies: - - Service Catalog - - Core Data - interfaces: - - REST API - -relationships: - - source: Orchestration - target: Service Catalog - type: "uses" - description: "Orchestration retrieves service definitions from the Service Catalog." - - - source: API Gateway - target: Orchestration - type: "routes to" - description: "API Gateway routes requests to the Orchestration component." - - - source: Create API - target: Service Catalog - type: "depends on" - description: "Create API retrieves resource configurations from the Service Catalog." - - - source: Read API - target: Core Data - type: "reads from" - description: "Read API retrieves resource information from the Core Data store." - - - source: Update API - target: Service Catalog - type: "updates" - description: "Update API updates resource definitions in the Service Catalog." - - - source: Update API - target: Core Data - type: "updates" - description: "Update API updates resource information in the Core Data store." \ No newline at end of file diff --git a/architecture/roles-responsibilities/DCM-Expected_Functions.pdf b/architecture/roles-responsibilities/DCM-Expected_Functions.pdf deleted file mode 100644 index 40742d8..0000000 Binary files a/architecture/roles-responsibilities/DCM-Expected_Functions.pdf and /dev/null differ diff --git a/architecture/roles-responsibilities/DCM-Expected_Relationships.pdf b/architecture/roles-responsibilities/DCM-Expected_Relationships.pdf deleted file mode 100644 index ea39c56..0000000 Binary files a/architecture/roles-responsibilities/DCM-Expected_Relationships.pdf and /dev/null differ diff --git a/architecture/roles-responsibilities/core_functions.md b/architecture/roles-responsibilities/core_functions.md deleted file mode 100644 index 7a880a3..0000000 --- a/architecture/roles-responsibilities/core_functions.md +++ /dev/null @@ -1,187 +0,0 @@ -# DCM Logical Components - -## Purpose, Role, Responsibility - -- **Purpose:** Why it exists -- **Role:** What it does -- **Responsibility:** What it owns - ---- - -## Control Plane - -### Functions - -#### Cost Analysis -- **Purpose:** Allow consumers to understand true cost of requesting/utilizing a service throughout its lifecycle. -- **Role:** Provide full costing of services during the lifecycle of a service through DCM. -- **Responsibility:** All accounting functions required to inform consumers of the true cost of services throughout the lifecycle of a service estimate or request. - -#### Orchestration -- **Purpose:** Ensure the lifecycle of a request from instantiation to fulfillment. -- **Role:** Coordinate and manage required DCM functions to facilitate the realization of the request. -- **Responsibility:** Manage and coordinate other functions in DCM to fulfill a request. - -#### Audit *(Combine with Observability?)* -- **Purpose:** Allow appropriate personas to review/validate transactions end-to-end through the DCM architecture. -- **Role:** Provide reports, analysis, and interactive interrogation of transaction details to support SRE, Auditors, Security teams, and Consumer personas. -- **Responsibility:** All functions related to reporting, analysis, and interrogation of transactions for security, RBAC, and operational details. - -#### Service Catalog *(Should this be the initiator of a service request?)* -- **Purpose:** Provide catalog of services available for consumption by consumers. -- **Role:** Facilitate the presentation of available services based on RBAC rules. -- **Responsibility:** Manage the lifecycle of available services and present a catalog based on RBAC rules. - -#### Rules Engine -- **Purpose:** Perform business, operational, and decision-making rules for various information domains (e.g., RBAC, service validation, data injection/manipulation). -- **Role:** Execute logic based on predefined rules/rulesets triggered by request payload data. -- **Responsibility:** All business logic required to facilitate a request payload becoming realized. - -#### IDM / IAM -- **Purpose:** Authenticate and retrieve info on consumers and providers of DCM. - -#### Observability -- **Purpose:** Log actions and report on functions and data used during DCM lifecycle functions and activities. - -#### Widget Discovery -- **Purpose:** Discover, audit, and report on infrastructure and data to determine the state of identified items. - -#### Message Bus -- **Purpose:** Send messages into and out of DCM in a common message queue format. - -#### API Gateway -- **Purpose:** Central clearing house for the DCM control plane. Provides API ingress for consumers, internal processes, and egress for Service Provider integration. Manages internal processes to ensure information is passed between appropriate DCM control plane services. - -#### Request Payload Processor -- **Purpose:** Create the target service payload based on consumer input, core data layers, and service layers combined into a unified data model. Consumed by service providers to effect the desired state. - -#### Drift Remediation / Notification -- **Purpose:** Detect drift between discovered state, requested state, and provisioned state. Perform notification and/or remediation as designated by Rules Engine. - ---- - -### Data Sources - -#### Service Catalog Cache -- **Purpose:** Storage cache for services offered by service providers. - -#### Core Rules -- **Purpose:** Rules scoped to all actions possibly taken by DCM, regardless of service provided. Also stores service-level rules not owned by the service provider. - - *Example: Security Team enforcing rules for a service* -- **Caveat:** In a modular design, service-specific rules may be injected via the service provider to ensure coordination with security/audit teams. - -#### IDM -- **Purpose:** Store for user authentication and information in support of RBAC services. - -#### Core Layers -- **Purpose:** Layers for the data payload outside the scope of specific services. - - *Example: Data Center layer, Rack layer* - -#### Core Data *(Could be combined with Core Layers?)* -- **Purpose:** Information outside the scope of specific services. - - *Example: Data Center Name, Service categories* - -#### Business Data / Groups *(Could be combined with IDM?)* -- **Purpose:** Group information for business groups related to consumers/services. - - *Example: Business Unit info, Organizational info, Product Owner* - -#### Discovered Widgets *(May belong in service providers)* -- **Purpose:** Cache information discovered by Widget Discovery. Used to compare discovered, requested, and provisioned state. - - *Example: Used for drift remediation and notification* - -#### Requested Widgets -- **Purpose:** Store all request payloads provided to service providers. Used for audit and drift remediation. - -#### Realized Widgets -- **Purpose:** Store all realized request payloads as provisioned by service providers. -- **Intent/Use:** Record what was realized/provisioned, including additional details from providers. Used for audit, reporting, and drift remediation. - -#### Log Store -- **Purpose:** Store all actions/data used during the lifecycle of a DCM action/process. -- **Intent/Use:** Used by audit, reporting, and historical record processes. - -#### Service Layer Cache -- **Purpose:** Cache of all layer sources of data used to create a request payload for a service. -- **Intent/Use:** Stores the data layers used to create the request payload. - ---- - -## Consumer Ingress - -### Functions - -#### Web UI -- **Purpose:** Provide web UI services to end users/web consumers of DCM. -- **Intent/Use:** Build UI, interact with consumers, and provide DCM services via web interface. - -#### Consumer API -- **Purpose:** Provide consumer-level API spec/services. -- **Intent/Use:** Direct consumption of services via unified API model. - -### Data Sources - -- N/A - ---- - -## Egress - -### Functions - -#### Messaging -- **Purpose:** Provide messaging protocol from DCM to external systems. -- **Intent/Use:** Interaction with external services outside the Interoperability API. - -#### Interoperability API -- **Purpose:** Provide common API spec and data model to interact with Service Providers. -- **Intent/Use:** Pass data to a DCM service provider for consumption and effecting changes or exchanging data. - -### Data Sources - -- N/A - ---- - -## Service Provider - -### Functions - -#### Services API -- **Purpose:** Communicate with DCM in unified API spec and data format. -- **Intent/Use:** Allow DCM to pass data to the Service Provider to effect changes or exchange data. - -#### Naturalization -- **Purpose:** Convert DCM unified data model to tool/service provider-specific format. -- **Intent/Use:** Enable unified data/API model execution via native tooling or automation. - -#### Realization -- **Purpose:** Effect the desired change or exchange information. -- **Intent/Use:** Service Provider tooling/automation performs steps to effect change or gather information. - -#### Denaturalization -- **Purpose:** Convert tool/provider-specific data into the DCM Unified data model. -- **Intent/Use:** Allow DCM to interact with realized data in its natural model for drift and audit processes. - -### Data Sources - -#### Service Catalog / APIs -- **Purpose:** List of services provided and appropriate APIs. -- **Intent/Use:** Inform DCM of available services and APIs. - -#### Service Rules -- **Purpose:** Business rules specific to services provided. -- **Intent/Use:** Apply conditional, data manipulation, and enforcement policies (e.g., Security, Authorization, Validation). - -#### Service Layer SCM -- **Purpose:** Location of service-specific data layers. -- **Intent/Use:** Inform DCM where to get required data layers for request payloads. - -### Data - -#### Service Request Payload -- **Purpose:** Data defining the requested service/data exchange. -- **Intent/Use:** Data in DCM format to be converted to provider-specific format for realization. - -#### Realized Request Payload -- **Purpose:** Data created as a result of realization, converted back to DCM format. -- **Intent/Use:** Snapshot of realized state for audit and drift processes. \ No newline at end of file diff --git a/architecture/sequence-diagrams/AI Prompt - List services for Use.md b/architecture/sequence-diagrams/AI Prompt - List services for Use.md deleted file mode 100644 index d7b681c..0000000 --- a/architecture/sequence-diagrams/AI Prompt - List services for Use.md +++ /dev/null @@ -1,30 +0,0 @@ -Sequence Prompts - List services for User - -Objects: -User -Web UI -Ingress API -IDM Service -Auth Source -Service Catalog -Request Payload Processor -Entity Layer Store -Rules Engine -Rules Store -Core Data -Service API -Service -Service Automation -Provisioned Store - -Messages: -Web UI requests services from Ingress API -Ingress API requests services from Service Catalog -Service Catalog forwards Service list and User to Rules Engine -Rules Engine requests User Group / Role membership from IDM -IDM returns Groups / Roles to Rules Engine -Rules Engine retrieves appropriate rules from Rules Store -Rules Store returns rules to Rules Engine -Rules Engine returns list of authorized services to Service Catalog -Service Catalog returns Service Catalog items to Web UI -Web UI returns Service UI to User diff --git a/architecture/sequence-diagrams/AI Prompt - Request service for U.md b/architecture/sequence-diagrams/AI Prompt - Request service for U.md deleted file mode 100644 index 5d17423..0000000 --- a/architecture/sequence-diagrams/AI Prompt - Request service for U.md +++ /dev/null @@ -1,33 +0,0 @@ -Sequence Prompts - Request service for User - -Objects: -User -Web UI -Ingress API -IDM Service -Auth Source -Service Catalog -Request Payload Processor -Entity Layer Store -Rules Engine -Rules Store -Core Data -Service API -Service -Service Automation -Provisioned Store - -Messages: -User selects and enters data for service to Web UI -Web UI sends service data to Ingress API -Ingress API send service data to Request Processor -Request Processor retrieves appropriate data layers from Core Layer Store -Core Layer Store returns appropriate data layers to Request Processor -Request Processor retrieves appropriate data layers via Service Layer Cache -Service Layer Cache returns appropriate data layers to Request Processor -Request Processor sends prepared request payload to Rules Engine -Rules Engine evaluates rules, approves request and updates request payload as needed -Rules Engine returns prepared request payload to Request Processor -Request Processor stores request payload in Requested Widget Store -Requested Widget Store returns confirmation to Request Processor -Request Processor sends request payload to API Gateway \ No newline at end of file diff --git a/architecture/sequence-diagrams/AI Prompt - Service Realization.md b/architecture/sequence-diagrams/AI Prompt - Service Realization.md deleted file mode 100644 index 4f2b409..0000000 --- a/architecture/sequence-diagrams/AI Prompt - Service Realization.md +++ /dev/null @@ -1,30 +0,0 @@ -Sequence Prompts - Service Realization - -Objects: -User -Web UI -Ingress API -IDM Service -Auth Source -Service Catalog -Request Payload Processor -Entity Layer Store -Rules Engine -Rules Store -Core Data -Service API -Service -Service Automation -Provisioned Store - -Messages: -API Gateway sends prepared request payload to appropriate Service Provider -Service Provider converts unified payload data to tool naturalized -Service Provider send native tool payload to Service Automation -Service Automation performs the required actions to realized the payload -Service Automation sends realized data payload with status to Service Provider -Service Provider converts realized data payload to unified payload format -Service Provider sends realized unified data payload to API Gateway -API Gateway stores realized data payload in Provisioned Data Store -API Gateway sends status back to Web UI -Web UI displays status to User \ No newline at end of file diff --git a/architecture/sequence-diagrams/AI Prompt - User Web UI Login.md b/architecture/sequence-diagrams/AI Prompt - User Web UI Login.md deleted file mode 100644 index f043c26..0000000 --- a/architecture/sequence-diagrams/AI Prompt - User Web UI Login.md +++ /dev/null @@ -1,28 +0,0 @@ -Sequence Prompts - User Web UI Login - -Objects: -User -Web UI -Ingress API -IDM Service -Auth Source -Service Catalog -Request Payload Processor -Entity Layer Store -Rules Engine -Rules Store -Core Data -Service API -Service -Service Automation -Provisioned Store - -Messages: -User initiates login to Web UI -Web UI sends authentication request to the Ingress API -Ingress API validates user login credentials via IDM -IDM Service authenticates user with target Auth source -Auth Source returns authentication success to IDM Service -IDM returns authentication token to Ingress API -Ingress API returns token to Web UI -Web UI returns token and entry page to User \ No newline at end of file diff --git a/architecture/sequence-diagrams/full-login-service-realization.md b/architecture/sequence-diagrams/full-login-service-realization.md deleted file mode 100644 index 71157c2..0000000 --- a/architecture/sequence-diagrams/full-login-service-realization.md +++ /dev/null @@ -1,68 +0,0 @@ -```mermaid -sequenceDiagram - participant User - participant WebUI as "Web UI" - participant IngressAPI as "Ingress API" - participant IDM as "IDM Service" - participant AuthSource as "Auth Source" - participant ServiceCatalog as "Service Catalog" - participant RulesEngine as "Rules Engine" - participant RulesStore as "Rules Store" - participant RequestProcessor as "Request Payload Processor" - participant CoreLayerStore as "Core Layer Store" - participant ServiceLayerCache as "Service Layer Cache" - participant RequestedWidgetStore as "Requested Widget Store" - participant APIGateway as "API Gateway" - participant ServiceProvider as "Service Provider" - participant ServiceAutomation as "Service Automation" - participant ProvisionedDataStore as "Provisioned Data Store" - - %% User Login - User->>WebUI: Initiate login - WebUI->>IngressAPI: Send authentication request - IngressAPI->>IDM: Validate login credentials - IDM->>AuthSource: Authenticate user - AuthSource-->>IDM: Return authentication success - IDM-->>IngressAPI: Return authentication token - IngressAPI-->>WebUI: Return token - WebUI-->>User: Return token and entry page - - %% List Services - WebUI->>IngressAPI: Request services - IngressAPI->>ServiceCatalog: Request services - ServiceCatalog->>RulesEngine: Forward service list & user - RulesEngine->>IDM: Request user group/role membership - IDM-->>RulesEngine: Return groups/roles - RulesEngine->>RulesStore: Retrieve rules - RulesStore-->>RulesEngine: Return rule - RulesEngine-->>ServiceCatalog: Return authorized services - ServiceCatalog-->>WebUI: Return service catalog items - WebUI-->>User: Return service UI - - %% Request Service - User->>WebUI: Selects and enters data for service - WebUI->>IngressAPI: Sends service data - IngressAPI->>RequestProcessor: Sends service data - RequestProcessor->>CoreLayerStore: Retrieve appropriate data layers - CoreLayerStore-->>RequestProcessor: Return data layers - RequestProcessor->>ServiceLayerCache: Retrieve appropriate data layers - ServiceLayerCache-->>RequestProcessor: Return data layers - RequestProcessor->>RulesEngine: Send prepared request payload - RulesEngine->>RulesEngine: Evaluate rules, approve & update payload - RulesEngine-->>RequestProcessor: Return prepared request payload - RequestProcessor->>RequestedWidgetStore: Store request payload - RequestedWidgetStore-->>RequestProcessor: Return confirmation - RequestProcessor->>APIGateway: Send request payload - - %% Service Realization - APIGateway->>ServiceProvider: Send prepared request payload - ServiceProvider->>ServiceProvider: Convert unified payload to tool naturalized - ServiceProvider->>ServiceAutomation: Send native tool payload - ServiceAutomation->>ServiceAutomation: Perform required actions - ServiceAutomation-->>ServiceProvider: Send realized data payload with status - ServiceProvider->>ServiceProvider: Convert realized data to unified format - ServiceProvider-->>APIGateway: Send realized unified data payload - APIGateway->>ProvisionedDataStore: Store realized data payload - APIGateway-->>WebUI: Send status - WebUI-->>User: Display status -``` \ No newline at end of file diff --git a/architecture/sequence-diagrams/full-stack-diagram.md b/architecture/sequence-diagrams/full-stack-diagram.md deleted file mode 100644 index 9e175b3..0000000 --- a/architecture/sequence-diagrams/full-stack-diagram.md +++ /dev/null @@ -1,88 +0,0 @@ -## Web Application Infrastructure Stack - -This diagram represents a complete web application infrastructure stack, from physical hardware to application components. - -**I. Physical Infrastructure (Layer 0 - Base)** - -* **Data Centers:** - * Geographically distributed for redundancy & low latency - * Tiered security (perimeter, zone, host) - * Environmental controls (temperature, humidity, fire suppression) -* **Power Infrastructure:** - * Redundant Power Feeds (Multiple Utility Providers) - * UPS (Uninterruptible Power Supplies) - Battery Backup - * Generators - Long-term Power Outage Protection - * Power Distribution Units (PDUs) - Controlled Power Distribution -* **Networking Infrastructure:** - * Multiple Internet Service Providers (ISPs) - Redundancy - * Core Routers & Switches - High-bandwidth backbone - * Firewall Clusters - Network perimeter security - * Load Balancers - Traffic distribution & failover - * DNS Servers - Domain name resolution - * Cabling (Fiber Optic, Ethernet) - Network connectivity -* **Server Racks:** - * Standard 19-inch racks - * Secure locking mechanisms - * Cable management systems - * Cooling systems (fans, liquid cooling) - -**II. Hardware Layer (Layer 1)** - -* **Servers:** (Dedicated or Virtualized) - * **Web Servers:** (e.g., Apache, Nginx) - Handle incoming HTTP requests - * High-performance CPUs - * Large amounts of RAM - * Fast storage (SSDs or NVMe) - * **Application Servers:** (e.g., Java EE servers, Node.js, Python/Django/Flask) - Execute application logic - * Powerful CPUs - * Significant RAM - * Fast storage - * **Database Servers:** (e.g., MySQL, PostgreSQL, MongoDB, Oracle) - Store and manage data - * Fast CPUs - * Large amounts of RAM - * High-performance storage (RAID arrays, SSDs) - * **Cache Servers:** (e.g., Redis, Memcached) - Store frequently accessed data for faster retrieval - * High-speed RAM - * Fast network connections - * **Firewall Servers:** Dedicated hardware/software firewalls -* **Networking Devices:** - * Switches (Layer 2 & Layer 3) - * Routers - * Load Balancers (Hardware & Software) - -**III. System Software Layer (Layer 2)** - -* **Operating Systems:** (Linux, Windows Server) – Base for all servers -* **Virtualization Software:** (VMware, Hyper-V, KVM) – Enable server virtualization -* **Containerization Software:** (Docker, Kubernetes) – Package and deploy applications in containers -* **Database Management Systems:** (MySQL, PostgreSQL, MongoDB, Oracle) – Manage databases -* **Web Servers:** (Apache, Nginx) - Handle HTTP requests -* **Caching Software:** (Redis, Memcached) -* **Monitoring & Logging Tools:** (Prometheus, Grafana, ELK Stack, Splunk) – Monitor system health and performance. -* **Security Software:** (Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS), Anti-virus, Vulnerability Scanners) - -**IV. Application Layer (Layer 3)** - -* **Web Application:** (e.g., written in Python/Django, Java/Spring, Node.js/Express) – The core application logic. -* **API Gateway:** Manage and secure APIs. -* **Message Queues:** (e.g., RabbitMQ, Kafka) - Asynchronous communication between services. -* **Search Engine:** (e.g., Elasticsearch, Solr) - Indexing and searching data. -* **Content Delivery Network (CDN):** Caching static content closer to users. - -**V. Network & Security Considerations** - -* **Firewalls:** (Network, Web Application) - Protect against unauthorized access. -* **Intrusion Detection/Prevention Systems (IDS/IPS):** Monitor and block malicious activity. -* **VPN (Virtual Private Network):** Secure remote access. -* **DDoS Protection:** Mitigate Distributed Denial of Service attacks. -* **SSL/TLS Encryption:** Secure communication between clients and servers. -* **Regular Security Audits and Penetration Testing** - -**Data Flow:** - -1. User sends request to CDN (if applicable). -2. Request is routed to Load Balancer. -3. Load Balancer distributes request to Web Servers. -4. Web Servers process request and interact with Application Servers. -5. Application Servers interact with Database Servers. -6. Data is returned to the user through the same path in reverse. diff --git a/architecture/sequence-diagrams/list-services.md b/architecture/sequence-diagrams/list-services.md deleted file mode 100644 index d404bdc..0000000 --- a/architecture/sequence-diagrams/list-services.md +++ /dev/null @@ -1,22 +0,0 @@ -```mermaid -sequenceDiagram - participant User - participant WebUI as "Web UI" - participant IngressAPI as "Ingress API" - participant ServiceCatalog as "Service Catalog" - participant RulesEngine as "Rules Engine" - participant IDM as "IDM Service" - participant RulesStore as "Rules Store" - - User->>WebUI: Request services - WebUI->>IngressAPI: Request services - IngressAPI->>ServiceCatalog: Request services - ServiceCatalog->>RulesEngine: Forward service list & user - RulesEngine->>IDM: Request user group/role membership - IDM-->>RulesEngine: Return groups/roles - RulesEngine->>RulesStore: Retrieve rules - RulesStore-->>RulesEngine: Return rules - RulesEngine-->>ServiceCatalog: Return authorized services - ServiceCatalog-->>WebUI: Return service catalog items - WebUI-->>User: Return service UI -``` \ No newline at end of file diff --git a/architecture/sequence-diagrams/provision-vm-with-vpc.md b/architecture/sequence-diagrams/provision-vm-with-vpc.md deleted file mode 100644 index 4f9af06..0000000 --- a/architecture/sequence-diagrams/provision-vm-with-vpc.md +++ /dev/null @@ -1,44 +0,0 @@ -sequenceDiagram - participant User - participant Authentication Server - participant Core Data Store - participant Service Catalog - participant Service Operational Store - participant Compute Service Provider - - User->>Authentication Server: Login with Credentials - activate Authentication Server - Authentication Server->>Authentication Server: Verify Credentials - alt Authentication Successful - Authentication Server-->>User: Login Successful - deactivate Authentication Server - User->>Compute Service Provider: Request Compute Offering - activate Compute Service Provider - Compute Service Provider->>Core Data Store: Retrieve Available VPCs (based on Role) - activate Core Data Store - Core Data Store-->>Compute Service Provider: Available VPCs - deactivate Core Data Store - Compute Service Provider->>Service Catalog: Retrieve Compute Offering Details - activate Service Catalog - Service Catalog-->>Compute Service Provider: Offering Details - deactivate Service Catalog - Compute Service Provider->>Service Operational Store: Retrieve Policies & Rules - activate Service Operational Store - Service Operational Store-->>Compute Service Provider: Policies & Rules - deactivate Service Operational Store - Compute Service Provider->>Compute Service Provider: Validate Request (Policies, Rules, VPC) - alt Request Valid - Compute Service Provider->>Compute Service Provider: Build Request Payload (Common Data Format) - Compute Service Provider->>Compute Service Provider: Transform Payload - Compute Service Provider->>Compute Service Provider: Provision VM (via Provider API) - Compute Service Provider <-- Compute Service Provider: Provision Status - Compute Service Provider->>Compute Service Provider: Update Provision Status - Compute Service Provider --> User: Report Success - else Request Invalid - Compute Service Provider --> User: Report Failure (Error Details) - end - deactivate Compute Service Provider - else Authentication Failed - Authentication Server --> User: Login Failed (Error Details) - deactivate Authentication Server - end diff --git a/architecture/sequence-diagrams/request-service.md b/architecture/sequence-diagrams/request-service.md deleted file mode 100644 index bf12e01..0000000 --- a/architecture/sequence-diagrams/request-service.md +++ /dev/null @@ -1,26 +0,0 @@ -```mermaid -sequenceDiagram - participant User - participant WebUI as "Web UI" - participant IngressAPI as "Ingress API" - participant RequestProcessor as "Request Processor" - participant CoreLayerStore as "Core Layer Store" - participant ServiceLayerCache as "Service Layer Cache" - participant RulesEngine as "Rules Engine" - participant RequestedWidgetStore as "Requested Widget Store" - participant APIGateway as "API Gateway" - - User->>WebUI: Selects and enters data for service - WebUI->>IngressAPI: Sends service data - IngressAPI->>RequestProcessor: Sends service data - RequestProcessor->>CoreLayerStore: Retrieve appropriate data layers - CoreLayerStore-->>RequestProcessor: Return data layers - RequestProcessor->>ServiceLayerCache: Retrieve appropriate data layers - ServiceLayerCache-->>RequestProcessor: Return data layers - RequestProcessor->>RulesEngine: Send prepared request payload - RulesEngine->>RulesEngine: Evaluate rules, approve & update payload - RulesEngine-->>RequestProcessor: Return prepared request payload - RequestProcessor->>RequestedWidgetStore: Store request payload - RequestedWidgetStore-->>RequestProcessor: Return confirmation - RequestProcessor->>APIGateway: Send request payload -``` \ No newline at end of file diff --git a/architecture/sequence-diagrams/service-realization.md b/architecture/sequence-diagrams/service-realization.md deleted file mode 100644 index 7f9c6cb..0000000 --- a/architecture/sequence-diagrams/service-realization.md +++ /dev/null @@ -1,20 +0,0 @@ -```mermaid -sequenceDiagram - participant APIGateway as "API Gateway" - participant ServiceProvider as "Service Provider" - participant ServiceAutomation as "Service Automation" - participant ProvisionedDataStore as "Provisioned Data Store" - participant WebUI as "Web UI" - participant User - - APIGateway->>ServiceProvider: Send prepared request payload - ServiceProvider->>ServiceProvider: Convert unified payload to tool naturalized - ServiceProvider->>ServiceAutomation: Send native tool payload - ServiceAutomation->>ServiceAutomation: Perform required actions - ServiceAutomation-->>ServiceProvider: Send realized data payload with status - ServiceProvider->>ServiceProvider: Convert realized data to unified format - ServiceProvider-->>APIGateway: Send realized unified data payload - APIGateway->>ProvisionedDataStore: Store realized data payload - APIGateway-->>WebUI: Send status - WebUI-->>User: Display status -``` \ No newline at end of file diff --git a/architecture/sequence-diagrams/user-login.md b/architecture/sequence-diagrams/user-login.md deleted file mode 100644 index da037d6..0000000 --- a/architecture/sequence-diagrams/user-login.md +++ /dev/null @@ -1,17 +0,0 @@ -```mermaid -sequenceDiagram - participant User - participant WebUI as "Web UI" - participant IngressAPI as "Ingress API" - participant IDM as "IDM Service" - participant AuthSource as "Auth Source" - - User->>WebUI: Initiate login - WebUI->>IngressAPI: Send authentication request - IngressAPI->>IDM: Validate login credentials - IDM->>AuthSource: Authenticate user - AuthSource-->>IDM: Return authentication success - IDM-->>IngressAPI: Return authentication token - IngressAPI-->>WebUI: Return token - WebUI-->>User: Return token and entry page -``` \ No newline at end of file diff --git a/example_use_cases/DCM-VM_VPC_Placement.pdf b/example_use_cases/DCM-VM_VPC_Placement.pdf deleted file mode 100644 index 99f157d..0000000 Binary files a/example_use_cases/DCM-VM_VPC_Placement.pdf and /dev/null differ diff --git a/overview.md b/overview.md deleted file mode 100644 index d5231cf..0000000 --- a/overview.md +++ /dev/null @@ -1,54 +0,0 @@ -DCM (Data Center Management) is envisioned as a framework designed to provide a "hyperscaler-like cloud experience" for on-premises infrastructure. It aims to centralize and automate the management, observation, and lifecycle of IT/IS services within an enterprise data center. - -Here's an architectural overview of DCM, based on the provided documents: - -**Core Principles & Goals:** - -* **Centralized Management**: Consolidate disparate automation efforts under a single control plane or "single pane of glass" to manage various data center components. -* **Automation & Self-Service**: Leverage existing automation tools (like Ansible Automation Platform) to offer a self-service model for provisioning, configuration, and day-to-day management. The goal is to enable application teams to define and deploy infrastructure as code. -* **Declarative Infrastructure**: Define the desired state of infrastructure, with the system working to achieve and maintain that state, aiding in reconciliation between discovered and intended inventory. -* **Composability**: Emphasize a composable architecture that allows for reusable services and operational efficiencies. -* **Technology Agnostic Framework**: Designed to be flexible and consume common data formats, translating them to appropriate technologies, though an opinionated model is available. -* **Full Lifecycle Management**: Enable complete lifecycle management of IT/IS services and ephemeral/declarative infrastructure. -* **Mimics Kubernetes Design**: Draws inspiration from Kubernetes' orchestration and declarative control plane. - -**Key Architectural Components:** - -1. **Centralized Data Model**: - * **Nature**: Git-based, flexible, and transformable. - * **Purpose**: Contains entity definitions for governance, metadata, location-specific, and application-specific data. - * **Consumption**: Designed to be consumed by various tools like Terraform and Ansible. - * **Storage**: Must be stored in a Git repository, potentially using a YAML format mimicking Kubernetes. - * **Rules**: Employs concepts like host entities, base entities, layer entities (for changes and overrides), and realized entities (widgets). -2. **API-Driven Interface**: - * **Core Design**: Everything within DCM is intended to be an API service, enabling consistent interaction. -3. **Automation Platform**: - * **Core Engine**: DCM is intended to be vendor agnostic, the framework will define the roles, resposibilities, interactions, and expected capabilities. This will allow organizations to implement the tools of their choice and still ensure interoperability. - * **Repository**: This is an integration detail which will be part of an opinated implementation. - * **Integrations**: This is an integration detail which will be part of an opinated implementation. -4. **Service Catalog**: - * **Functionality**: Provides a product catalog for service provisioning and management, enabling users to request and consume IT/IS services. -5. **Orchestration and Workflow Engine**: - * **Components**: Includes workflow orchestration capabilities, a request processor, and an execution plane. - * **Process**: Handles service requests, potentially involving entitlement verification, budget checks, and resource availability before executing the provisioning or management tasks. (e.g., "Request New MongoDB Cluster" example workflow shows request payload, entitlement verification, and sub-requests like new VM provisioning). -6. **Centralized Core Services**: - * **Includes**: Service provisioning, governance, observability, request processing, and centralized logging. - * **Inventory**: As part of the operation of the DCM data model, an inventory system of truth will be exposed along with extensive meta data, allowing for emergent capabilities such as auditing, data correlation, inventory system of truth, risk modeling, etc... -7. **Extensive extendability**: - * **Functionality**: Due to the API first, unified data model and extensive meta data, adding new capabilities to DCM is straight forward and consice. - - - -**Interaction Flow (Conceptual):** - -1. **User Request**: An application team or end-user requests a service via the self-service portal/API (e.g., "Request New MongoDB Cluster"). -2. **Request Processor**: The request is received and processed, including checks for authorization, budget, and resource availability. -3. **Service Manifest**: The request is translated into a service manifest, leveraging the centralized data model. -4. **Orchestration**: The orchestration engine takes the service manifest and coordinates various atomic services to fulfill the request. -5. **Execution Plane**: The automation platform, aka service provider, executes the necessary automation to provision and configure the infrastructure and services. -6. **Inventory Update**: The inventory is updated to reflect the deployed resources, and realization mechanisms to ensure the deployed state matches the intended state. -7. **Observability**: Centralized logging and observability tools provide insights into the deployed services and infrastructure. - -**Comparison and Evolution**: - -The architecture acknowledges lessons learned from previous efforts like CloudForms, aiming for a robust implementation that avoids past pitfalls while retaining valid design goals. It also seeks to integrate existing internal tools and certified collections, building upon existing automation efforts rather than reinventing them. \ No newline at end of file diff --git a/scope/questions.md b/scope/questions.md deleted file mode 100644 index b31d43a..0000000 --- a/scope/questions.md +++ /dev/null @@ -1 +0,0 @@ -* what about identifying, or resolving, discrepencies that cross over service domains?