diff --git a/.env.example b/.env.example index a98a0c1..8870ba3 100644 --- a/.env.example +++ b/.env.example @@ -4,8 +4,15 @@ # POSTGRES_PASSWORD=adminpass # K8s container service provider (profile: k8s-container) +# K8S_CONTAINER_SP_KUBECONFIG=/path/to/kubeconfig +# K8S_CONTAINER_SP_NAMESPACE=default # K8S_CONTAINER_SP_NAME=k8s-container-provider # K8S_CONTAINER_SP_EXTERNAL_SVC_TYPE=NodePort + +# Three-tier demo app service provider (profile: three-tier) +# Same kubeconfig + cluster namespace as k8s-container (K8S_CONTAINER_SP_* above). +# THREE_TIER_SP_NAME=three-tier-provider + # ACM cluster service provider (profile: acm-cluster) # Supports two platforms: KubeVirt (default) and BareMetal. # Both enabled by default (SP_ENABLED_PLATFORMS=kubevirt,baremetal). @@ -28,4 +35,5 @@ # POLICY_MANAGER_VERSION=main # KUBEVIRT_SERVICE_PROVIDER_VERSION=main # K8S_CONTAINER_SERVICE_PROVIDER_VERSION=main -# ACM_CLUSTER_SERVICE_PROVIDER_VERSION=main \ No newline at end of file +# ACM_CLUSTER_SERVICE_PROVIDER_VERSION=main +# THREE_TIER_DEMO_SERVICE_PROVIDER_VERSION=main diff --git a/RUN.md b/RUN.md index cbe328a..943300d 100644 --- a/RUN.md +++ b/RUN.md @@ -77,12 +77,34 @@ export ACM_CLUSTER_SP_DEFAULT_INFRA_ENV="my-infra-env" export ACM_CLUSTER_SP_AGENT_NAMESPACE="my-agent-namespace" ``` +### Three-tier demo app service provider + +To include the `three-tier-demo-service-provider`, set the required environment variables and +activate the `three-tier` profile: + +```bash +export K8S_CONTAINER_SP_KUBECONFIG="/path/to/kubeconfig" +make run-with-providers PROFILES=three-tier +``` + +When using Kind, complete the k8s-container setup (steps 1–5 in [K8s Container +SP with Kind](docs/k8s-container-sp-kind.md)) first. +For Pet Clinic usage, see [Three-Tier Demo App with Kind](docs/three-tier-app-kind.md). + +Optionally override the provider name or cluster namespace (`K8S_CONTAINER_SP_NAMESPACE` applies +to both k8s-container and three-tier SPs): + +```bash +export THREE_TIER_SP_NAME=my-provider +export K8S_CONTAINER_SP_NAMESPACE=default +``` + ### All providers To start all providers at once, set the required environment variables and run: ```bash -export KUBERNETES_KUBECONFIG="/path/to/kubeconfig" +export KUBEVIRT_KUBECONFIG="/path/to/kubeconfig" export K8S_CONTAINER_SP_KUBECONFIG="/path/to/kubeconfig" export ACM_CLUSTER_SP_KUBECONFIG="/path/to/kubeconfig" export ACM_CLUSTER_SP_PULL_SECRET="" @@ -92,12 +114,14 @@ export ACM_CLUSTER_SP_AGENT_NAMESPACE="my-agent-namespace" make run-with-providers ``` -This defaults to the `providers` Compose profile (all service providers). To start a single provider instead, pass `PROFILES=`: +This defaults to the `providers` Compose profile (all service providers, including the +three-tier demo SP). To start a single provider instead, pass `PROFILES=`: ```bash make run-with-providers PROFILES=kubevirt make run-with-providers PROFILES=k8s-container make run-with-providers PROFILES=acm-cluster +make run-with-providers PROFILES=three-tier ``` ## Verifying the deployment @@ -131,29 +155,32 @@ make compose-down ## Configuration -| Variable | Default | Description | -| ---------------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------- | -| `POSTGRES_USER` | `admin` | PostgreSQL username | -| `POSTGRES_PASSWORD` | `adminpass` | PostgreSQL password | -| `KUBERNETES_NAMESPACE` | `default` | Kubernetes namespace for KubeVirt VMs | -| `KUBERNETES_KUBECONFIG` | `~/.kube/config` | Path to kubeconfig on the host | -| `K8S_CONTAINER_SP_KUBECONFIG` | `~/.kube/config` | Path to kubeconfig on the host for the k8s-container-service-provider | -| `K8S_CONTAINER_SP_NAMESPACE` | `default` | Kubernetes namespace for k8s containers | -| `K8S_CONTAINER_SP_NAME` | `k8s-container-provider` | Provider name for the k8s-container-service-provider | -| `K8S_CONTAINER_SP_EXTERNAL_SVC_TYPE` | `NodePort` | Kubernetes Service type for external ports (`NodePort` or `LoadBalancer`) | -| `ACM_CLUSTER_SP_KUBECONFIG` | `~/.kube/config` | Path to kubeconfig on the host for the acm-cluster-service-provider | -| `ACM_CLUSTER_SP_NAMESPACE` | `default` | Kubernetes namespace for ACM hosted clusters | -| `ACM_CLUSTER_SP_NAME` | `acm-cluster-sp` | Provider name for the acm-cluster-service-provider | -| `ACM_CLUSTER_SP_BASE_DOMAIN` | _(none)_ | Base DNS domain for hosted clusters; can be overridden per-request via `provider_hints.acm.base_domain` | -| `ACM_CLUSTER_SP_PULL_SECRET` | _(required)_ | Base64-encoded dockerconfigjson pull secret for ACM hosted clusters | -| `ACM_CLUSTER_SP_DEFAULT_INFRA_ENV` | _(none)_ | **BareMetal only.** Default InfraEnv name; can be overridden per-request via `provider_hints.acm.infra_env` | -| `ACM_CLUSTER_SP_AGENT_NAMESPACE` | _(none)_ | **BareMetal only.** Namespace where Agent resources are located | -| `SERVICE_PROVIDER_MANAGER_VERSION` | `main` | Image tag for service-provider-manager | -| `CATALOG_MANAGER_VERSION` | `main` | Image tag for catalog-manager | -| `POLICY_MANAGER_VERSION` | `main` | Image tag for policy-manager | -| `PLACEMENT_MANAGER_VERSION` | `main` | Image tag for placement-manager | -| `KUBEVIRT_SERVICE_PROVIDER_VERSION` | `main` | Image tag for kubevirt-service-provider | -| `K8S_CONTAINER_SERVICE_PROVIDER_VERSION` | `main` | Image tag for k8s-container-service-provider | -| `ACM_CLUSTER_SERVICE_PROVIDER_VERSION` | `main` | Image tag for acm-cluster-service-provider | +| Variable | Default | Description | +| ------------------------------------------ | --------------------------- | ----------------------------------------------------------------------------------------------------------- | +| `POSTGRES_USER` | `admin` | PostgreSQL username | +| `POSTGRES_PASSWORD` | `adminpass` | PostgreSQL password | +| `KUBERNETES_NAMESPACE` | `default` | Kubernetes namespace for KubeVirt VMs | +| `KUBERNETES_KUBECONFIG` | `~/.kube/config` | Path to kubeconfig on the host | +| `KUBEVIRT_PROVIDER_NAME` | `kubevirt-service-provider` | Provider name and Compose service `container_name` | +| `K8S_CONTAINER_SP_KUBECONFIG` | `~/.kube/config` | Path to kubeconfig on the host for the k8s-container-service-provider | +| `K8S_CONTAINER_SP_NAMESPACE` | `default` | Kubernetes namespace for k8s containers | +| `K8S_CONTAINER_SP_NAME` | `k8s-container-provider` | Provider name for the k8s-container-service-provider | +| `K8S_CONTAINER_SP_EXTERNAL_SVC_TYPE` | `NodePort` | Kubernetes Service type for external ports (`NodePort` or `LoadBalancer`) | +| `ACM_CLUSTER_SP_KUBECONFIG` | `~/.kube/config` | Path to kubeconfig on the host for the acm-cluster-service-provider | +| `ACM_CLUSTER_SP_NAMESPACE` | `default` | Kubernetes namespace for ACM hosted clusters | +| `ACM_CLUSTER_SP_NAME` | `acm-cluster-sp` | Provider name for the acm-cluster-service-provider | +| `ACM_CLUSTER_SP_BASE_DOMAIN` | _(none)_ | Base DNS domain for hosted clusters; can be overridden per-request via `provider_hints.acm.base_domain` | +| `ACM_CLUSTER_SP_PULL_SECRET` | _(required)_ | Base64-encoded dockerconfigjson pull secret for ACM hosted clusters | +| `ACM_CLUSTER_SP_DEFAULT_INFRA_ENV` | _(none)_ | **BareMetal only.** Default InfraEnv name; can be overridden per-request via `provider_hints.acm.infra_env` | +| `ACM_CLUSTER_SP_AGENT_NAMESPACE` | _(none)_ | **BareMetal only.** Namespace where Agent resources are located | +| `SERVICE_PROVIDER_MANAGER_VERSION` | `main` | Image tag for service-provider-manager | +| `CATALOG_MANAGER_VERSION` | `main` | Image tag for catalog-manager | +| `POLICY_MANAGER_VERSION` | `main` | Image tag for policy-manager | +| `PLACEMENT_MANAGER_VERSION` | `main` | Image tag for placement-manager | +| `KUBEVIRT_SERVICE_PROVIDER_VERSION` | `main` | Image tag for kubevirt-service-provider | +| `K8S_CONTAINER_SERVICE_PROVIDER_VERSION` | `main` | Image tag for k8s-container-service-provider | +| `ACM_CLUSTER_SERVICE_PROVIDER_VERSION` | `main` | Image tag for acm-cluster-service-provider | +| `THREE_TIER_DEMO_SERVICE_PROVIDER_VERSION` | `main` | Image tag for three-tier-demo-service-provider | +| `THREE_TIER_SP_NAME` | `three-tier-provider` | Provider name for the three-tier-demo-service-provider | See [Image versions](README.md#image-versions) in the README for available tag formats and how to update. diff --git a/compose.yaml b/compose.yaml index 87a9591..9406386 100644 --- a/compose.yaml +++ b/compose.yaml @@ -129,7 +129,7 @@ services: postgres: { condition: service_healthy } k8s-container-service-provider: - profiles: ["providers", "k8s-container"] + profiles: ["providers", "k8s-container", "three-tier"] image: quay.io/dcm-project/k8s-container-service-provider:${K8S_CONTAINER_SERVICE_PROVIDER_VERSION:-main} pull_policy: always environment: @@ -184,6 +184,30 @@ services: depends_on: gateway: { condition: service_started } + three-tier-demo-service-provider: + profiles: ["providers", "three-tier"] + image: quay.io/dcm-project/three-tier-app-demo-service-provider:${THREE_TIER_DEMO_SERVICE_PROVIDER_VERSION:-main} + pull_policy: always + environment: + <<: *db-common + DB_NAME: three-tier-sp + CONTAINER_SP_URL: "http://k8s-container-service-provider:8080" + SP_NAME: ${THREE_TIER_SP_NAME:-three-tier-provider} + SP_ENDPOINT: "http://three-tier-demo-service-provider:8080" + DCM_REGISTRATION_URL: "http://service-provider-manager:8080/api/v1alpha1" + SP_NATS_URL: "nats://nats:4222" + SP_K8S_NAMESPACE: ${K8S_CONTAINER_SP_NAMESPACE:-default} + SP_K8S_KUBECONFIG: /kubeconfig + expose: + - "8080" + volumes: + - ${K8S_CONTAINER_SP_KUBECONFIG:-~/.kube/config}:/kubeconfig:ro,z + depends_on: + service-provider-manager: { condition: service_started } + nats: { condition: service_started } + k8s-container-service-provider: { condition: service_started } + postgres: { condition: service_healthy } + volumes: postgres_data: {} nats_data: {} diff --git a/docs/three-tier-app-kind.md b/docs/three-tier-app-kind.md new file mode 100644 index 0000000..6b12b1b --- /dev/null +++ b/docs/three-tier-app-kind.md @@ -0,0 +1,272 @@ +# Three-Tier Demo App Service Provider with Kind + +The Three-Tier Demo App Service Provider (SP) is a DCM plugin that provisions a Pet Clinic application +into a Kubernetes cluster. It requires the k8s-container-service-provider to be configured and running. + +## Setup (one-time, until containers are recreated) + +### CLI or curl + +In the Compose setup from this repo, the API gateway is **http://localhost:9080** by default. The +sections below give **`dcm`** ([CLI repo](https://github.com/dcm-project/cli)) and **`curl`** for the +same operations. They are equivalent, so use whichever you prefer. +Build or install **`dcm`** per the [CLI README](https://github.com/dcm-project/cli/blob/main/README.md). +Set **`DCM_API_GATEWAY_URL`** or **`~/.dcm/config.yaml`** if the gateway is not on localhost:9080. + +### Prerequisites + +Before starting the three-tier SP, you must complete the k8s-container-service-provider setup: + +1. Follow steps 1–5 in [k8s-container-sp-kind.md](k8s-container-sp-kind.md) to set up: + - A Kind cluster connected to the Compose network + - A kubeconfig configured to use the `kubernetes` alias + - The k8s-container-service-provider running and healthy + +Verify the setup: + +```bash +dcm sp provider list +``` + +Or: + +```bash +curl -s http://localhost:9080/api/v1alpha1/providers | jq '.providers[] | {name, health_status}' +``` + +> **Note:** The response should include `k8s-container-provider` in the list of available providers. + +### 1. Start the three-tier SP + +Using the Makefile (recommended): + +```bash +export K8S_CONTAINER_SP_KUBECONFIG="$(pwd)/kubeconfig.yaml" +make run-with-providers PROFILES=three-tier +``` + +Or the same compose invocation without Make: + +```bash +export K8S_CONTAINER_SP_KUBECONFIG="$(pwd)/kubeconfig.yaml" +podman compose --profile three-tier up -d +``` + +> **Note:** The three-tier profile automatically includes the k8s-container-service-provider +> as a dependency. Both services will start together. + +Verify it is running: + +```bash +podman ps --format "table {{.Names}}\t{{.Status}}" | grep -E 'three-tier|k8s-container' +``` + +Check the SP is registered with DCM: + +```bash +dcm sp provider list +``` + +Or: + +```bash +curl -s http://localhost:9080/api/v1alpha1/providers | jq '.providers[] | select(.name | contains("three-tier"))' +``` + +Note the provider’s `name` (by default `three-tier-provider` unless you changed `THREE_TIER_SP_NAME` in Compose). + +### 2. Provision the Pet Clinic application + +Register a placement policy once, then create a catalog item instance via the gateway (**`dcm`** or +**`curl`**). The **First time only (policy)** note sits next to the commands in each subsection. + +#### With DCM CLI + +Create a folder and both YAML files there (copy and paste the whole block): + +```bash +mkdir -p /tmp/dcm-petclinic + +cat > /tmp/dcm-petclinic/three-tier-placement.yaml <<'EOF' +display_name: Three-tier placement +policy_type: GLOBAL +enabled: true +priority: 100 +rego_code: | + package policies.three_tier_default + + main := { + "rejected": false, + "selected_provider": "three-tier-provider" + } if { + input.spec.catalog_item_id == "pet-clinic" + } +EOF + +cat > /tmp/dcm-petclinic/my-petclinic.yaml <<'EOF' +api_version: v1alpha1 +display_name: my-petclinic +spec: + catalog_item_id: pet-clinic + user_values: + - path: database.engine + value: postgres + - path: database.version + value: "18" +EOF +``` + +**`three-tier-placement.yaml`** is a placement policy so Policy Manager routes `pet-clinic` to +**`three-tier-provider`**. **`my-petclinic.yaml`** is the catalog instance (Pet Clinic item and DB +settings). + +**Gateway URL:** This repo does not ship a DCM config file. The CLI defaults to +`http://localhost:9080`, which matches a local api-gateway stack; **no config file is required** for +that case. Otherwise set `export DCM_API_GATEWAY_URL=…` for the session, or create +`~/.dcm/config.yaml`: + +```yaml +api-gateway-url: http://localhost:9080 +``` + +If Policy Manager has never been configured, add the rule below so the +platform selects your three-tier provider (same `name` as in step **1**, usually `three-tier-provider`). +Without it, the instance create call can fail with `policy response missing selected provider`. + +Rego sees the instance **`spec`** as **`input.spec`**. This example matches +`catalog_item_id == "pet-clinic"`; other catalog items skip this policy. Adjust +the path if your API nests `catalog_item_id` differently. + +```bash +dcm policy create --from-file /tmp/dcm-petclinic/three-tier-placement.yaml --id three-tier-placement +``` + +Then provision the three tier demo app: + +```bash +dcm catalog item list +dcm catalog instance create --from-file /tmp/dcm-petclinic/my-petclinic.yaml +``` + +#### With `curl` + +**First time only (policy):** Same placement rule as above. POST the policy below only if it is not +already present (duplicate returns **409**). + +```bash +curl -s -X POST 'http://localhost:9080/api/v1alpha1/policies?id=three-tier-placement' \ + -H 'Content-Type: application/json' \ + -d @- <<'JSON' +{ + "display_name": "Three-tier placement", + "policy_type": "GLOBAL", + "enabled": true, + "priority": 100, + "rego_code": "package policies.three_tier_default\n\nmain := {\n \"rejected\": false,\n \"selected_provider\": \"three-tier-provider\"\n} if {\n input.spec.catalog_item_id == \"pet-clinic\"\n}\n" +} +JSON +``` + +List catalog items to find the Pet Clinic offering: + +```bash +curl -s http://localhost:9080/api/v1alpha1/catalog-items | jq . +``` + +> **Note:** Look for a catalog item with a `display_name` that indicates a Pet Clinic service. Note its `uid` value. + +Create a catalog item instance: + +```bash +curl -sS -X POST http://localhost:9080/api/v1alpha1/catalog-item-instances \ + -H "Content-Type: application/json" \ + -d '{ + "api_version": "v1alpha1", + "display_name": "my-petclinic", + "spec": { + "catalog_item_id": "pet-clinic", + "user_values": [ + { "path": "database.engine", "value": "postgres" }, + { "path": "database.version", "value": "18" } + ] + } +}' +``` + +> **Note:** Catalog Manager only applies `user_values` whose `path` matches an **editable** field on +> that catalog item. For the seeded Pet Clinic offering (`pet-clinic`), only **`database.engine`** +> and **`database.version`** are editable; app and web images use the catalog defaults. + +### 3. Verify the Pet Clinic application is running + +Monitor the Pet Clinic deployment in Kubernetes: + +```bash +kubectl get pods -n default +``` + +Wait for the Pet Clinic pod(s) to reach `Running` status. + +Find the services: + +```bash +kubectl get svc -n default +``` + +The **web** tier is a Service whose name ends with `-web` (HTTP). On Kind, that Service is usually **ClusterIP** only, so your browser cannot reach it directly from the host. Forward a local port to it (adjust the service name and ports to match `kubectl get svc`; web often uses port **80**): + +```bash +kubectl port-forward -n default svc/-web 8080:80 +``` + +Then open **http://localhost:8080** in a browser. +Use **Ctrl+C** in the terminal to stop forwarding. + +## Troubleshooting + +### The three-tier SP fails to start + +If **`compose up`** errors or a container exits, inspect: + +```bash +podman compose --profile three-tier ps +podman ps -a --format "{{.Names}}\t{{.Status}}" | grep -i three-tier +podman compose --profile three-tier logs --tail=80 three-tier-demo-service-provider +podman compose --profile three-tier logs --tail=80 k8s-container-service-provider +``` + +Then check logs for a specific container name if needed: + +```bash +podman logs +``` + +Common issues: + +- **Kubeconfig not mounted correctly:** Verify `K8S_CONTAINER_SP_KUBECONFIG` is set and the file exists. +- **k8s-container-service-provider not running:** Ensure the k8s-container-service-provider is healthy. +- **NATS or Postgres not ready:** Check that `nats` and `postgres` services are running. + +### Pet Clinic pod fails to start + +Check the pod events: + +```bash +kubectl describe pod -n default +``` + +Check logs: + +```bash +kubectl logs -n default +``` + +### Typical errors and causes + +| Problem | Cause | +|---|---| +| `policy response missing selected provider` | No enabled Policy Manager policy sets `selected_provider`, or the value does not match a registered provider `name`. Add the policy in step **2**. | +| `provider '…' is not in ready state (not_ready)` | Service Provider Manager periodically GETs `{endpoint}/health`. If that fails repeatedly, the provider becomes `not_ready` and SPRM rejects provisioning. Ensure the three-tier process responds with 2xx on that path (the demo SP redirects `/health` to `/api/v1alpha1/health`), then wait for the next health check cycle or restart the stack. Verify with `GET /api/v1alpha1/providers` and `health_status: "ready"`. | +| Three-tier SP cannot provision apps without k8s-container-service-provider | The three-tier SP is a high-level orchestration layer that delegates resource provisioning to a k8s-container-service-provider | +| App is unreachable from the host | Kind often exposes the web Service as ClusterIP only. Use `kubectl port-forward` to the **`-web`** Service (step **3**). | +| Deployment hangs or fails | Missing environment variables or unhealthy dependencies (NATS, Postgres, k8s-container-service-provider) | diff --git a/hack/postgres-init/01-create-databases.sql b/hack/postgres-init/01-create-databases.sql index ac34009..363cb86 100644 --- a/hack/postgres-init/01-create-databases.sql +++ b/hack/postgres-init/01-create-databases.sql @@ -3,3 +3,4 @@ CREATE DATABASE "service-provider"; CREATE DATABASE "policy-manager"; CREATE DATABASE "catalog-manager"; CREATE DATABASE "placement-manager"; +CREATE DATABASE "three-tier-sp";