diff --git a/MIGRATION.md b/MIGRATION.md new file mode 100644 index 0000000..90a6d07 --- /dev/null +++ b/MIGRATION.md @@ -0,0 +1,888 @@ +# Migration: Single-Server Platforma → Kubernetes + +End-to-end procedure to move an existing single-server Platforma instance (running the standalone binary or Docker image) onto a Kubernetes cluster managed by this Helm chart. + +The migration moves two pieces of state: + +1. **Database** — RocksDB on the source server, dumped to a gzipped file and restored into the chart's `platforma-database` PVC. +2. **Primary storage** — bucket-to-bucket S3 sync (or equivalent on GCP) into the new cloud bucket the cluster will use. + +The procedure is split into three phases that run before, during, and after the `helm install`. Re-run safety works differently for the in-cluster pipeline vs. the AWS-side work: + +- **PVC-bound in-cluster steps** (Step 1.4 DB download, 1.5 restore, 1.6 cleanup, and Phase 3 invalidate) gate on a `/data/database/.migration-complete` marker written by Phase 3 after cache invalidation succeeds. Re-running any of them after a successful end-to-end migration is a no-op. +- **`s5cmd` sync** (Step 1.3) is idempotent on its own — it compares ETags and skips already-copied objects, so re-running is cheap. +- **DB dump** (Step 0.3) just overwrites `backup.gz` on the source host. +- **S3 Batch Replication** (Step 0.6.5) is **not idempotent** as written — `ClientRequestToken` uses `$(date +%s)`, so each `aws s3control create-job` call creates a new job. Don't blindly re-run the create-job command; instead, check the existing job's status (Step 0.6.6) and only re-create if the original truly failed. + +## Audience and Scope + +This guide is written for an operator standing up the target cluster with **Terraform** (or any IaC equivalent of `eksctl` / CloudFormation). It assumes: + +- You have provisioned the target cluster's infrastructure following the resources listed in [`infrastructure/aws/advanced-installation.md`](infrastructure/aws/advanced-installation.md) — that doc enumerates every IAM policy, IRSA role, S3 bucket, EFS filesystem, ACM certificate, and StorageClass that the Helm chart depends on. Translate each `aws ...` call into the equivalent Terraform resource. +- You have `kubectl` access to the cluster and admin permissions in the target namespace (`platforma` by default). +- You have AWS CLI access to both the **source** primary-storage S3 bucket (read) and the **destination** bucket (write). Cross-account migrations need an extra IAM user — see [Cross-Account Sync](#cross-account-sync) below. + +If you are setting up the cluster from scratch on AWS using CloudFormation, the same migration is wired into [`cloudformation-eks-1-35.yaml`](infrastructure/aws/cloudformation-eks-1-35.yaml) via the `MigrationDatabaseS3Uri` / `MigrationSourceBucket` parameters. This guide is the manual equivalent. + +## Prerequisites + +| Requirement | Notes | +|---|---| +| Source Platforma version | Same major.minor as the target chart's image. Cross-version restore is not supported. | +| Source database dump | Created with `curl -s http://localhost:9091/db/state_raw \| gzip > backup.gz`, then uploaded to S3. The endpoint is part of the debug API — it requires `--debug-enabled` (or `PL_DEBUG_ENABLED=true`) on the source server, and binds to `127.0.0.1:9091` by default. See [Phase 0](#phase-0-source-side-preparation). | +| Target cluster ready | Cluster Autoscaler, Kueue + AppWrapper, EFS CSI, EBS CSI, S3 IRSA all working. See [advanced-installation.md](infrastructure/aws/advanced-installation.md) Steps 1–8. | +| Target S3 bucket created | Empty. The migration will sync data into it. | +| `MI_LICENSE` available | The migration uses the Platforma image to run `--restore-db` and `--invalidate-caches`, both of which need a valid license. | +| `kubectl` and AWS CLI | Configured for the target cluster and account. | +| Source instance frozen | Stop writes during the migration. The DB dump is a point-in-time snapshot. New work submitted after the dump is taken will be lost. | + +## Migration Variables + +Set these once before running any step. Each pre-helm command references them. + +```bash +# Target cluster +export NAMESPACE="platforma" +export REGION="eu-central-1" +export CLUSTER_NAME="my-platforma-cluster" # EKS cluster name — used for IAM role/policy names +export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)" +export S3_BUCKET="platforma--" # destination bucket (already created) + +# Platforma image tag used by the migration pods (must match the source server's major.minor) +export PLATFORMA_VERSION="3.0.1" +# Helm chart version installed in Phase 2 — usually tracks PLATFORMA_VERSION but can differ +export CHART_VERSION="${PLATFORMA_VERSION}" + +# License — required by the Platforma-image pods (restore + invalidate) +export MI_LICENSE="..." # paste the license token, or load from a file + +# Source dump (required) +export MIGRATION_DATABASE_S3_URI="s3://my-backups/platforma-backup.gz" + +# Source primary storage (optional — omit to skip storage sync) +export MIGRATION_SOURCE_BUCKET="old-platforma-bucket" +export MIGRATION_SOURCE_PREFIX="" # leave empty for bucket root +export MIGRATION_SOURCE_REGION="eu-west-1" # defaults to $REGION if empty + +# Cross-account credentials (optional) +export MIGRATION_DB_ACCESS_KEY="" # AKIA... — needed if dump is in another account +export MIGRATION_DB_SECRET_KEY="" +export MIGRATION_STORAGE_ACCESS_KEY="" # AKIA... — needed if source bucket is in another account +export MIGRATION_STORAGE_SECRET_KEY="" +``` + +## Architecture of the Migration Job + +All migration work runs as one-shot pods inside the target namespace. They use: + +- The official Platforma image (`quay.io/milaboratories/platforma:${PLATFORMA_VERSION}`) for DB restore and cache invalidation. +- The `amazon/aws-cli:2.27.22` image for S3 download/sync. +- Pod `securityContext`: `fsGroup: 1010`, `runAsUser: 1010`, `runAsGroup: 1010` (matches the chart's non-root `pl` user — UID/GID **must** be 1010 or restored RocksDB files will be unreadable by the server pod). +- The `platforma-database` PVC (RWO, 50Gi, gp3) mounted at `/data/database`. +- The `platforma-license` Secret for the Platforma-image pods. + +The reference implementation is the script at [`infrastructure/aws/migration.sh`](infrastructure/aws/migration.sh). The YAML below is what that script generates and applies — provided here so you can run the migration manually or wire it into your own Terraform `null_resource` / `helm_release` `provisioner`. + +--- + +## Phase 0: Source-Side Preparation + +Steps 0.1–0.5 run on the **source** single-server host (the machine running the standalone binary or Docker image). Step 0.6 runs from a workstation with AWS CLI access — it is entirely AWS-side and has no dependency on the target Kubernetes cluster, so kicking it off here lets AWS copy data in the background while you complete the rest of Phase 0 and all of Phase 1. + +Phase 0 produces: + +1. A gzipped database dump uploaded to S3 (referenced by `MIGRATION_DATABASE_S3_URI`). +2. A frozen primary-storage S3 bucket whose name you record as `MIGRATION_SOURCE_BUCKET`. +3. For multi-TB buckets: a running S3 Batch Replication job that will populate the destination bucket before Phase 2. + +### Step 0.1: Freeze the Source Instance + +Stop all writes — pause Desktop App users and any automated submitters. The database dump and the storage bucket must represent the same point in time; new objects written after the dump is taken will be invisible to the restored DB and orphaned in storage. + +### Step 0.2: Enable the Debug API on the Source + +The `/db/state_raw` endpoint is gated behind the debug API. If it is not already enabled, restart the source server with `--debug-enabled` (or set `PL_DEBUG_ENABLED=true`). The endpoint binds to `127.0.0.1:9091` by default — run the dump command **on the source host itself**, or temporarily change `--debug-ip` to bind to a routable interface. + +Verify: + +```bash +curl -sf http://localhost:9091/db/stats >/dev/null && echo "debug API reachable" +``` + +### Step 0.3: Dump the Database + +```bash +curl -s http://localhost:9091/db/state_raw | gzip > backup.gz +``` + +The endpoint streams a tab-separated key/value dump of RocksDB; piping into `gzip` writes a single compressed file. Dump time scales with DB size — a 10 GB live DB typically compresses to 1–3 GB. + +> The same dump can also be produced offline with the `pl-db-cli` binary that ships in the Platforma image (`/app/pl-db-cli dump --db-dir=...`) if the source server cannot expose the debug API. + +### Step 0.4: Upload the Dump to S3 + +Use whatever S3 bucket the new cluster can read from. For same-account migrations, any bucket the migration IRSA role can `s3:GetObject` from works — that role is defined later in [Step 1.2.1](#step-121-recommended--irsa-service-account-for-migration-pods), so make sure the bucket you choose here will be covered by the policy you attach there. + +```bash +aws s3 cp backup.gz s3://my-backups/platforma-backup.gz +``` + +Record the URI as `MIGRATION_DATABASE_S3_URI` in the variables block above. + +### Step 0.5: Identify the Source Primary-Storage Bucket + +If the source instance used S3 (not local disk) for primary storage, note its bucket name and region — they become `MIGRATION_SOURCE_BUCKET` and `MIGRATION_SOURCE_REGION`. The bucket is copied into the destination either by [Step 0.6](#step-06-s3-batch-replication-for-multi-tb-buckets) (S3 Batch Replication, recommended for multi-TB buckets — start it now, finishes in the background) or by [Step 1.3](#step-13-sync-primary-storage-s5cmd--1-tb) (in-cluster `s5cmd`, for buckets up to ~1 TB). Confirm the bucket is no longer being written to (Step 0.1) before starting either. + +### Step 0.6: S3 Batch Replication for Multi-TB Buckets + +Use this path **instead of the in-cluster `s5cmd` sync (Step 1.3)** when the source bucket holds multiple TB or tens of millions of objects. S3 Batch Replication is an AWS-managed bulk copy that runs on the S3 service fleet rather than from a pod — throughput scales with the manifest size, not with your VPC bandwidth or pod CPU, and it can copy a multi-TB bucket in hours that would take days with a single `s5cmd` pod. + +All steps run from a workstation with AWS CLI access to the cluster account. Kick the job off here — Batch Replication has no dependency on the Kubernetes cluster, so it copies in the background while you complete the rest of Phase 0 (DB dump) and Phase 1 (cluster setup, DB restore). The destination bucket must be fully populated **before `helm install`** in Phase 2; verify completion (Step 0.6.6) before moving on. + +#### Prerequisites + +| Requirement | Notes | +|---|---| +| **Versioning enabled on both buckets** | Replication operates on object versions. AWS rejects the replication config otherwise. | +| **Both buckets in the same AWS partition** | `aws` (commercial), `aws-us-gov`, etc. Cross-partition replication is not supported. | +| **Same-account or pre-configured cross-account trust** | For cross-account, the destination bucket policy must grant the source-account replication role `s3:ReplicateObject`, `s3:ReplicateDelete`, `s3:ReplicateTags`, `s3:GetObjectVersionTagging`, and `s3:ObjectOwnerOverrideToBucketOwner`. | +| **Manifest report bucket (optional but recommended)** | Batch Replication can generate the manifest on the fly; the completion report needs an S3 location to write to. Reuse the destination bucket with a `reports/` prefix. | + +#### Step 0.6.1: Enable Versioning on Both Buckets + +```bash +aws s3api put-bucket-versioning \ + --bucket ${MIGRATION_SOURCE_BUCKET} \ + --region ${MIGRATION_SOURCE_REGION} \ + --versioning-configuration Status=Enabled + +aws s3api put-bucket-versioning \ + --bucket ${S3_BUCKET} \ + --region ${REGION} \ + --versioning-configuration Status=Enabled +``` + +Versioning cannot be fully removed once enabled, only suspended. Suspending after the migration is fine — Platforma does not depend on versioning at runtime. + +#### Step 0.6.2: Create the Replication IAM Role + +This role is assumed by S3 itself when it copies each object. It needs read on the source and write on the destination. + +```bash +# Trust policy — only s3.amazonaws.com can assume it +cat > /tmp/replication-trust.json < /tmp/replication-policy.json < /tmp/replication-config.json < /tmp/batch-trust.json < /tmp/batch-policy.json < /tmp/batch-job.json </dev/null || true + +kubectl create secret generic platforma-license \ + -n ${NAMESPACE} \ + --from-literal=MI_LICENSE="${MI_LICENSE}" +``` + +If the cluster does not already have a `gp3` StorageClass, create one. Labels and annotations let Helm adopt it later — without them, `helm install` fails with `invalid ownership metadata`. + +```yaml +# gp3-storageclass.yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: gp3 + labels: + app.kubernetes.io/managed-by: Helm + annotations: + meta.helm.sh/release-name: platforma + meta.helm.sh/release-namespace: platforma +provisioner: ebs.csi.aws.com +parameters: + type: gp3 +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Delete +allowVolumeExpansion: true +``` + +```bash +kubectl apply -f gp3-storageclass.yaml +``` + +### Step 1.2: Pre-Create the Database PVC + +The chart will adopt this PVC on install. Same labelling trick as the StorageClass. + +```yaml +# database-pvc.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: platforma-database + namespace: platforma + labels: + app.kubernetes.io/managed-by: Helm + annotations: + meta.helm.sh/release-name: platforma + meta.helm.sh/release-namespace: platforma +spec: + accessModes: [ReadWriteOnce] + storageClassName: gp3 + resources: + requests: + storage: 50Gi +``` + +```bash +kubectl apply -f database-pvc.yaml +``` + +### Step 1.2.1: Recommended — IRSA Service Account for Migration Pods + +Inline AWS keys in the pod YAML (`MIGRATION_DB_ACCESS_KEY` / `MIGRATION_STORAGE_ACCESS_KEY`) work for cross-account migrations, but for the **common case** where the dump bucket and the source primary-storage bucket live in the same AWS account as the cluster, attach an IRSA role to a dedicated service account and let the migration pods assume it. No long-lived keys, no Secret juggling. + +The chart's runtime SAs (`platforma`, `platforma-jobs`) only have permissions on the *destination* bucket — they cannot read the source bucket, so reusing them does not work. Create a separate SA with read access to the source plus write access to the destination. + +**Step A — IAM policy and role.** Run on a workstation with `aws` CLI access to the cluster account: + +```bash +# Inline policy: read on source, read+write on destination +cat > /tmp/migration-policy.json < /tmp/migration-trust.json < The same pattern works on GKE — replace the IAM role + annotation with a Workload Identity binding (`iam.gke.io/gcp-service-account`). + +### Step 1.3: Sync Primary Storage (`s5cmd`, < 1 TB) + +**Skip this step if you already started S3 Batch Replication in [Step 0.6](#step-06-s3-batch-replication-for-multi-tb-buckets)** — that path handles multi-TB buckets entirely on the AWS side. This step is the in-cluster alternative for buckets up to ~1 TB / 10 M objects, where running a pod is simpler than configuring Batch Replication. + +Also skip if your source instance kept its primary storage on a local filesystem and you want the new cluster to start with an empty bucket. + +| Bucket size | Use | +|---|---| +| **< ~1 TB / < 10 M objects** | This step (`s5cmd` in a pod) — simple, idempotent, runs alongside the other migration pods. | +| **≥ 1 TB or ≥ 10 M objects** | [Step 0.6](#step-06-s3-batch-replication-for-multi-tb-buckets) (S3 Batch Replication) — kicked off in Phase 0, AWS-managed, runs in parallel with everything else. | + +Platforma cannot serve any project that references objects in primary storage until those objects exist in the destination bucket — restoring the database alone is not enough. **Start this step before the database restore** so it runs in parallel with the much faster DB work in Steps 1.4–1.5. + +The pod below uses [`s5cmd`](https://github.com/peak/s5cmd) instead of `aws s3 sync`. Both tools rely on S3 server-side COPY (no data leaves AWS, no egress charges) when source and destination are S3 buckets, but `s5cmd` parallelizes listing and copy operations aggressively — in practice **10–30× faster** than `aws s3 sync` for buckets with many small objects, which is the common shape of Platforma's primary storage. + +```yaml +# mgr-sync.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mgr-sync + namespace: platforma +spec: + serviceAccountName: platforma-migration # IRSA SA from Step 1.2.1 — drop for cross-account + securityContext: + fsGroup: 1010 + runAsUser: 1010 + runAsGroup: 1010 + restartPolicy: Never + containers: + - name: mgr-sync + image: peakcom/s5cmd:v2.3.0 + command: ["/bin/sh", "-ec"] + args: + - | + # --numworkers: parallel S3 operations (default 256 is fine; raise for very wide buckets). + # sync uses server-side COPY when both sides are S3 — no bytes through this pod. + # Trailing /* on the source is required by s5cmd to expand the prefix. + # s5cmd sync is idempotent (ETag-compared), so re-running on completed buckets is cheap. + s5cmd --numworkers 256 \ + --source-region ${MIGRATION_SOURCE_REGION} \ + --destination-region ${REGION} \ + sync "s3://${MIGRATION_SOURCE_BUCKET}/${MIGRATION_SOURCE_PREFIX}*" \ + "s3://${S3_BUCKET}/" + echo Storage sync complete + # env section ONLY needed for cross-account source bucket — omit when using IRSA + # env: + # - { name: AWS_ACCESS_KEY_ID, value: "${MIGRATION_STORAGE_ACCESS_KEY}" } + # - { name: AWS_SECRET_ACCESS_KEY, value: "${MIGRATION_STORAGE_SECRET_KEY}" } +``` + +This pod intentionally does **not** mount the `platforma-database` PVC. The DB-handling pods (`mgr-download`, `mgr-restore`, `mgr-invalidate`) all mount that PVC with `ReadWriteOnce` access — Kubernetes would refuse to schedule `mgr-sync` to a different node while one of them is running. Skipping the mount lets `mgr-sync` run truly in parallel with the DB pipeline. + +```bash +envsubst < mgr-sync.yaml | kubectl apply -f - +kubectl wait pod/mgr-sync -n ${NAMESPACE} \ + --for=jsonpath='{.status.phase}'=Succeeded --timeout=21600s +kubectl logs -f mgr-sync -n ${NAMESPACE} +kubectl delete pod/mgr-sync -n ${NAMESPACE} +``` + +Tuning notes: + +- `--numworkers` controls the size of the worker pool. 256 saturates a single small pod on a same-region intra-AWS copy; raise to 1024 for very large buckets and bump pod CPU requests accordingly. +- `s5cmd sync` is idempotent — it compares object size/ETag and skips unchanged keys, so reruns after a failure resume cheaply. +- `s5cmd` does not attempt to copy ACLs or object tags, so the `aws s3 sync ... --copy-props none` workaround used previously for cross-account migrations is unnecessary here. +- `MIGRATION_SOURCE_PREFIX` should end with `/` if set (e.g. `data/`). Without a trailing slash, the `${MIGRATION_SOURCE_PREFIX}*` glob would match sibling prefixes too — e.g. prefix `data` matches both `data/` and `data2/`. + +### Step 1.4: Download the Database Dump + +```yaml +# mgr-download.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mgr-download + namespace: platforma +spec: + serviceAccountName: platforma-migration # IRSA SA from Step 1.2.1 — drop this for cross-account + securityContext: + fsGroup: 1010 + runAsUser: 1010 + runAsGroup: 1010 + restartPolicy: Never + volumes: + - name: db + persistentVolumeClaim: + claimName: platforma-database + containers: + - name: mgr-download + image: amazon/aws-cli:2.27.22 + command: ["/bin/sh", "-ec"] + args: + - | + if [ -f /data/database/.migration-complete ]; then echo Already completed; exit 0; fi + aws s3 cp ${MIGRATION_DATABASE_S3_URI} /data/database/backup.gz + echo Download complete + # env section ONLY needed for cross-account dumps — omit when using IRSA + # env: + # - { name: AWS_ACCESS_KEY_ID, value: "${MIGRATION_DB_ACCESS_KEY}" } + # - { name: AWS_SECRET_ACCESS_KEY, value: "${MIGRATION_DB_SECRET_KEY}" } + volumeMounts: + - { name: db, mountPath: /data/database } +``` + +When using the `platforma-migration` SA from Step 1.2.1, the pod assumes the IAM role transparently — no `env:` block needed. For cross-account dumps that the SA's role does not cover, uncomment the `env:` block and inline the access keys (or mount them from a Secret). + +```bash +envsubst < mgr-download.yaml | kubectl apply -f - +kubectl wait pod/mgr-download -n ${NAMESPACE} \ + --for=jsonpath='{.status.phase}'=Succeeded --timeout=600s +kubectl logs mgr-download -n ${NAMESPACE} +kubectl delete pod/mgr-download -n ${NAMESPACE} +``` + +### Step 1.5: Restore the Database + +```yaml +# mgr-restore.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mgr-restore + namespace: platforma +spec: + securityContext: + fsGroup: 1010 + runAsUser: 1010 + runAsGroup: 1010 + restartPolicy: Never + volumes: + - { name: db, persistentVolumeClaim: { claimName: platforma-database } } + - { name: main, emptyDir: {} } + containers: + - name: mgr-restore + image: quay.io/milaboratories/platforma:${PLATFORMA_VERSION} + command: ["/bin/sh", "-ec"] + args: + - | + if [ -f /data/database/.migration-complete ]; then echo Already completed; exit 0; fi + /app/platforma --restore-db=/data/database/backup.gz --db-dir=/data/database --force + echo Database restored + env: + - name: PL_LICENSE + valueFrom: + secretKeyRef: { name: platforma-license, key: MI_LICENSE } + volumeMounts: + - { name: db, mountPath: /data/database } + - { name: main, mountPath: /data/main } +``` + +```bash +envsubst < mgr-restore.yaml | kubectl apply -f - +kubectl wait pod/mgr-restore -n ${NAMESPACE} \ + --for=jsonpath='{.status.phase}'=Succeeded --timeout=1800s +kubectl logs mgr-restore -n ${NAMESPACE} +kubectl delete pod/mgr-restore -n ${NAMESPACE} +``` + +`--force` overwrites any partial DB from a previous attempt. Restore time scales with dump size — give a generous timeout (default 30 min above; bump to several hours for multi-hundred-GB dumps). + +### Step 1.6: Delete the Dump File + +```bash +kubectl run mgr-cleanup -n ${NAMESPACE} --restart=Never \ + --image=amazon/aws-cli:2.27.22 \ + --overrides='{"spec":{"securityContext":{"fsGroup":1010,"runAsUser":1010,"runAsGroup":1010},"containers":[{"name":"mgr-cleanup","image":"amazon/aws-cli:2.27.22","command":["sh","-ec","if [ -f /data/database/.migration-complete ]; then echo Already completed; exit 0; fi; rm -f /data/database/backup.gz"],"volumeMounts":[{"name":"db","mountPath":"/data/database"}]}],"volumes":[{"name":"db","persistentVolumeClaim":{"claimName":"platforma-database"}}]}}' +kubectl wait pod/mgr-cleanup -n ${NAMESPACE} \ + --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s +kubectl delete pod/mgr-cleanup -n ${NAMESPACE} +``` + +--- + +## Phase 2: Helm Install + +Standard install — see [`README.md`](README.md) and [`advanced-installation.md`](infrastructure/aws/advanced-installation.md) Step 10. The chart adopts the pre-existing PVC and StorageClass and starts the Platforma server. On first start the server applies any in-binary schema migrations against the restored DB. + +```bash +helm install platforma oci://ghcr.io/milaboratory/platforma-helm/platforma \ + --version ${CHART_VERSION} \ + -n ${NAMESPACE} \ + -f infrastructure/aws/values-aws-s3.yaml \ + --set storage.main.s3.bucket=${S3_BUCKET} \ + --set storage.main.s3.region=${REGION} \ + ... # other --set flags from advanced-installation.md Step 10 +``` + +Wait for the rollout to settle: + +```bash +kubectl wait --for=condition=Ready pod \ + -l app.kubernetes.io/name=platforma -n ${NAMESPACE} --timeout=300s +``` + +--- + +## Phase 3: Post-Helm (Cache Invalidation) + +After the chart's first reconciliation, internal caches in the restored DB still point at the **old** primary-storage URLs. Invalidating them rewrites those references to the new bucket. + +The Platforma binary that runs invalidation needs exclusive RWO access to the database PVC, so we briefly scale the server down to zero. + +### Step 3.1: Scale Platforma Down + +```bash +kubectl scale deployment/platforma -n ${NAMESPACE} --replicas=0 +kubectl wait --for=delete pod -l app.kubernetes.io/name=platforma \ + -n ${NAMESPACE} --timeout=120s +``` + +### Step 3.2: Run Invalidation + +```yaml +# mgr-invalidate.yaml +apiVersion: v1 +kind: Pod +metadata: + name: mgr-invalidate + namespace: platforma +spec: + securityContext: + fsGroup: 1010 + runAsUser: 1010 + runAsGroup: 1010 + restartPolicy: Never + volumes: + - { name: db, persistentVolumeClaim: { claimName: platforma-database } } + - { name: main, emptyDir: {} } + containers: + - name: mgr-invalidate + image: quay.io/milaboratories/platforma:${PLATFORMA_VERSION} + command: ["/bin/sh", "-ec"] + args: + - | + if [ -f /data/database/.migration-complete ]; then echo Already completed; exit 0; fi + /app/platforma --invalidate-caches --main-root=/data/main --db-dir=/data/database + date -u > /data/database/.migration-complete + echo Caches invalidated + env: + - name: PL_LICENSE + valueFrom: + secretKeyRef: { name: platforma-license, key: MI_LICENSE } + volumeMounts: + - { name: db, mountPath: /data/database } + - { name: main, mountPath: /data/main } +``` + +```bash +envsubst < mgr-invalidate.yaml | kubectl apply -f - +kubectl wait pod/mgr-invalidate -n ${NAMESPACE} \ + --for=jsonpath='{.status.phase}'=Succeeded --timeout=1800s +kubectl logs mgr-invalidate -n ${NAMESPACE} +kubectl delete pod/mgr-invalidate -n ${NAMESPACE} +``` + +The marker file `/data/database/.migration-complete` makes this step and every other PVC-mounting step (1.4 download, 1.5 restore, 1.6 cleanup) a no-op on re-run. Step 1.3 (`s5cmd`) and the Phase 0 AWS-side steps are not marker-gated — see the [intro](#migration-single-server-platforma--kubernetes) for how they handle re-runs. + +### Step 3.3: Scale Platforma Back Up + +```bash +kubectl scale deployment/platforma -n ${NAMESPACE} --replicas=1 +kubectl rollout status deployment/platforma -n ${NAMESPACE} --timeout=300s +``` + +--- + +## Verification + +```bash +kubectl get pvc -n ${NAMESPACE} # platforma-database Bound, > 0 used +kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=platforma | grep -i 'ready\|migration' +aws s3 ls s3://${S3_BUCKET}/ --summarize # object count matches source +``` + +Connect from the Desktop App; existing projects, results, and credentials should appear. + +--- + +## Cross-Account Sync + +If the source S3 bucket lives in a different AWS account, the destination bucket policy must grant the source identity write access. Pattern: create an IAM user (or role) in the **source** account, generate access keys, and add a bucket policy on the **destination** bucket like: + +```json +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": { "AWS": "arn:aws:iam:::user/migration-user" }, + "Action": ["s3:PutObject", "s3:ListBucket", "s3:AbortMultipartUpload"], + "Resource": [ + "arn:aws:s3:::${S3_BUCKET}", + "arn:aws:s3:::${S3_BUCKET}/*" + ] + }] +} +``` + +Then export the access keys as `MIGRATION_STORAGE_ACCESS_KEY` / `MIGRATION_STORAGE_SECRET_KEY` (Step 1.3). The same pattern applies to the database-dump bucket (`MIGRATION_DB_*`). + +--- + +## Rollback + +If the migration fails after `helm install`: + +1. `helm uninstall platforma -n ${NAMESPACE}` — removes the Deployment but **keeps PVCs by default**. Verify with `kubectl get pvc -n ${NAMESPACE}`. +2. `kubectl delete pvc platforma-database -n ${NAMESPACE}` — wipe the bad DB. +3. Repeat Phase 1 with the same dump. + +Apart from the debug-API restart in [Step 0.2](#step-02-enable-the-debug-api-on-the-source), the source single-server instance is untouched throughout — keep it running until you have validated the new cluster. + diff --git a/docs/provisioning-request-test-results.md b/docs/provisioning-request-test-results.md new file mode 100644 index 0000000..399187c --- /dev/null +++ b/docs/provisioning-request-test-results.md @@ -0,0 +1,83 @@ +# ProvisioningRequest Test Results + +Tested on `platforma-cluster-lab` (EKS, eu-west-1) on 2026-04-14 and 2026-04-16. + +## Setup + +- EKS 1.35, Kueue v0.16.1, AppWrapper v1.2.0 +- Cluster Autoscaler v1.35.0 (chart 9.56.0) with: + - `--enable-provisioning-requests=true` + - `--kube-api-content-type=application/json` + - Supplementary RBAC for `provisioningrequests`, `/status`, `podtemplates` +- ProvisioningRequest CRD installed from `cluster-autoscaler-1.35.0` tag +- Provisioning class: `best-effort-atomic-scale-up.autoscaling.x-k8s.io` +- AdmissionCheck on batch ClusterQueue only (not UI) +- Batch node groups: `m7i.4xlarge` (16 vCPU / 64 GiB), scale-from-zero + +## Test 1: Scale-From-Zero + +**Scenario:** Zero batch nodes, submit a 1 GiB job. + +**Result:** Kueue created ProvisioningRequest → CA responded `Accepted=True, Provisioned=True` → workload admitted → CA scaled up batch node → job ran and completed. + +**Verdict:** PASS. `best-effort-atomic-scale-up` handles scale-from-zero correctly. + +## Test 2: Resource Fragmentation + +**Scenario:** 4 batch nodes each running a 48 GiB filler (~10 GiB free per node, ~40 GiB aggregate free). Submit a 30 GiB victim job. + +**Without ProvisioningRequest:** Kueue admitted the job (30 GiB < 40 GiB aggregate). Pod got `FailedScheduling: 2 Insufficient memory`. CA triggered an unnecessary 3rd node scale-up. If the cluster had been at max nodes, the job would have timed out after AppWrapper's 30-minute `warmupGracePeriodDuration`. + +**With ProvisioningRequest:** CA responded `Provisioned=True (CapacityIsProvisioned)` — it provisioned a new node for the victim instead of admitting onto fragmented capacity. Job ran successfully on the new node. + +**Verdict:** PASS. Fragmentation is handled correctly — CA provisions a fresh node rather than letting Kueue admit a job that can't schedule. + +## Test 3: Pool at Capacity (Quota Exhausted) + +**Scenario:** Batch queue quota temporarily reduced to 4 CPU / 96 GiB. A filler job (3 CPU / 80 GiB) uses most of the quota. A victim job (2 CPU / 20 GiB) is submitted — exceeds remaining quota. AppWrapper `warmupGracePeriodDuration` set to 2 minutes to verify no premature timeout. + +| Time | Event | +|------|-------| +| T+0s | Filler admitted, running. Victim submitted. | +| T+7s | Victim stays `Suspended` — Kueue holds it (quota insufficient). | +| T+2m | Past AppWrapper's 2-minute timeout — victim still `Suspended`, no error. | +| T+8m | Victim still `Suspended` — no timeout triggered. | +| T+10m | Filler completes (`Succeeded`). Quota freed. | +| T+10m+9s | Kueue automatically admits victim → ProvisioningRequest `Provisioned=True`. | +| T+11m | Victim runs on existing batch node and completes (`Succeeded`). | + +**Verdict:** PASS. When Kueue holds a workload due to insufficient quota: +- AppWrapper stays `Suspended` indefinitely — the `warmupGracePeriodDuration` timer does NOT start until the workload is admitted +- Once quota frees up, Kueue automatically admits the waiting workload +- The entire queue → admit → run cycle is automatic with no manual intervention + +## Test 4: Provisioning Class Comparison + +| Class | Scale-from-zero | Steady-state fragmentation | Behavior | +|-------|----------------|---------------------------|----------| +| `best-effort-atomic-scale-up.autoscaling.x-k8s.io` | PASS — provisions node | PASS — provisions fresh node | Recommended | +| `check-capacity.autoscaling.x-k8s.io` | Responds `CapacityIsNotFound`, no scale-up — workload stays queued forever | Should work (checks per-node fit) | Not suitable for scale-from-zero | + +## Key Findings + +1. **AppWrapper timeout is safe.** The `warmupGracePeriodDuration` only starts counting after Kueue admits the workload and AppWrapper creates inner resources. While the workload is held in Kueue's queue (`Suspended`), no timer runs. + +2. **Kueue queue-to-admit is automatic.** When capacity becomes available (quota freed, nodes available), Kueue re-evaluates pending workloads and admits them without manual intervention. + +3. **ProvisioningRequest prevents fragmented admission.** Without it, Kueue admits based on aggregate capacity. With it, CA checks per-node fit and provisions a suitable node if needed. + +4. **Three CA configuration gotchas:** + - Class names must be FQDN (`best-effort-atomic-scale-up.autoscaling.x-k8s.io`, not `best-effort-atomic`) + - `--kube-api-content-type=application/json` required (CA bug [#8855](https://github.com/kubernetes/autoscaler/issues/8855)) + - Supplementary RBAC needed for `provisioningrequests`, `/status`, `podtemplates` + +## Infrastructure Requirements + +For ProvisioningRequest to work, the following must be in place: + +1. ProvisioningRequest CRD (installed before Kueue starts) +2. Cluster Autoscaler with `--enable-provisioning-requests=true` and `--kube-api-content-type=application/json` +3. CA RBAC for `provisioningrequests`, `provisioningrequests/status`, `podtemplates` +4. Kueue with `ProvisioningACC` feature gate (enabled by default in v0.10+) + +All handled automatically by the CloudFormation template. diff --git a/docs/upgrade-provisioning-request-gke.md b/docs/upgrade-provisioning-request-gke.md new file mode 100644 index 0000000..dffdfef --- /dev/null +++ b/docs/upgrade-provisioning-request-gke.md @@ -0,0 +1,145 @@ +# Upgrade Guide: ProvisioningRequest on GKE + +Starting with Platforma Helm chart v3.2.2, the chart creates ProvisioningRequest admission checks that prevent Kueue resource fragmentation. + +## GKE vs AWS + +GKE has **native ProvisioningRequest support** — the CRD is pre-installed and the built-in cluster autoscaler handles it automatically. No separate CRD install, no Cluster Autoscaler flags, no RBAC grants needed. + +| Component | AWS EKS | GKE | +|-----------|---------|-----| +| ProvisioningRequest CRD | Manual install required | Pre-installed | +| Cluster Autoscaler flags | `--enable-provisioning-requests`, `--kube-api-content-type` | Not needed (built-in) | +| CA RBAC | Manual ClusterRole + Binding | Not needed | +| Kueue `ProvisioningACC` | Must enable | Must enable | +| Kueue memory bump | Recommended (512Mi → 1Gi) | Recommended | +| Chart `provisioningRequest.enabled` | Must set | Must set | + +## Prerequisites + +Verify the CRD exists (should be pre-installed on GKE 1.29+): + +```bash +kubectl get crd provisioningrequests.autoscaling.x-k8s.io +``` + +If missing, your GKE version may not support it. Check your cluster version: + +```bash +kubectl version --short +``` + +ProvisioningRequest requires GKE 1.29 or later. + +## Step 1: Upgrade Kueue + +Add `ProvisioningACC` feature gate and bump memory limits: + +```bash +helm upgrade kueue oci://registry.k8s.io/kueue/charts/kueue \ + -n --reuse-values \ + --set controllerManager.manager.resources.limits.memory=1Gi \ + --set controllerManager.manager.resources.requests.memory=512Mi \ + --set featureGates.ProvisioningACC=true +``` + +Verify the provisioning controller started: + +```bash +kubectl logs -n -l control-plane=controller-manager --tail=20 | grep -i provision +# Should see: "Caches populated" for ProvisioningRequestConfig and ProvisioningRequest +``` + +If you see `Skipping provisioning controller setup`, the CRD wasn't found at Kueue startup. Restart Kueue: + +```bash +kubectl rollout restart deployment kueue-controller-manager -n +``` + +## Step 2: Upgrade Platforma Chart + +Add `provisioningRequest.enabled: true` to your values file: + +```yaml +kueue: + provisioningRequest: + enabled: true +``` + +Then upgrade: + +```bash +helm upgrade platforma oci://ghcr.io/milaboratory/platforma-helm/platforma \ + --version 3.2.2 \ + -n \ + -f values.yaml \ + --atomic --timeout 15m +``` + +Or with `--reuse-values`: + +```bash +helm upgrade platforma oci://ghcr.io/milaboratory/platforma-helm/platforma \ + --version 3.2.2 \ + -n --reuse-values \ + --set kueue.provisioningRequest.enabled=true +``` + +## Verification + +```bash +# 1. AdmissionCheck is Active +kubectl get admissionchecks +# Should show: platforma-provreq (or -platforma-provreq) + +# 2. ProvisioningRequestConfig exists +kubectl get provisioningrequestconfig +# Should show: platforma-provreq-config + +# 3. Batch ClusterQueue references the AdmissionCheck +kubectl get clusterqueue -platforma-batch -o yaml | grep -A3 admissionChecksStrategy +# Should show: platforma-provreq + +# 4. Kueue provisioning controller is active +kubectl logs -n -l control-plane=controller-manager --tail=20 | grep provision +``` + +## Clusters Without Node Autoscaling + +If your GKE cluster has fixed-size node pools (no autoscaling), disable ProvisioningRequest: + +```yaml +kueue: + provisioningRequest: + enabled: false +``` + +Without an autoscaler responding to ProvisioningRequests, the AdmissionCheck stays Inactive and blocks all batch job admissions. + +## Troubleshooting + +### AdmissionCheck Inactive + +``` +Can't admit new workloads: references inactive AdmissionCheck(s): platforma-provreq +``` + +**Cause:** Kueue's provisioning controller didn't start — CRD was missing when Kueue first started. + +**Fix:** Verify the CRD exists, then restart Kueue: + +```bash +kubectl get crd provisioningrequests.autoscaling.x-k8s.io +kubectl rollout restart deployment kueue-controller-manager -n +``` + +### Kueue OOMKilled + +**Cause:** Too many completed AppWrappers accumulating. + +**Fix:** Increase Kueue memory limit to at least 1Gi and clean up stale AppWrappers: + +```bash +kubectl get appwrappers -n --no-headers -o name | \ + xargs -n 100 kubectl delete -n --wait=false +``` diff --git a/docs/upgrade-provisioning-request.md b/docs/upgrade-provisioning-request.md new file mode 100644 index 0000000..087a0a1 --- /dev/null +++ b/docs/upgrade-provisioning-request.md @@ -0,0 +1,218 @@ +# Upgrade Guide: ProvisioningRequest Integration + +Starting with Platforma Helm chart v3.2.2, the chart creates ProvisioningRequest admission checks that prevent Kueue resource fragmentation. This requires additional infrastructure components. + +## What Changed + +Kueue admits workloads based on aggregate cluster capacity, but Kubernetes scheduling requires per-node fit. Without ProvisioningRequest, a 30 GiB job gets admitted when 3 nodes each have 10 GiB free (30 GiB aggregate), but no single node can run it — causing AppWrapper timeouts or unnecessary node scale-ups. + +The chart now creates: +- `ProvisioningRequestConfig` with class `best-effort-atomic-scale-up.autoscaling.x-k8s.io` +- `AdmissionCheck` referencing the config +- `admissionChecksStrategy` on the batch `ClusterQueue` (not UI — UI tasks are small) + +## CloudFormation Deployments + +Update the CF template and run a stack update. The template mirrors all infra component version strings as properties on `TriggerHelmDeploy` — any version change automatically triggers the HelmDeployer CodeBuild, which installs all prerequisites (CRD, CA flags, RBAC, Kueue memory) before the Platforma chart upgrade. + +> **Note:** For non-version buildspec changes (new Helm flags, RBAC additions, install order), the template uses a `BuildSpecRevision` property that must be bumped manually. See the README for details. + +For GKE deployments, see [upgrade-provisioning-request-gke.md](upgrade-provisioning-request-gke.md). + +The rest of this guide is for reference and for manual (non-CF) AWS EKS deployments. + +## Manual Deployments + +The following must be in place **before** upgrading the Platforma chart. The order matters. + +### 1. Install ProvisioningRequest CRD + +The CRD is not bundled with any Helm chart. Install it matching your Cluster Autoscaler version: + +```bash +CA_VERSION="1.35.0" # Must match your Cluster Autoscaler image tag +kubectl apply --server-side -f \ + https://raw.githubusercontent.com/kubernetes/autoscaler/cluster-autoscaler-${CA_VERSION}/cluster-autoscaler/apis/config/crd/autoscaling.x-k8s.io_provisioningrequests.yaml +``` + +Kueue checks for this CRD at startup. If it was installed after Kueue, restart Kueue: + +```bash +kubectl rollout restart deployment kueue-controller-manager -n kueue-system +``` + +Verify the ProvisioningRequest controller started: + +```bash +kubectl logs -n kueue-system -l control-plane=controller-manager | grep -i 'provisioning' +# Should see: "Successful initial Provisioning Request sync" +# Should NOT see: "Skipping provisioning controller setup" +``` + +### 2. Upgrade Cluster Autoscaler + +Two flags are required: + +```bash +helm upgrade cluster-autoscaler autoscaler/cluster-autoscaler \ + -n --reuse-values \ + --set extraArgs.enable-provisioning-requests=true \ + --set extraArgs.kube-api-content-type=application/json +``` + +| Flag | Purpose | +|------|---------| +| `--enable-provisioning-requests` | Enables the ProvisioningRequest processor in CA | +| `--kube-api-content-type=application/json` | Workaround for CA [bug #8855](https://github.com/kubernetes/autoscaler/issues/8855) — CA defaults to protobuf encoding which fails for CRD status updates | + +**Important:** Always pin the chart version with `--version` when using `--reuse-values`. Without it, Helm pulls the latest chart which may be incompatible with your CA image. + +### 3. Add Cluster Autoscaler RBAC + +The CA Helm chart's built-in ClusterRole does not include ProvisioningRequest or PodTemplate permissions: + +```bash +kubectl apply -f - <<'EOF' +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: cluster-autoscaler-provisioning-requests +rules: + - apiGroups: ["autoscaling.x-k8s.io"] + resources: ["provisioningrequests", "provisioningrequests/status"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] + - apiGroups: [""] + resources: ["podtemplates"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: cluster-autoscaler-provisioning-requests +subjects: + - kind: ServiceAccount + name: cluster-autoscaler + namespace: # e.g., platforma or kube-system +roleRef: + kind: ClusterRole + name: cluster-autoscaler-provisioning-requests + apiGroup: rbac.authorization.k8s.io +EOF +``` + +### 4. Upgrade Platforma Chart + +```bash +helm upgrade platforma -n --reuse-values --version +``` + +The chart creates the ProvisioningRequestConfig, AdmissionCheck, and patches the batch ClusterQueue automatically. + +## Verification + +After upgrade, verify the full chain: + +```bash +# 1. CRD exists +kubectl get crd provisioningrequests.autoscaling.x-k8s.io + +# 2. Kueue ProvisioningRequest controller is active +kubectl logs -n kueue-system -l control-plane=controller-manager --tail=50 | grep -i provision +# Look for: "Successful initial Provisioning Request sync" + +# 3. CA can process ProvisioningRequests +kubectl logs -n -l app.kubernetes.io/name=aws-cluster-autoscaler --tail=50 | grep -i provision +# Look for: "Successful initial Provisioning Request sync" +# Should NOT see: "protobuf marshalling" errors or "forbidden" errors + +# 4. AdmissionCheck is active +kubectl get admissionchecks +# Should show platforma-provreq (or -platforma-provreq) + +# 5. Batch ClusterQueue references the AdmissionCheck and is Active +kubectl get clusterqueue -platforma-batch -o yaml | grep -A5 'admissionChecksStrategy\|conditions' +# Should show: admissionChecks with platforma-provreq, condition Ready=True + +# 6. UI ClusterQueue has NO admissionCheck (fast path) +kubectl get clusterqueue -platforma-ui -o yaml | grep admissionChecksStrategy +# Should return empty +``` + +## Clusters Without Cluster Autoscaler + +For bare-metal, Hetzner, or other clusters without a cloud autoscaler, disable ProvisioningRequest: + +```yaml +# values.yaml +kueue: + provisioningRequest: + enabled: false +``` + +Without this, the AdmissionCheck stays Inactive and blocks all batch job admissions. + +## Troubleshooting + +### AdmissionCheck Inactive + +``` +Can't admit new workloads: references inactive AdmissionCheck(s): platforma-provreq +``` + +**Cause:** Kueue's ProvisioningRequest controller didn't start. + +**Fix:** Check if the CRD exists (`kubectl get crd provisioningrequests.autoscaling.x-k8s.io`). If missing, install it and restart Kueue. + +### CA Protobuf Error + +``` +object *v1.ProvisioningRequest does not implement the protobuf marshalling interface +``` + +**Cause:** Missing `--kube-api-content-type=application/json` flag on CA. + +**Fix:** Add the flag and restart CA. + +### CA RBAC Errors + +``` +provisioningrequests.autoscaling.x-k8s.io is forbidden +podtemplates is forbidden +``` + +**Cause:** CA ClusterRole missing ProvisioningRequest/PodTemplate permissions. + +**Fix:** Apply the supplementary RBAC (Step 3 above). + +### ProvisioningRequest Stays Without Conditions + +CA syncs ProvisioningRequests but never sets `Accepted`/`Provisioned` conditions. + +**Cause:** Wrong provisioning class name. CA expects FQDN names like `best-effort-atomic-scale-up.autoscaling.x-k8s.io`. Short names like `check-capacity` are silently ignored. + +**Fix:** Verify the ProvisioningRequestConfig has the FQDN class name. + +### Jobs Stuck at Scale-From-Zero With `check-capacity` + +``` +Provisioned: False, reason: CapacityIsNotFound +``` + +**Cause:** `check-capacity.autoscaling.x-k8s.io` only checks existing nodes. At scale-from-zero there's nothing to check. + +**Fix:** Switch to `best-effort-atomic-scale-up.autoscaling.x-k8s.io` (the default). + +### Kueue OOMKilled + +``` +Last State: Terminated, Reason: OOMKilled +``` + +**Cause:** Too many completed AppWrappers accumulating. Kueue watches all of them. + +**Fix:** Increase Kueue memory limit to at least 1Gi. Clean up stale AppWrappers: + +```bash +kubectl get appwrappers -n --no-headers -o name | \ + xargs -n 100 kubectl delete -n --wait=false +``` diff --git a/infrastructure/aws/README.md b/infrastructure/aws/README.md index b4f1508..767be6e 100644 --- a/infrastructure/aws/README.md +++ b/infrastructure/aws/README.md @@ -268,6 +268,26 @@ Check the CodeBuild project logs — links are in the Outputs tab (`HelmDeployer - **vCPU quota exceeded** — the stack checks your AWS On-Demand vCPU quota before deploying. If it's too low, request an increase at [Service Quotas console](https://console.aws.amazon.com/servicequotas/home/services/ec2/quotas/L-1216C47A) +### How Infrastructure Re-Deployment Works + +The template uses two CodeBuild projects: + +- **HelmDeployer** — installs/upgrades infra controllers (Kueue, Cluster Autoscaler, External DNS, ALB Controller) +- **PlatformaDeployer** — installs/upgrades the Platforma Helm chart + +Each is triggered by a CloudFormation custom resource that only re-runs when its **properties change**. The `TriggerHelmDeploy` resource mirrors all infra component version strings as properties: + +| Property | Source | +|----------|--------| +| `KueueVersion` | `KUEUE_VERSION` env var in HelmDeployerProject | +| `AppWrapperVersion` | `APPWRAPPER_VERSION` | +| `ClusterAutoscalerChartVersion` | `CLUSTER_AUTOSCALER_CHART_VERSION` | +| `ExternalDnsVersion` | `EXTERNAL_DNS_VERSION` | +| `AlbControllerVersion` | `ALB_CONTROLLER_VERSION` | +| `BuildSpecRevision` | Bump for non-version changes (new flags, RBAC, install order) | + +When you update an infra component version in the CodeBuild env vars, **also update the matching property on `TriggerHelmDeploy`**. Otherwise CloudFormation won't detect the change and the HelmDeployer won't run. + --- ## Cleanup diff --git a/infrastructure/aws/advanced-installation.md b/infrastructure/aws/advanced-installation.md index b5c33c3..14b2beb 100644 --- a/infrastructure/aws/advanced-installation.md +++ b/infrastructure/aws/advanced-installation.md @@ -284,10 +284,45 @@ helm install cluster-autoscaler autoscaler/cluster-autoscaler \ --set extraArgs.max-node-provision-time=5m \ --set extraArgs.initial-node-group-backoff-duration=1m \ --set extraArgs.max-node-group-backoff-duration=5m \ + --set extraArgs.enable-provisioning-requests=true \ + --set extraArgs.kube-api-content-type=application/json \ --atomic --timeout 5m ``` -### Configuration options +### Grant ProvisioningRequest RBAC + +The Helm chart's built-in ClusterRole does not include ProvisioningRequest or PodTemplate permissions. Add them: + +```bash +kubectl apply -f - < **ProvisioningRequest**: The chart automatically creates ProvisioningRequest admission checks that prevent resource fragmentation — Kueue asks the Cluster Autoscaler whether pods can actually be scheduled before admitting them. This requires the ProvisioningRequest CRD (Step 8), `--enable-provisioning-requests=true` and `--kube-api-content-type=application/json` on the Cluster Autoscaler (Step 3), and the ProvisioningRequest RBAC (Step 3). + Verify: ```bash diff --git a/infrastructure/aws/cloudformation-eks-1-35.yaml b/infrastructure/aws/cloudformation-eks-1-35.yaml index 781b271..a81d202 100644 --- a/infrastructure/aws/cloudformation-eks-1-35.yaml +++ b/infrastructure/aws/cloudformation-eks-1-35.yaml @@ -52,10 +52,27 @@ Metadata: Parameters: - DeploymentSize - SystemNodeAZ + - Label: + default: 'Container registry' + Parameters: + - EnableECRCache + - UseECRCacheForPlatforma - Label: default: Storage Parameters: - S3BucketName + - Label: + default: 'Migration (optional, one-time restore from existing server)' + Parameters: + - MigrationDatabaseS3Uri + - MigrationDbAccessKey + - MigrationDbSecretKey + - MigrationSourceBucket + - MigrationSourcePrefix + - MigrationSourceRegion + - MigrationStorageAccessKey + - MigrationStorageSecretKey + - MigrationStorageUserArn - Label: default: 'Data libraries (optional, up to 3 external read-only S3 buckets)' Parameters: @@ -100,6 +117,10 @@ Metadata: default: 'Platforma chart version (empty = use built-in default)' PlatformaImage: default: Custom container image (leave empty for chart default) + EnableECRCache: + default: 'ECR pull-through cache for container images' + UseECRCacheForPlatforma: + default: 'Route Platforma image pulls through the ECR cache' S3BucketName: default: S3 bucket name (auto-generate if empty) DomainName: @@ -108,6 +129,24 @@ Metadata: default: 'Route53 hosted zone ID (e.g. Z0123456789ABCDEF)' AuthMethod: default: 'Authentication method (htpasswd or ldap)' + MigrationDatabaseS3Uri: + default: 'Database dump S3 URI (e.g. s3://bucket/backup.gz)' + MigrationSourceBucket: + default: 'Source primary storage S3 bucket name' + MigrationSourcePrefix: + default: 'Source bucket prefix (leave empty for root)' + MigrationSourceRegion: + default: 'Source bucket region (leave empty = cluster region)' + MigrationDbAccessKey: + default: 'AWS access key for database dump bucket' + MigrationDbSecretKey: + default: 'AWS secret key for database dump bucket' + MigrationStorageAccessKey: + default: 'AWS access key for source primary storage bucket' + MigrationStorageSecretKey: + default: 'AWS secret key for source primary storage bucket' + MigrationStorageUserArn: + default: 'IAM ARN for cross-account write access to destination bucket' EnableDemoLibrary: default: 'Enable MiLaboratories demo data library' DataLibrary1Name: @@ -164,8 +203,13 @@ Parameters: ClusterName: Type: String Default: platforma-cluster - AllowedPattern: '^[a-zA-Z0-9][a-zA-Z0-9_-]{0,99}$' - ConstraintDescription: 'Alphanumeric, hyphens, underscores, 1-100 characters' + # Upper bound of 25 characters keeps the derived ECR pull-through cache + # prefix (`quay-${ClusterName}`) under AWS's 30-char limit. Lowercase + # alphanumeric and hyphens only — the auto-generated S3 bucket name + # (`platforma-${ClusterName}-...`) cannot contain underscores or + # uppercase, and S3 is the tightest naming constraint downstream. + AllowedPattern: '^[a-z0-9][a-z0-9-]{0,24}$' + ConstraintDescription: 'Lowercase alphanumeric, hyphens, 1-25 characters' Description: EKS cluster name PlatformaNamespace: @@ -276,6 +320,79 @@ Parameters: NoEcho: true Description: Password for the LDAP search user. Required when LdapSearchUser is set. + # --- Migration (from existing server) --- + MigrationDatabaseS3Uri: + Type: String + Default: '' + Description: > + S3 URI to the database dump file (e.g. s3://my-bucket/backup.gz). + Created on the source server: curl -s http://localhost:9091/db/state_raw | gzip > backup.gz + Leave empty to skip migration (fresh install). + + MigrationSourceBucket: + Type: String + Default: '' + AllowedPattern: '^([a-z0-9]([a-z0-9.-]{1,61}[a-z0-9])?)?$' + ConstraintDescription: 'Lowercase letters, digits, hyphens, periods, 3-63 chars, or empty' + Description: > + Source S3 bucket for primary storage migration. Data will be synced + to the new bucket created by this stack. Leave empty to skip storage sync. + + MigrationSourcePrefix: + Type: String + Default: '' + Description: Prefix within the source bucket. Leave empty for the bucket root. + + MigrationSourceRegion: + Type: String + Default: '' + AllowedPattern: '^([a-z]{2}(-[a-z]+)+-\d{1,2})?$' + ConstraintDescription: 'Must be a valid AWS region or empty' + Description: > + Region of the source S3 bucket. Defaults to cluster region if empty. + + MigrationDbAccessKey: + Type: String + Default: '' + AllowedPattern: '^((?:AKIA|ASIA)[A-Z0-9]{16})?$' + ConstraintDescription: 'Must be a valid AWS access key ID or empty' + Description: > + AWS access key for the database dump S3 bucket. + Required if the bucket is in a different AWS account. + + MigrationDbSecretKey: + Type: String + Default: '' + NoEcho: true + Description: > + AWS secret key for the database dump S3 bucket. + + MigrationStorageAccessKey: + Type: String + Default: '' + AllowedPattern: '^((?:AKIA|ASIA)[A-Z0-9]{16})?$' + ConstraintDescription: 'Must be a valid AWS access key ID or empty' + Description: > + AWS access key for the source primary storage S3 bucket. + Required if the bucket is in a different AWS account. + + MigrationStorageSecretKey: + Type: String + Default: '' + NoEcho: true + Description: > + AWS secret key for the source primary storage S3 bucket. + + MigrationStorageUserArn: + Type: String + Default: '' + AllowedPattern: '^(arn:aws:iam::\d{12}:(user|role)/[a-zA-Z0-9_./@-]+)?$' + ConstraintDescription: 'Must be a valid IAM user or role ARN, or empty' + Description: > + IAM ARN of the user/role that will sync data to the new S3 bucket. + Required for cross-account migration — grants write access on the + destination bucket. Example: arn:aws:iam::123456789012:user/migration-user + # --- Data libraries --- EnableDemoLibrary: Type: String @@ -445,7 +562,7 @@ Parameters: AllowedValues: [small, medium, large, xlarge] Description: > Cluster sizing profile. Controls node group scaling limits and Kueue quotas. - All deployment sizes support the same maximum single-job size (62 vCPU / 500Gi). + All deployment sizes support the same maximum single-job size (62 vCPU / 484Gi). Larger sizes allow more jobs to run simultaneously. small: ~4 large jobs, or 16 small — for small teams or testing (~400 vCPU quota). medium: ~8 large jobs, or 32 small — for mid-sized research groups (~700 vCPU quota). @@ -464,6 +581,29 @@ Parameters: your platforma-database PV resides (e.g. if volumes are in eu-west-1b, choose "b"). + # --- Container registry --- + EnableECRCache: + Type: String + Default: 'false' + AllowedValues: ['true', 'false'] + Description: > + Create an ECR pull-through cache for container images. When enabled, + Platforma pulls block images from a same-region ECR cache instead of + the default registry (containers.pl-open.science). ECR fetches from + Quay.io on first pull and caches locally. Disable for restricted + environments that cannot reach Quay.io. + + UseECRCacheForPlatforma: + Type: String + Default: 'true' + AllowedValues: ['true', 'false'] + Description: > + Route Platforma workload image pulls through the ECR pull-through cache. + Only takes effect when EnableECRCache=true; ignored otherwise. Set to + false to keep the cache resources in place while having Platforma pull + directly from upstream — useful for debugging cache issues or rolling + client behaviour back without tearing down the cache state. + # --- Storage --- S3BucketName: Type: String @@ -512,17 +652,22 @@ Mappings: EKS: Version: '1.35' Platforma: - Version: '3.0.2' + Version: '3.2.2' # Kueue ClusterQueue quotas and node group MaxSize per deployment size. # Quotas are derived from pool sizes (worst case: all jobs are max size). - # CPU: (64-vCPU node count) x 63 allocatable. Memory: (r7i.16xl count) x 500Gi. + # CPU: (64-vCPU node count) x 63 allocatable. Memory: (r7i.16xl count) x 484Gi. + # 484Gi = r7i.16xlarge EKS allocatable (~486 GiB) minus ~1 GiB EKS DaemonSet + # overhead (aws-node, kube-proxy, ebs-csi) minus 1 GiB safety margin for + # customer-installed DaemonSets (Datadog, Falco, Wiz, etc.). Matches the GCP + # per-job ceiling on n2d-highmem-64 so the same deployment_size label means + # the same workload capacity on both clouds. # UI quota is fixed at 64 vCPU / 256 GiB regardless of size. DeploymentSize: small: - # Kueue quotas (derived: 2x63=126 CPU, 1x500=500Gi) + # Kueue quotas (derived: 2x63=126 CPU, 1x484=484Gi) BatchCpu: '126' - BatchMemoryGi: '500' + BatchMemoryGi: '484' UiCpu: '64' UiMemoryGi: '256' # Node group MaxSize per pool @@ -533,9 +678,9 @@ Mappings: MaxBatch64c512g: '1' MaxUi: '4' medium: - # Kueue quotas (derived: 4x63=252 CPU, 2x500=1000Gi) + # Kueue quotas (derived: 4x63=252 CPU, 2x484=968Gi) BatchCpu: '252' - BatchMemoryGi: '1000' + BatchMemoryGi: '968' UiCpu: '64' UiMemoryGi: '256' MaxBatch16c64g: '8' @@ -545,9 +690,9 @@ Mappings: MaxBatch64c512g: '2' MaxUi: '8' large: - # Kueue quotas (derived: 8x63=504 CPU, 4x500=2000Gi) + # Kueue quotas (derived: 8x63=504 CPU, 4x484=1936Gi) BatchCpu: '504' - BatchMemoryGi: '2000' + BatchMemoryGi: '1936' UiCpu: '64' UiMemoryGi: '256' MaxBatch16c64g: '16' @@ -557,9 +702,9 @@ Mappings: MaxBatch64c512g: '4' MaxUi: '16' xlarge: - # Kueue quotas (derived: 16x63=1008 CPU, 8x500=4000Gi) + # Kueue quotas (derived: 16x63=1008 CPU, 8x484=3872Gi) BatchCpu: '1008' - BatchMemoryGi: '4000' + BatchMemoryGi: '3872' UiCpu: '64' UiMemoryGi: '256' MaxBatch16c64g: '32' @@ -573,6 +718,7 @@ Mappings: # Conditions # ============================================================================= Conditions: + ECRCacheEnabled: !Equals [!Ref EnableECRCache, 'true'] CreateNewVpc: !Equals [!Ref VpcId, ''] AutoGenerateS3Name: !Equals [!Ref S3BucketName, ''] HasPublicSubnets: !Not [!Equals [!Select [0, !Ref PublicSubnetIds], '']] @@ -588,6 +734,7 @@ Conditions: HasLibrary3Irsa: !And - !Not [!Equals [!Ref DataLibrary3Bucket, '']] - !Equals [!Ref DataLibrary3AccessKey, ''] + HasMigrationStorageUserArn: !Not [!Equals [!Ref MigrationStorageUserArn, '']] UseDefaultPlatformaVersion: !Equals [!Ref PlatformaVersion, ''] IsHtpasswd: !Equals [!Ref AuthMethod, 'htpasswd'] SystemAZa: !Equals [!Ref SystemNodeAZ, 'a'] @@ -996,6 +1143,18 @@ Resources: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly + Policies: !If + - ECRCacheEnabled + - - PolicyName: ecr-pull-through-cache + PolicyDocument: + Version: '2012-10-17' + Statement: + - Effect: Allow + Action: + - ecr:BatchImportUpstreamImage + - ecr:CreateRepository + Resource: !Sub 'arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/quay/*' + - !Ref 'AWS::NoValue' Tags: - Key: platforma Value: !Ref ClusterName @@ -2051,6 +2210,10 @@ Resources: Value: !Ref HostedZoneId - Name: KUBERNETES_VERSION Value: !FindInMap [Constants, EKS, Version] + # ----------------------------------------------------------------- + # INFRA VERSIONS — duplicated in TriggerHelmDeploy properties. + # When changing any version below, also update TriggerHelmDeploy. + # ----------------------------------------------------------------- - Name: KUEUE_VERSION Value: '0.16.1' - Name: APPWRAPPER_VERSION @@ -2112,14 +2275,16 @@ Resources: - | [ "${TEARDOWN:-false}" = "true" ] && exit 0 echo "=== [2/5] KUEUE + APPWRAPPER ===" + kubectl apply --server-side -f https://raw.githubusercontent.com/kubernetes/autoscaler/cluster-autoscaler-${KUBERNETES_VERSION}.0/cluster-autoscaler/apis/config/crd/autoscaling.x-k8s.io_provisioningrequests.yaml helm upgrade --install kueue oci://registry.k8s.io/kueue/charts/kueue \ --version $KUEUE_VERSION \ -n kueue-system --create-namespace \ --set controllerManager.manager.resources.limits.cpu=500m \ - --set controllerManager.manager.resources.limits.memory=512Mi \ + --set controllerManager.manager.resources.limits.memory=1Gi \ --set controllerManager.manager.resources.requests.cpu=100m \ - --set controllerManager.manager.resources.requests.memory=256Mi \ + --set controllerManager.manager.resources.requests.memory=512Mi \ --set featureGates.AppWrapper=true \ + --set featureGates.ProvisioningACC=true \ --set 'integrations.frameworks[0]=batch/job' \ --set 'integrations.frameworks[1]=jobset.x-k8s.io/jobset' \ --set 'integrations.frameworks[2]=workload.codeflare.dev/appwrapper' \ @@ -2129,6 +2294,11 @@ Resources: --set 'integrations.podOptions.namespaceSelector.matchExpressions[0].values[1]=kueue-system' \ --set metrics.enableClusterQueueResources=true \ --atomic --timeout 5m + # Restart Kueue if the ProvisioningRequest CRD was just installed — + # Kueue only discovers CRDs at startup and skips the provisioning + # controller if the CRD wasn't present when it first started. + kubectl rollout restart deployment kueue-controller-manager -n kueue-system + kubectl wait --for=condition=Available deployment/kueue-controller-manager -n kueue-system --timeout=120s kubectl apply --server-side -f https://github.com/project-codeflare/appwrapper/releases/download/$APPWRAPPER_VERSION/install.yaml kubectl wait --for=condition=Available deployment/appwrapper-controller-manager -n appwrapper-system --timeout=120s kubectl delete validatingwebhookconfiguration appwrapper-validating-webhook-configuration --ignore-not-found @@ -2157,8 +2327,51 @@ Resources: --set extraArgs.max-node-provision-time=5m \ --set extraArgs.initial-node-group-backoff-duration=1m \ --set extraArgs.max-node-group-backoff-duration=5m \ + --set extraArgs.enable-provisioning-requests=true \ + --set extraArgs.kube-api-content-type=application/json \ --atomic --timeout 5m + # Grant CA access to ProvisioningRequest and PodTemplate resources. + # roleRef is immutable on a ClusterRoleBinding: if an earlier install + # (e.g. manual "provisioning-request-admin" role from pre-3.2.x testing) + # bound this name to a different ClusterRole, `kubectl apply` fails with + # "cannot change roleRef". Recreate the binding when roleRef drifts, and + # drop the legacy orphan ClusterRole. + EXPECTED_ROLE="cluster-autoscaler-provisioning-requests" + CURRENT_ROLE=$(kubectl get clusterrolebinding cluster-autoscaler-provisioning-requests \ + -o jsonpath='{.roleRef.name}' 2>/dev/null || true) + if [ -n "$CURRENT_ROLE" ] && [ "$CURRENT_ROLE" != "$EXPECTED_ROLE" ]; then + echo "Recreating stale ClusterRoleBinding (was bound to $CURRENT_ROLE)" + kubectl delete clusterrolebinding cluster-autoscaler-provisioning-requests + fi + kubectl delete clusterrole provisioning-request-admin --ignore-not-found + kubectl apply -f - < /tmp/platforma-values.yaml << PLATFORMA_VALUES environment: aws storage: @@ -2612,10 +2858,12 @@ Resources: alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/certificate-arn: $CERTIFICATE_ARN alb.ingress.kubernetes.io/backend-protocol-version: GRPC + alb.ingress.kubernetes.io/healthcheck-path: /grpc.health.v1.Health/Check + alb.ingress.kubernetes.io/success-codes: '0' kueue: maxJobResources: cpu: 62 - memory: 500Gi + memory: 484Gi mode: dedicated pools: ui: @@ -2632,6 +2880,8 @@ Resources: - key: dedicated value: batch effect: NoSchedule + provisioningRequest: + enabled: true dedicated: resources: ui: @@ -2647,9 +2897,10 @@ Resources: memory: 16Gi limits: cpu: 8 - memory: 16Gi + memory: 32Gi nodeSelector: node.kubernetes.io/pool: system + extraArgs: ${APP_EXTRA_ARGS:-[]} dataSources: ${DATASOURCES_YAML:-[]} PLATFORMA_VALUES @@ -2694,6 +2945,12 @@ Resources: fi fi + # --- Data migration (DISABLED — re-enable when ready) --- + if [ -n "$MIGRATION_DATABASE_S3_URI" ]; then + echo "INFO: Migration is disabled in this template version. Migration parameters are ignored." + echo "INFO: To run migration, use the migration-enabled template version." + fi + if [ -n "$PLATFORMA_IMAGE" ]; then IMG_REPO=$(echo "$PLATFORMA_IMAGE" | sed 's/:.*//') IMG_TAG=$(echo "$PLATFORMA_IMAGE" | sed 's/.*://') @@ -2709,6 +2966,8 @@ Resources: -n $NAMESPACE \ -f /tmp/platforma-values.yaml \ --atomic --timeout 15m + + # --- Data migration post-helm (DISABLED) --- post_build: commands: - | @@ -2830,11 +3089,19 @@ Resources: Properties: ServiceToken: !GetAtt TriggerHelmDeployFunction.Arn CodeBuildProject: !Sub '${ClusterName}-helm-deployer' - # Properties below trigger re-deployment when changed during stack update. - # The Lambda reads CodeBuildProject; the rest exist only so CloudFormation - # detects property changes and sends an UPDATE event. + # --- Trigger properties --- + # CF re-invokes this resource whenever ANY property value changes. + # Version properties MUST match HelmDeployerProject env vars above. + # This duplication is intentional — CF needs property changes to + # detect when re-deployment is required. KubernetesVersion: !FindInMap [Constants, EKS, Version] - ForceUpdateInfra: "1" # Bump to force infra re-deployment + KueueVersion: '0.16.1' + AppWrapperVersion: 'v1.2.0' + ClusterAutoscalerChartVersion: '9.56.0' + ExternalDnsVersion: '1.20.0' + AlbControllerVersion: '3.0.0' + # Bump for non-version buildspec changes (new flags, RBAC, install order) + BuildSpecRevision: '3' TriggerPlatformaDeploy: Type: AWS::CloudFormation::CustomResource @@ -2849,13 +3116,15 @@ Resources: ServiceToken: !GetAtt TriggerHelmDeployFunction.Arn CodeBuildProject: !Sub '${ClusterName}-platforma-deployer' # Properties below trigger re-deployment when changed during stack update. - ForceUpdatePlatforma: "1" # Bump to force Platforma re-deployment + ForceUpdatePlatforma: "2" # Bump to force Platforma re-deployment PlatformaVersion: !If - UseDefaultPlatformaVersion - !FindInMap [Constants, Platforma, Version] - !Ref PlatformaVersion PlatformaImage: !Ref PlatformaImage DeployPlatforma: !Ref DeployPlatforma + EnableECRCache: !Ref EnableECRCache + UseECRCacheForPlatforma: !Ref UseECRCacheForPlatforma LicenseKey: !Ref LicenseKey AuthMethod: !Ref AuthMethod HtpasswdContent: !Ref HtpasswdContent @@ -2943,6 +3212,49 @@ Resources: - !Ref EfsSecurityGroup # --------------------------------------------------------------------------- + # --------------------------------------------------------------------------- + # ECR Pull-Through Cache (optional) + # --------------------------------------------------------------------------- + + ECRPullThroughCacheRule: + Type: AWS::ECR::PullThroughCacheRule + Condition: ECRCacheEnabled + Properties: + EcrRepositoryPrefix: quay + UpstreamRegistryUrl: quay.io + + ECRCacheRepository: + Type: AWS::ECR::Repository + Condition: ECRCacheEnabled + DependsOn: ECRPullThroughCacheRule + Properties: + RepositoryName: quay/milaboratories/pl-containers + # Allow CF to delete the repo when EnableECRCache flips to false even + # if it still holds cached images — the whole point of the teardown is + # to reclaim the storage, so failing the update on "repo not empty" + # defeats the purpose. + EmptyOnDelete: true + LifecyclePolicy: + LifecyclePolicyText: | + { + "rules": [ + { + "rulePriority": 1, + "description": "Expire images older than 90 days", + "selection": { + "tagStatus": "any", + "countType": "sinceImagePushed", + "countUnit": "days", + "countNumber": 90 + }, + "action": { "type": "expire" } + } + ] + } + Tags: + - Key: platforma + Value: !Ref ClusterName + # 7. Storage — S3 bucket (primary storage) # --------------------------------------------------------------------------- S3Bucket: @@ -2993,6 +3305,20 @@ Resources: # the previous deny-all-except-role pattern because it locked out # account administrators from cleanup after stack deletion # (the IRSA role is deleted with the stack, leaving no one with access). + - !If + - HasMigrationStorageUserArn + - Sid: AllowMigrationUserWrite + Effect: Allow + Principal: + AWS: !Ref MigrationStorageUserArn + Action: + - s3:PutObject + - s3:GetObject + - s3:ListBucket + Resource: + - !GetAtt S3Bucket.Arn + - !Sub '${S3Bucket.Arn}/*' + - !Ref AWS::NoValue # --------------------------------------------------------------------------- # 8. ACM Certificate (DNS-validated via Route53) diff --git a/infrastructure/aws/kueue-values.yaml b/infrastructure/aws/kueue-values.yaml index 04fa15c..9b85e9b 100644 --- a/infrastructure/aws/kueue-values.yaml +++ b/infrastructure/aws/kueue-values.yaml @@ -14,16 +14,21 @@ controllerManager: resources: limits: cpu: 500m - memory: 512Mi + memory: 1Gi requests: cpu: 100m - memory: 256Mi + memory: 512Mi -# -- AppWrapper support -# Requires separate AppWrapper controller installation: +# -- Feature gates +# AppWrapper: Required for job wrapping. Requires separate AppWrapper controller: # kubectl apply --server-side -f https://github.com/project-codeflare/appwrapper/releases/download/v1.2.0/install.yaml +# ProvisioningACC: Enables ProvisioningRequest-based admission checks. +# Prevents resource fragmentation by checking with the autoscaler whether pods +# can actually be scheduled before admitting workloads. +# Enabled by default in Kueue v0.10+. Listed explicitly for visibility. featureGates: AppWrapper: true + ProvisioningACC: true # -- Framework integrations integrations: diff --git a/infrastructure/aws/migration.sh b/infrastructure/aws/migration.sh new file mode 100644 index 0000000..eae9a3e --- /dev/null +++ b/infrastructure/aws/migration.sh @@ -0,0 +1,306 @@ +#!/usr/bin/env bash +# ============================================================================= +# Platforma data migration script +# ============================================================================= +# Usage: +# migration.sh pre-helm — download dump, restore DB, sync storage +# migration.sh post-helm — invalidate caches, write marker, restart +# migration.sh — run both phases (for standalone use) +# ============================================================================= +set -euo pipefail + +if [ -z "${MIGRATION_DATABASE_S3_URI:-}" ]; then + echo "MIGRATION_DATABASE_S3_URI not set, skipping migration" + exit 0 +fi + +if [ -z "${PLATFORMA_IMAGE:-}" ] && [ -z "${PLATFORMA_VERSION:-}" ]; then + echo "ERROR: PLATFORMA_IMAGE or PLATFORMA_VERSION must be set" + exit 1 +fi + +PHASE="${1:-all}" +PVC="platforma-database" +DB_DIR="/data/database" +MARKER="$DB_DIR/.migration-complete" +PL_IMG="${PLATFORMA_IMAGE:-quay.io/milaboratories/platforma:${PLATFORMA_VERSION}}" +AWS_IMG="amazon/aws-cli:2.27.22" + +# Helper: wait for pod completion, stream logs, cleanup +wait_pod() { + local name="$1" timeout="$2" + kubectl wait pod/"$name" -n "$NAMESPACE" --for=condition=Ready --timeout=120s 2>/dev/null || true + kubectl logs -f "$name" -n "$NAMESPACE" || true + kubectl wait pod/"$name" -n "$NAMESPACE" --for=jsonpath='{.status.phase}'=Succeeded --timeout="${timeout}s" + kubectl delete pod/"$name" -n "$NAMESPACE" --ignore-not-found + echo " Pod $name completed" +} + +# Helper: generate pod YAML (aws-cli image, optional env vars) +make_pod() { + local name="$1" image="$2" script="$3" + shift 3 + + local indented_script + indented_script=$(echo "$script" | sed 's/^/ /') + + local env_section="" + if [ $# -gt 0 ]; then + env_section=" env:" + while [ $# -ge 2 ]; do + env_section="${env_section} + - name: ${1} + value: \"${2}\"" + shift 2 + done + fi + + cat </dev/null || true + + # Ensure gp3 storage class exists + if ! kubectl get sc gp3 2>/dev/null; then + echo "Creating gp3 storage class..." + kubectl apply -f - </dev/null; then + echo "Creating database PVC..." + kubectl apply -n "$NAMESPACE" -f - </dev/null || true + kubectl annotate pvc "$PVC" -n "$NAMESPACE" meta.helm.sh/release-name=platforma meta.helm.sh/release-namespace="$NAMESPACE" --overwrite 2>/dev/null || true + fi + + # Step 1: Download database dump + echo "--- Step 1/3: Downloading database dump ---" + DL_SCRIPT="${SKIP} +aws s3 cp ${MIGRATION_DATABASE_S3_URI} ${DB_DIR}/backup.gz +echo Download complete" + + kubectl delete pod/mgr-download -n "$NAMESPACE" --ignore-not-found 2>/dev/null + if [ -n "${MIGRATION_DB_ACCESS_KEY:-}" ]; then + make_pod mgr-download "$AWS_IMG" "$DL_SCRIPT" \ + AWS_ACCESS_KEY_ID "$MIGRATION_DB_ACCESS_KEY" \ + AWS_SECRET_ACCESS_KEY "$MIGRATION_DB_SECRET_KEY" \ + | kubectl apply -n "$NAMESPACE" -f - + else + make_pod mgr-download "$AWS_IMG" "$DL_SCRIPT" | kubectl apply -n "$NAMESPACE" -f - + fi + wait_pod mgr-download 600 + + # Step 2: Restore database + echo "--- Step 2/3: Restoring database ---" + RS_SCRIPT="${SKIP} +/app/platforma --restore-db=${DB_DIR}/backup.gz --db-dir=${DB_DIR} --force +echo Database restored" + + kubectl delete pod/mgr-restore -n "$NAMESPACE" --ignore-not-found 2>/dev/null + make_pl_pod mgr-restore "$RS_SCRIPT" | kubectl apply -n "$NAMESPACE" -f - + wait_pod mgr-restore 1800 + + # Step 3: Sync primary storage + if [ -n "${MIGRATION_SOURCE_BUCKET:-}" ]; then + echo "--- Step 3/3: Syncing primary storage ---" + SR="${MIGRATION_SOURCE_REGION:-$REGION}" + SYNC_SCRIPT="${SKIP} +aws s3 sync s3://${MIGRATION_SOURCE_BUCKET}/${MIGRATION_SOURCE_PREFIX:-} s3://${S3_BUCKET}/ --source-region ${SR} --region ${REGION} --no-progress --copy-props none +echo Storage sync complete" + + kubectl delete pod/mgr-sync -n "$NAMESPACE" --ignore-not-found 2>/dev/null + if [ -n "${MIGRATION_STORAGE_ACCESS_KEY:-}" ]; then + make_pod mgr-sync "$AWS_IMG" "$SYNC_SCRIPT" \ + AWS_ACCESS_KEY_ID "$MIGRATION_STORAGE_ACCESS_KEY" \ + AWS_SECRET_ACCESS_KEY "$MIGRATION_STORAGE_SECRET_KEY" \ + | kubectl apply -n "$NAMESPACE" -f - + else + make_pod mgr-sync "$AWS_IMG" "$SYNC_SCRIPT" | kubectl apply -n "$NAMESPACE" -f - + fi + wait_pod mgr-sync 86400 + else + echo "--- Step 3/3: Skipping storage sync (no source bucket) ---" + fi + + # Cleanup dump file + echo "--- Cleaning up dump file ---" + kubectl delete pod/mgr-cleanup -n "$NAMESPACE" --ignore-not-found 2>/dev/null + make_pod mgr-cleanup "$AWS_IMG" "rm -f ${DB_DIR}/backup.gz; echo Cleanup done" | kubectl apply -n "$NAMESPACE" -f - + wait_pod mgr-cleanup 120 + + echo "=== Pre-helm migration complete ===" +} + +# ========================================================================= +# Phase: post-helm (invalidate caches after Platforma applies migrations) +# ========================================================================= +run_post_helm() { + echo "=== Platforma cache invalidation (post-helm) ===" + + SKIP="if [ -f ${MARKER} ]; then echo Already completed; exit 0; fi" + + # Wait for Platforma to be ready (migrations applied) + echo "--- Waiting for Platforma to be ready ---" + kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=platforma -n "$NAMESPACE" --timeout=300s + + # Scale down Platforma to release database PVC (RWO) + echo "--- Scaling down Platforma for cache invalidation ---" + kubectl scale deployment/platforma -n "$NAMESPACE" --replicas=0 + kubectl wait --for=delete pod -l app.kubernetes.io/name=platforma -n "$NAMESPACE" --timeout=120s 2>/dev/null || true + + # Invalidate caches + echo "--- Invalidating caches ---" + IC_SCRIPT="${SKIP} +/app/platforma --invalidate-caches --main-root=/data/main --db-dir=${DB_DIR} +date -u > ${MARKER} +echo Caches invalidated" + + kubectl delete pod/mgr-invalidate -n "$NAMESPACE" --ignore-not-found 2>/dev/null + make_pl_pod mgr-invalidate "$IC_SCRIPT" | kubectl apply -n "$NAMESPACE" -f - + wait_pod mgr-invalidate 1800 + + # Scale Platforma back up + echo "--- Starting Platforma ---" + kubectl scale deployment/platforma -n "$NAMESPACE" --replicas=1 + kubectl rollout status deployment/platforma -n "$NAMESPACE" --timeout=300s + + echo "=== Migration fully complete ===" +} + +# ========================================================================= +# Main +# ========================================================================= +case "$PHASE" in + pre-helm) + run_pre_helm + ;; + post-helm) + run_post_helm + ;; + all) + run_pre_helm + echo "" + echo "WARNING: 'all' mode requires Platforma to be running between phases." + echo "Use 'pre-helm' and 'post-helm' separately in CF deployments." + echo "" + run_post_helm + ;; + *) + echo "Usage: $0 [pre-helm|post-helm|all]" + exit 1 + ;; +esac diff --git a/infrastructure/aws/values-aws-s3.yaml b/infrastructure/aws/values-aws-s3.yaml index 26842b7..89ee772 100644 --- a/infrastructure/aws/values-aws-s3.yaml +++ b/infrastructure/aws/values-aws-s3.yaml @@ -63,9 +63,13 @@ ingress: kueue: # Max resources for a single job (rejects jobs that exceed this). + # 484Gi = r7i.16xlarge EKS allocatable (~486 GiB) minus ~1 GiB EKS DaemonSet + # overhead minus 1 GiB safety margin for customer-installed DaemonSets. + # Matches GCP per-job ceiling on n2d-highmem-64 so deployment_size means the + # same workload capacity on both clouds. Reduce further if you run heavier DS. maxJobResources: cpu: 62 - memory: 500Gi + memory: 484Gi mode: dedicated # Node pools — controls where jobs run. # Must match your EKS node group labels and taints. @@ -93,7 +97,7 @@ kueue: memory: 256Gi batch: cpu: 126 - memory: 500Gi + memory: 484Gi # Platforma server runs on system nodes (not on compute nodes used for jobs). app: @@ -103,6 +107,6 @@ app: memory: 16Gi limits: cpu: 8 - memory: 16Gi + memory: 32Gi nodeSelector: node.kubernetes.io/pool: system