Integration Airflow#38
Closed
abir-oumghar wants to merge 37 commits into
Closed
Conversation
- Add KuboCD package definition for Apache Airflow 1.17.0 - Configure CeleryExecutor with PostgreSQL and Redis - Set up automatic provider installation (kubernetes, spark) - Add release configuration for airflow namespace - Configure ingress at airflow.okdp.local
- Create ServiceAccount 'spark' for Spark applications in airflow namespace - Add Role for Spark driver/executor pods management - Add Role for Airflow to manage SparkApplications CRD - Configure ClusterRole for Spark Operator cross-namespace access - Set up RoleBindings for service accounts
- Add 'Workflows' catalog with Airflow package - Configure kadCatalogsInfo with workflow orchestration display - Add kadServicesInfo for Airflow with description and icons - Configure airflow context with database and S3 storage - Add airflow endpoint URL configuration - Register airflow database in PostgreSQL provisioning
- Add spark_pi_example.py DAG for Scala Spark Pi calculation - Add pyspark_s3_wordcount.py DAG with S3/MinIO integration - Include comprehensive README with usage and troubleshooting - Add .gitignore for Python/Airflow artifacts - Demonstrate SparkKubernetesOperator usage and monitoring
- Add Airflow 2.10.5 package with Helm chart 1.17.0 - Configure CeleryExecutor with PostgreSQL and Redis - Enable Ingress for web UI at airflow.okdp.local - Add release configuration for kubocd-system deployment
- Update airflow package to use correct tag and minimal schema - Add sparkOperator.serviceAccount to context - Simplify release to use no parameters - Fix spark RBAC and DAGs configuration
- Fix tag to 2.9.3-p01 (matching existing OCI package) - Remove invalid Helm values (airflow.image, redis.auth, hosts[].path, hosts[].pathType, service.port) that are not in chart 1.17.0 schema - Remove custom parameters (postgresqlEnabled, ingressHost, etc.) to avoid schema validation errors - Use embedded PostgreSQL and Redis (no external dependencies) - Add dependencies: [] to prevent WAIT_DEPS in CI - Use correct context variables (certificateIssuers.selfSigned.name) - Reduce resource requests for Kind CI cluster compatibility - Disable unused components (flower, pgbouncer, statsd) - Remove ingressHost parameter from release
- Use LocalExecutor instead of CeleryExecutor (no Redis, no Workers) - Disable dagProcessor (scheduler handles it with LocalExecutor) - Disable Redis (not needed with LocalExecutor) - Set workers replicas to 0 - Enable migrateDatabaseJob - Reduces pods from ~8 to ~3, fixing CI timeout on Kind cluster
- Switch to LocalExecutor (no Redis, no workers, no dagProcessor) - Reduce webserver/scheduler requests to 128Mi/25m CPU - Reduce triggerer requests to 64Mi/25m CPU - Add minimal PostgreSQL resources (64Mi/25m CPU) - Add minimal resources for migrateDatabaseJob and createUserJob - Restore context schema for certificateIssuers and ingress - Keep triggerer and PostgreSQL enabled
…Off in CI The default bitnami/postgresql:16.1.0-debian-11-r15 image used by the Airflow Helm chart subchart fails with ImagePullBackOff on the Kind CI cluster, causing all Airflow pods to remain stuck in wait-for-airflow-migrations BackOff loop. Fix: explicitly set postgresql.image.tag to '16' (rolling stable tag) which always resolves to the latest available bitnami/postgresql 16.x image on Docker Hub.
…ubchart The bundled bitnami/postgresql subchart fails with ImagePullBackOff on the CI Kind cluster due to Docker Hub rate limiting. This blocks all Airflow pods in wait-for-airflow-migrations BackOff indefinitely. Fix: switch to the shared CNPG PostgreSQL instance already deployed on the platform (postgresql-instance-rw.cnpg-system.svc), following the same pattern used by Keycloak: - Disable postgresql.enabled - Configure data.metadataConnection to point to CNPG - Add airflow credentials to local-secrets-provider - Add dependencies on external-secrets, database-server, and ingress This eliminates the Docker Hub image pull entirely. Airflow now uses only images from apache/airflow (Docker Hub official, not rate-limited) and the CNPG PostgreSQL (ghcr.io, no rate limiting).
…on before schema creation - Adds 'tools' to dependencies to ensure the secret replicator is ready - Fixes race condition where PostgreSQL schema creation jobs start before secrets are replicated - This resolves the 'secret not found' errors for database credentials in CI
added 6 commits
February 19, 2026 15:44
abir-oumghar
force-pushed
the
integration-airflow
branch
from
d2ab8b9 to
4d8d636
Compare
abir-oumghar
force-pushed
the
integration-airflow
branch
from
81c4ea4 to
fd00125
Compare
|
This PR is huge and doesn't seem like it can be merged. The first issue seems to be adding Airflow to the sandbox; this can be done without the "Pipeline Examples" and "Spark Integration." Could you reduce the PR size by only including "feat" commits to add Airflow? |
Contributor
Author
|
The content of this PR was merged directly into Following your feedback, I opened PR #47 which is much smaller and focused: it only adds what was left on the Closing this one to avoid confusion. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The deployment configures the core Airflow package and establishes a dedicated OIDC client for secure access. It also sets up the Spark Operator and RBAC permissions needed to trigger Spark jobs directly from Airflow DAGs.
Key Changes