Chart: Implement deployPerBundle feature for DAG processor deployments

[uplsh580]

Related Issue

Issue: #61037

Description

This PR implements per-bundle DAG Processor deployments feature, allowing users to create separate Kubernetes deployments for each DAG bundle defined in dagBundleConfigList. This enables independent resource isolation, scaling, and configuration per bundle.

Motivation

Currently, when multiple DAG bundles are configured, all bundles are processed by a single DAG processor deployment. This limits the ability to:

• Scale bundles independently based on their workload
• Apply bundle-specific resource configurations
• Isolate bundle processing failures
• Use bundle-specific node selectors, affinities, or tolerations

Changes

Configuration (values.yaml)

Added new dagProcessor.deployPerBundle section:

dagProcessor:
  deployPerBundle:
    enabled: false  # Enable per-bundle deployments
    args: ["bash", "-c", "exec airflow dag-processor --bundle-name {{ bundleName }}"]
    bundleOverrides: {}  # Per-bundle configuration overrides

Features

1. Per-bundle Deployments: When deployPerBundle.enabled is true, creates a separate Deployment for each bundle in dagBundleConfigList
2. Bundle-specific Args: Supports templated args with {{ bundleName }} placeholder that gets replaced with actual bundle name
3. Bundle Overrides: Allows per-bundle configuration overrides via bundleOverrides map:
   • Resources (CPU, memory)
   • Replicas
   • Node selectors, affinities, tolerations
   • Environment variables
   • Pod disruption budgets
   • And other deployment settings
4. Per-bundle PodDisruptionBudget: Creates separate PDBs for each bundle when enabled
5. Backward Compatibility: When deployPerBundle.enabled is false, maintains existing single deployment behavior

Implementation Details

• Refactored deployment logic into a reusable helper template dag-processor.deployment
• Refactored PDB logic into a reusable helper template dag-processor.poddisruptionbudget
• Supports per-bundle enable/disable via bundleOverrides[bundleName].enabled

Files Changed

• chart/templates/dag-processor/dag-processor-deployment.yaml: Added per-bundle deployment logic
• chart/templates/dag-processor/dag-processor-poddisruptionbudget.yaml: Added per-bundle PDB logic
• chart/values.yaml: Added deployPerBundle configuration section
• chart/values.schema.json: Added schema validation for deployPerBundle
• helm-tests/tests/helm_tests/airflow_core/test_dag_processor_per_bundle.py: Added comprehensive test cases

Usage Example

dagProcessor:
  enabled: true

  dagBundleConfigList:
    - name: bundle1
      classpath: "airflow.providers.git.bundles.git.GitDagBundle"
      kwargs:
        git_conn_id: "GITHUB__SAMPLE"
        subdir: "dags"
        tracking_ref: "main"
        refresh_interval: 60
    - name: bundle2
      classpath: "airflow.providers.git.bundles.git.GitDagBundle"
      kwargs:
        git_conn_id: "GITHUB__SAMPLE2"
        subdir: "dags"
        tracking_ref: "main"
        refresh_interval: 60

  deployPerBundle:
    enabled: true
    bundleOverrides:
      bundle1:
        replicas: 3
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        podDisruptionBudget:
          config:
            minAvailable: 2
            maxUnavailable: ~
      bundle2:
        replicas: 1
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
        podDisruptionBudget:
          config:
            minAvailable: 1
            maxUnavailable: ~

This will create:

• {release-name}-dag-processor-bundle1 deployment with 3 replicas and production resources
• {release-name}-dag-processor-bundle2 deployment with 1 replica and standard resources
• Separate PodDisruptionBudgets for each bundle (if enabled)

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
• cursor

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[uplsh580]

Validation - deployPerBundle

bundle1

bundle2

poddisruptionbudgets

[uplsh580]

Validation - ASIS (Single Deployments)

dag processor log

poddisruptionbudgets

[choo121600]

Great work!
It looks like there’s a merge conflict. Could you please take a look?

[jscheffl]

I like the feature very much - mainly that different dag processors can be spawned - but am not sure if I really like the parallel structure of parameters for the deployment. I feel like it would be better in the bundle definition directly. But would like to have other opinions as well.

As we are currently discussing a structural change to a version 2.0 of the helm chart it might be we are hesitant to add this short term as a feature as if we in general re-structure (potentially) then it might be also very good suited in a 2.0 version.

[jscheffl]

I am not sure and would like opinions of other maintainers whether this structure is well suited. It is fully arallel to the definition of the bundles itself. Technically this is possilbe but then two lists need to be maintained with a risk of inconsistency.

Is there a reason that you did not add the structure below the bundle list definition?

[uplsh580]

Thank you for the feedback. I agree that maintaining two separate lists could lead to configuration drift and unnecessary complexity.

To address this, how about nesting the deployment configuration directly within the dagBundleConfigList? This way, each bundle's definition and its deployment strategy are coupled together, ensuring better consistency.

Here is an example of the proposed structure:

deployPerBundle: true  # default is `false`

dagBundleConfigList:
  - name: bundle1
    classpath: ...
    kwargs: ...
    deploymentOverride:           # Bundle-specific deployment settings
      enabled: true
      replicas: 3
      resources: ...
  - name: bundle2
    classpath: ...
    kwargs: {}
    # If 'deployment' is omitted, it could fall back to default values

[jscheffl]

Is there a reason that the args are populated? Which reasons would be there to override this? Is it needed to expose this as (yet another) parameter?

[uplsh580]

Thank you for pointing that out. To be honest, I also had concerns that exposing args might add unnecessary complexity and potentially confuse users.

The reason I included it was primarily for consistency with the existing single DAG processor configuration, which already allows overriding args in the current Helm Chart. I wanted to ensure that the bundle-specific processors followed the same pattern as the standard one, just in case users needed that same level of flexibility.

airflow/chart/values.yaml

Line 2383 in fbf687c

args: ["bash", "-c", "exec airflow dag-processor"]

To simplify this, I’d like to propose automating the argument injection instead.
When deployPerBundle is enabled, the template can take the base args from the global dagProcessor (e.g., ["bash", "-c", "exec airflow dag-processor"]) and automatically append the --bundle-name {{ bundleName }} flag for each specific deployment.
However, I am a bit cautious about this approach because I am not fully aware of all the different ways users might be overriding the default args. If a user has a very complex custom command, simply appending a flag at the end might not work in every scenario.

[uplsh580]

As we are currently discussing a structural change to a version 2.0 of the helm chart it might be we are hesitant to add this short term as a feature as if we in general re-structure (potentially) then it might be also very good suited in a 2.0 version.

That’s a fair point. While I’d personally love to see this feature implemented, I agree that structural consistency in version 2.0 is just as important. I’m happy to go with whatever the maintainers feel is best for the long-term roadmap, whether it's added now or as part of the 2.0 restructuring.

[uplsh580]

@jscheffl
I’ve updated the code based on your comments. I’d appreciate any further feedback.