Add config option [secrets]backends_order

Author: moiseenkov

airflow-core/docs/security/secrets/secrets-backend/index.rst

@@ -39,13 +39,15 @@ When looking up a connection/variable, by default Airflow will search environmen
 database second.
 
 If you enable an alternative secrets backend, it will be searched first, followed by environment variables,
-then metastore.  This search ordering is not configurable. Though, in some alternative secrets backend you might have
+then metastore. Though, in some alternative secrets backend you might have
 the option to filter which connection/variable/config is searched in the secret backend. Please look at the
 documentation of the secret backend you are using to see if such option is available.
 
 On the other hand, if a workers secrets backend is defined, the order of lookup has higher priority for the workers secrets
 backend and then the secrets backend.
 
+The secrets backends search ordering is also configurable via the configuration option ``[secrets]backends_order``.
+
 .. warning::
 
     When using environment variables or an alternative secrets backend to store secrets or variables, it is possible to create key collisions.
@@ -64,12 +66,21 @@ The ``[secrets]`` section has the following options:
     [secrets]
     backend =
     backend_kwargs =
+    backends_order =
 
 Set ``backend`` to the fully qualified class name of the backend you want to enable.
 
 You can provide ``backend_kwargs`` with json and it will be passed as kwargs to the ``__init__`` method of
 your secrets backend.
 
+``backends_order`` is a comma-separated list of secret backends. These backends will be used in the order they are specified.
+Please note that the ``environment_variable`` and ``metastore`` are required values and cannot be removed
+from the list. Supported values are:
+
+* ``custom``: Custom secret backend specified in the ``secrets[backend]`` configuration option.
+* ``environment_variable``: Standard environment variable backend ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+* ``metastore``: Standard metastore backend ``airflow.secrets.metastore.MetastoreBackend``.
+
 If you want to check which secret backend is currently set, you can use ``airflow config get-value secrets backend`` command as in
 the example below.
 
@@ -89,13 +100,21 @@ configure separate secrets backend for workers, you can do that using:
     [workers]
     secrets_backend =
     secrets_backend_kwargs =
-
+    backends_order =
 
 Set ``secrets_backend`` to the fully qualified class name of the backend you want to enable.
 
 You can provide ``secrets_backend_kwargs`` with json and it will be passed as kwargs to the ``__init__`` method of
 your secrets backend for the workers.
 
+``backends_order`` is a comma-separated list of secret backends for workers. These backends will be used in the order they are specified.
+Please note that the ``environment_variable`` and ``execution_api`` are required values and cannot be removed
+from the list. Supported values are:
+
+* ``custom``: Custom secret backend specified in the ``workers[secrets_backend]`` configuration option.
+* ``environment_variable``: Standard environment variable backend ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+* ``execution_api``: Standard execution_api backend ``airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend``.
+
 If you want to check which secret backend is currently set, you can use ``airflow config get-value workers secrets_backend`` command as in
 the example below.
 

airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml

@@ -94,6 +94,52 @@ paths:
       security:
       - OAuth2PasswordBearer: []
       - HTTPBearer: []
+  /ui/backends_order:
+    get:
+      tags:
+      - Config
+      summary: Get Backends Order Value
+      operationId: get_backends_order_value
+      security:
+      - OAuth2PasswordBearer: []
+      - HTTPBearer: []
+      parameters:
+      - name: accept
+        in: header
+        required: false
+        schema:
+          type: string
+          enum:
+          - application/json
+          - text/plain
+          - '*/*'
+          default: '*/*'
+          title: Accept
+      responses:
+        '200':
+          description: Successful Response
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Config'
+        '404':
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPExceptionResponse'
+          description: Not Found
+        '406':
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPExceptionResponse'
+          description: Not Acceptable
+        '422':
+          description: Validation Error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/HTTPValidationError'
   /ui/connections/hook_meta:
     get:
       tags:
@@ -1336,6 +1382,41 @@ components:
       - count
       title: CalendarTimeRangeResponse
       description: Represents a summary of DAG runs for a specific calendar time range.
+    Config:
+      properties:
+        sections:
+          items:
+            $ref: '#/components/schemas/ConfigSection'
+          type: array
+          title: Sections
+      additionalProperties: false
+      type: object
+      required:
+      - sections
+      title: Config
+      description: List of config sections with their options.
+    ConfigOption:
+      properties:
+        key:
+          type: string
+          title: Key
+        value:
+          anyOf:
+          - type: string
+          - prefixItems:
+            - type: string
+            - type: string
+            type: array
+            maxItems: 2
+            minItems: 2
+          title: Value
+      additionalProperties: false
+      type: object
+      required:
+      - key
+      - value
+      title: ConfigOption
+      description: Config option.
     ConfigResponse:
       properties:
         fallback_page_limit:
@@ -1398,6 +1479,23 @@ components:
       - multi_team
       title: ConfigResponse
       description: configuration serializer.
+    ConfigSection:
+      properties:
+        name:
+          type: string
+          title: Name
+        options:
+          items:
+            $ref: '#/components/schemas/ConfigOption'
+          type: array
+          title: Options
+      additionalProperties: false
+      type: object
+      required:
+      - name
+      - options
+      title: ConfigSection
+      description: Config Section Schema.
     ConnectionHookFieldBehavior:
       properties:
         hidden:

airflow-core/src/airflow/api_fastapi/core_api/routes/ui/config.py

@@ -19,13 +19,20 @@
 from json import loads
 from typing import Any
 
-from fastapi import Depends, status
+from fastapi import Depends, HTTPException, status
 
+from airflow.api_fastapi.common.headers import HeaderAcceptJsonOrText
 from airflow.api_fastapi.common.router import AirflowRouter
 from airflow.api_fastapi.common.types import UIAlert
+from airflow.api_fastapi.core_api.datamodels.config import (
+    Config,
+    ConfigOption,
+    ConfigSection,
+)
 from airflow.api_fastapi.core_api.datamodels.ui.config import ConfigResponse
 from airflow.api_fastapi.core_api.openapi.exceptions import create_openapi_http_exception_doc
 from airflow.api_fastapi.core_api.security import requires_authenticated
+from airflow.api_fastapi.core_api.services.public.config import _response_based_on_accept
 from airflow.configuration import conf
 from airflow.settings import DASHBOARD_UIALERTS
 from airflow.utils.log.log_reader import TaskLogReader
@@ -67,3 +74,32 @@ def get_configs() -> ConfigResponse:
     config.update({key: value for key, value in additional_config.items()})
 
     return ConfigResponse.model_validate(config)
+
+
+@config_router.get(
+    "/backends_order",
+    responses={
+        **create_openapi_http_exception_doc(
+            [
+                status.HTTP_404_NOT_FOUND,
+                status.HTTP_406_NOT_ACCEPTABLE,
+            ]
+        ),
+    },
+    response_model=Config,
+    dependencies=[Depends(requires_authenticated())],
+)
+def get_backends_order_value(
+    accept: HeaderAcceptJsonOrText,
+):
+    section, option = "secrets", "backends_order"
+    if not conf.has_option(section, option):
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"Option [{section}/{option}] not found.",
+        )
+
+    value = conf.get(section, option)
+
+    config = Config(sections=[ConfigSection(name=section, options=[ConfigOption(key=option, value=value)])])
+    return _response_based_on_accept(accept, config)

airflow-core/src/airflow/config_templates/config.yml

@@ -1339,6 +1339,22 @@ secrets:
       sensitive: true
       example: ~
       default: ""
+    backends_order:
+      description: |
+        .. note:: |experimental|
+
+        Comma-separated list of secret backends. These backends will be used in the order they are specified.
+        Please note that the `environment_variable` and `metastore` are required values and cannot be removed
+        from the list. Supported values are:
+
+        * ``custom``: Custom secret backend specified in the ``secrets[backend]`` configuration option.
+        * ``environment_variable``: Standard environment variable backend
+          ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+        * ``metastore``: Standard metastore backend ``airflow.secrets.metastore.MetastoreBackend``.
+      version_added: 3.2.0
+      type: string
+      example: ~
+      default: "custom,environment_variable,metastore"
     use_cache:
       description: |
         .. note:: |experimental|
@@ -1632,6 +1648,24 @@ workers:
       sensitive: true
       example: ~
       default: ""
+    backends_order:
+      description: |
+        .. note:: |experimental|
+
+        Comma-separated list of secret backends for workers. These backends will be used in the order they are
+        specified. Please note that the ``environment_variable`` and ``execution_api`` are required values and
+        cannot be removed from the list. Supported values are:
+
+        * ``custom``: Custom secret backend specified in the ``workers[secrets_backend]`` configuration
+          option.
+        * ``environment_variable``: Standard environment variable backend
+          ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+        * ``execution_api``: Standard execution_api backend
+          ``airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend``.
+      version_added: 3.2.0
+      type: string
+      example: ~
+      default: "custom,environment_variable,execution_api"
     min_heartbeat_interval:
       description: |
         The minimum interval (in seconds) at which the worker checks the task instance's