Add config option [secrets]backends_order by moiseenkov · Pull Request #45931 · apache/airflow

moiseenkov · 2025-01-22T13:03:26Z

Introduce a new configuration option for specifying secret backends load order:

[secrets]backends_order = custom,environment_variable,metastore

The default value represents current behavior, thus nothing will change for existing users.

airflow/config_templates/config.yml

moiseenkov · 2025-01-23T08:29:35Z

@eladkal , please take a look at the updates.

potiuk · 2025-01-25T19:46:16Z

I was initially against making it configurable, but seeing the simplicity and flexibility, I am in.

potiuk · 2025-01-25T19:46:32Z

@eladkal ?

VladaZakharova · 2025-02-06T10:29:00Z

hi there!
@potiuk
Can we merge this one please?

VladaZakharova · 2025-02-13T10:40:54Z

Hi @potiuk @eladkal ! Are there some other changes we need to make here? Or we can merge this one?

eladkal · 2025-02-13T15:05:12Z

We are on feature freeze for Airflow 3.
https://lists.apache.org/thread/r26htzl0w3th7pw0l1y31g6s14qbtwwt

potiuk · 2025-02-15T20:43:55Z

Yeah. I think that might be 3.1

eladkal

prevent accidental merge. We are on feature freeze for Airflow 3.
PR can not be merged till main branch is for 3.1

eladkal · 2025-03-31T13:10:09Z

airflow-core/src/airflow/config_templates/config.yml

+    backends_order:
+      description: |
+        Comma-separated list of secret backends. These backends will be used in the order they are specified.
+        Please note that the `environment_variable` and `metastore` are required values and cannot be removed
+        from the list. Supported values are:
+
+        * ``custom``: Custom secret backend specified in the ``secrets[backend]`` configuration option.
+        * ``environment_variable``: Standard environment variable backend
+          ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+        * ``metastore``: Standard metastore backend ``airflow.secrets.metastore.MetastoreBackend``.
+      version_added: 3.0.0
+      type: string
+      example: ~
+      default: "custom,environment_variable,metastore"


i am a bit concerned making it just a "hidden" setting.
I think we better to come up with a way to expose the chosen order in the UI. that way DAG authors can verify and use it for debug for questions like (why I see wrong value in variable).

Also, correct me if I am wrong the order affect only read, not write. maybe the setting should be backends_read_order?

Hello @eladkal ! I don't see any info regarding the secrets section the UI is using right now. So maybe we could skip it for now? What do you think?
Regarding the naming. As stated in the doc it is responsible for secrets backends search ordering. So maybe backends_search_order is more convenient?

right now it's not configurable so the Airflow docs is good enough but after this PR is accepted the Airflow docs can't help you... You'll need to check the deployment to understand the priority order.
In many deployments dag authors don't have access to the deployment code.

My thoughts on the UI is an improvement (I think crucial but that is just my own perspective) I welcome others to share their thoughts. This feature is already targeting 3.1+ so regardless if we do the UI part or not we can't merge it now. We need to wait for main branch to become 3.1 (probably only after we cut RC1 for Airflow 3(

Hi there!
I see that we are close to release Af3.1, so maybe we can merge this change?
Thanks :)

potiuk · 2025-07-01T08:48:38Z

This looks good to me - but I think it might be worth to raise a devlist discussion for it @VladaZakharova -> there were past discussions about changing the sequence of resolving configurations, and I know people have strong opinion about "fixed" vs. "confifurable" sequence - and there are arguments pro / against each of those options.

I think it would be good to raise a discussion asking what peopel think about it and try to reach consensus.

eladkal · 2025-07-01T10:45:42Z

I think it would be good to raise a discussion asking what peopel think about it and try to reach consensus.

I agree. I am not comfortable with making this change without the UI indication / other mechanisem that allows dag authors to see the cluster admin setup for backend order

amoghrajesh · 2025-11-20T06:36:18Z

@VladaZakharova @moiseenkov @Crowiant I will take a look at this one soon.

uranusjr · 2025-11-21T07:58:09Z

I wonder if this should just be called backends and have the description explain how the ordering is significant. The implementation is fine to me.

uranusjr · 2025-11-21T07:59:39Z

airflow-core/docs/security/secrets/secrets-backend/index.rst

    secrets_backend =
    secrets_backend_kwargs =
-
+    backends_order =


This doesn’t read right? The key implemented here is not under [workders].

Hello @uranusjr ! Thank you for your review and comments!
No, actually it should be under both sections: [secrets] and [workers]. Because after talking with @amoghrajesh We agreed that implementation should be extended on workers too.

amoghrajesh

We are certainly close on this one, some more work needs to be done to adjust the PR as per the latest codebase. For the UI changes, I am not so sure, so maybe someone else can review that: @potiuk / @eladkal since you had some suggestion there.

airflow-core/docs/security/secrets/secrets-backend/index.rst

airflow-core/src/airflow/configuration.py

amoghrajesh · 2025-11-25T07:32:13Z

airflow-core/src/airflow/configuration.py

+        execution_args = (
+            "airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend"
+            if "airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend"
+            in default_backends
+            else None
+        )


We are trying to avoid importing sdk in airflow core when possible

Could you expand a little bit on this? ExecutionAPISecretsBackend exists only in task-sdk and applies for the workers. Where else could it be imported?

Crowiant · 2026-01-12T14:16:03Z

Hello @amoghrajesh @uranusjr I responded to your comments in the PR. Could you please help with the review?

potiuk · 2026-01-22T11:57:59Z

@amoghrajesh @uranusjr -> any comments here? I think there were some important questions/comments from you and I would love to merge that one to make sure it makes it for 3.2 (it looks good in general).

eladkal · 2026-01-22T12:58:28Z

I don't have time to re-review but I still have some concerns. I advise to tag this as experimental feature so community will use this feature with judgment.
I will however ask to please present a UI image of how the indication of backend order is shown to users.

potiuk · 2026-01-22T13:57:44Z

I advise to tag this as experimental feature so community will use this feature with judgment.

Good idea @eladkal

Crowiant · 2026-01-29T19:55:37Z

Hello @potiuk @eladkal Thank you for your comments. I added experimental tag to the feature.
Here are screenshots of how backends order looks on UI:

potiuk · 2026-02-03T14:01:40Z

Can you please resolve conflicts and fix the remaining static check?

Crowiant · 2026-02-16T09:42:05Z

Hello @potiuk I updated the PR. Since checks are passing, can we merge if there are no objections?

potiuk · 2026-03-01T13:12:26Z

Sorry for that - I've been a bit busy - can you please rebase one more time as it needs more conflict resolution - also the UI part is somethign maybe @jason810496 can you re-review?

potiuk · 2026-03-04T17:11:32Z

@eladkal - looks like your concerns are alleviated - can you please remove "request changes"?

moiseenkov requested review from ashb, dstandish and potiuk as code owners January 22, 2025 13:03

boring-cyborg bot added the area:secrets label Jan 22, 2025

eladkal requested changes Jan 22, 2025

View reviewed changes

airflow/config_templates/config.yml Outdated Show resolved Hide resolved

airflow/config_templates/config.yml Show resolved Hide resolved

moiseenkov force-pushed the secrets-backends-order branch from 61b5ab6 to 4489808 Compare January 22, 2025 13:38

moiseenkov requested a review from eladkal January 22, 2025 13:45

moiseenkov force-pushed the secrets-backends-order branch from 4489808 to 58d80f3 Compare January 22, 2025 14:35

potiuk approved these changes Jan 25, 2025

View reviewed changes

raphaelauv mentioned this pull request Mar 11, 2025

Add a way to skip the secret_backend #19251

Open

2 tasks

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from 8c516f8 to 347e2f1 Compare March 31, 2025 13:00

eladkal requested changes Mar 31, 2025

View reviewed changes

eladkal added this to the Arflow 3.1+ milestone Mar 31, 2025

eladkal reviewed Mar 31, 2025

View reviewed changes

eladkal modified the milestones: Airflow 3.1+, Airflow 3.1.0 Apr 21, 2025

Crowiant force-pushed the secrets-backends-order branch from 347e2f1 to 2b750de Compare April 24, 2025 12:21

Crowiant force-pushed the secrets-backends-order branch from 2b750de to 4c0d958 Compare May 23, 2025 14:52

potiuk requested a review from eladkal June 30, 2025 21:02

uranusjr reviewed Nov 21, 2025

View reviewed changes

amoghrajesh reviewed Nov 25, 2025

View reviewed changes

potiuk mentioned this pull request Dec 6, 2025

Fix : 'airflow connections test' failure by adding DB fallback #59038

Closed

Crowiant force-pushed the secrets-backends-order branch from 64d53e7 to 437208f Compare December 17, 2025 13:41

Crowiant requested a review from vatsrahul1001 as a code owner December 17, 2025 13:41

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from 785af81 to 2015d8b Compare December 22, 2025 17:14

Crowiant force-pushed the secrets-backends-order branch from 2015d8b to a5092ed Compare January 8, 2026 21:42

Crowiant requested a review from choo121600 as a code owner January 8, 2026 21:42

Crowiant force-pushed the secrets-backends-order branch from a5092ed to 41b4d5d Compare January 12, 2026 14:14

Crowiant force-pushed the secrets-backends-order branch from 41b4d5d to 4ff5632 Compare January 29, 2026 19:53

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from fee065b to 9a7803e Compare February 6, 2026 18:16

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from c1085eb to 903e4fb Compare March 4, 2026 17:04

Add config option [secrets]backends_order

2e4517f

Conversation

moiseenkov commented Jan 22, 2025

Uh oh!

Uh oh!

Uh oh!

moiseenkov commented Jan 23, 2025

Uh oh!

potiuk commented Jan 25, 2025

Uh oh!

potiuk commented Jan 25, 2025

Uh oh!

VladaZakharova commented Feb 6, 2025

Uh oh!

VladaZakharova commented Feb 13, 2025

Uh oh!

eladkal commented Feb 13, 2025

Uh oh!

potiuk commented Feb 15, 2025

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

potiuk commented Jul 1, 2025

Uh oh!

eladkal commented Jul 1, 2025

Uh oh!

amoghrajesh commented Nov 20, 2025

Uh oh!

uranusjr commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Crowiant commented Jan 12, 2026

Uh oh!

potiuk commented Jan 22, 2026

Uh oh!

eladkal commented Jan 22, 2026

Uh oh!

potiuk commented Jan 22, 2026

Uh oh!

Crowiant commented Jan 29, 2026

Uh oh!

potiuk commented Feb 3, 2026

Uh oh!

Crowiant commented Feb 16, 2026

Uh oh!

potiuk commented Mar 1, 2026

Uh oh!

potiuk commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants