Conditional branching broken on Argo Workflows v3.7.11+ / v4.0.2+

[cboylston]

Description

Flows using conditional branching (self.next({...}, condition=...)) hang indefinitely on Argo Workflows v3.7.11+ and v4.0.2+. The convergence step after conditional branches is never scheduled.

Reproducer

The metaflow-music-agent example reproduces this. The ResearchAgentFlow uses:

self.next(
    {
        "continue": self.research,
        "done": self.done,
        "unknown": self.unknown,
        "failed": self.failed,
    },
    condition="decision",
)

done, unknown, and failed all converge on agent_complete -> join -> end. Only one branch executes per foreach iteration; the others are skipped.

Run with:

python agentflow.py --environment=pypi --with kubernetes run --max-artists 2

The research steps complete successfully, each branch reaches done or unknown, but agent_complete is never scheduled and the workflow hangs in Running state.

Root Cause

Two interacting issues:

1. How Metaflow generates input-paths for conditional join steps

For a convergence step like agent_complete, the generated Argo DAG template references outputs from all conditional branches unconditionally:

arguments:
  parameters:
    - name: input-paths
      value: "argo-{{workflow.name}}/:done/{{tasks.done.outputs.parameters.task-id}},failed/{{tasks.failed.outputs.parameters.task-id}},unknown/{{tasks.unknown.outputs.parameters.task-id}}"
depends: "done.Succeeded || failed.Succeeded || unknown.Succeeded"

The depends clause is correct (OR), but input-paths references task-id from all three branches. Skipped branches produce no outputs.

Metaflow's intended workaround (introduced in #2560) is to base64-encode the value and filter out unresolved paths at runtime via conditional_input_paths.py. This depends on Argo passing through unresolved {{tasks.X...}} expressions as literal strings.

2. Argo Workflows changed parameter resolution behavior

Argo PR argoproj/argo-workflows#15442 ("fix: requeue workflow if expected variables are missing", Feb 26 2026) changed dag.go from:

// Before (v3.7.10, v4.0.1 and earlier) — unresolved expressions passed through
newTaskStr, err := template.Replace(ctx, string(taskBytes), mergedParams, true)

to:

// After (v3.7.11, v4.0.2+) — unresolved tasks.*/steps.* references cause requeue
newTaskStr, err := template.ReplaceStrict(ctx, string(taskBytes), mergedParams, []string{"tasks", "steps"})
if err != nil {
    if template.IsMissingVariableErr(err) {
        woc.requeue()
        return nil, ErrRequeue  // pod never created, runtime filter never runs
    }
}

The controller now requeues indefinitely when it can't resolve {{tasks.failed.outputs.parameters.task-id}} for a skipped task. The conditional_input_paths.py runtime filter never gets a chance to execute.

Controller logs confirm this

level=WARN msg="was unable to find variable" error="failed to resolve {{tasks.failed.outputs.parameters.task-id}}"
level=WARN msg="was unable to find variable" error="failed to resolve {{tasks.done.outputs.parameters.task-id}}"
level=WARN msg="was unable to obtain the node" taskName=agent-complete

Affected Versions

Argo Version	Behavior	Conditionals work?
v3.5.x - v3.7.10	Replace(allowUnresolved=true)	Yes
v3.7.11+	ReplaceStrict	No
v4.0.0 - v4.0.1	Replace(allowUnresolved=true)	Yes
v4.0.2+	ReplaceStrict	No

Tested with Metaflow 2.19.22. Likely affects all versions since #2560 (Aug 2025).

History

For context, the original conditionals PR (#2550, Aug 19 2025) used safe Argo expressions that handled skipped tasks correctly:

{{=(get(tasks['done']?.outputs?.parameters, 'task-id') ?? 'no-task')}}

This was removed two days later in #2560 (Aug 21 2025) because ?. optional chaining doesn't work inside foreach (withParam nested DAGs evaluate all ?. as false). The replacement strategy relied on Argo's passthrough behavior, which is no longer guaranteed.

Argo-side tracking

This is related to argoproj/argo-workflows#15737. A fix (PR #15736) has been merged and cherry-picked to v4.0 and v3.7, but that fix targets expanded task properties (withItems/withParams). It may not fully cover the skipped conditional branch case — worth verifying once v3.7.12 / v4.0.4 are released.

Workaround

Downgrade Argo Workflows to v3.7.10 or v4.0.1.

Environment

• Metaflow: 2.19.22
• Argo Workflows: tested on v4.0.3 and v3.7.11
• Kubernetes: EKS
• Python: 3.12

[devendrakushwah80]

Hi, I have been looking into this issue and trying to understand what’s going on with conditional input paths and Argos ReplaceStrict behavior

From what I understand the current approach relies on unresolved #3043 expressions being passed through so they can be handled later at runtime But with ReplaceStrict these seem to fail early and the workflow keeps requeueing before that logic runs

It feels like this might need to be handled earlier during DAG generation instead of relying on unresolved expressions

Does that sound right or is there already a preferred way to handle skipped branches here

Happy to explore more or try a small prototype if that helps

[devendrakushwah80]

I’ve opened a PR with an initial fix for this #3052
This filters out conditional nodes while constructing input-paths to avoid unresolved references with ReplaceStrict
Happy to iterate further based on feedback

[Ritik-tech731]

Hi, I am a GSoC 2026 applicant and I am interested in working on this issue.
I have started exploring the Metaflow codebase and would like to understand if this issue is suitable for a beginner contributor. I would appreciate any guidance on where to start.
Thanks!

[Imen-source]

Hi maintainers, I’d like to work on creating a Windows- and Linux-friendly helper script (argo_branch_debugger.py) to debug conditional branches on Argo Workflows for this issue. Could you please confirm if this is approved as a contribution? Thank you!

[AyoubArdem]

Hi, I’d like to work on this issue and submit a PR if no one is currently assigned to it. I also wanted to mention that I applied to Metaflow through Google Summer of Code, and I’d be happy to contribute through this fix as part of getting more involved with the project. Please let me know if that works.

[odncode]

I've been reading through the history on this from #2550, #2560, and the Argo-side changes.
The core issue here is clear as day to be honest.

The problem in summary: Metaflow constructs input-paths at DAG generation time
by referencing {{tasks.X.outputs.parameters.task-id}} for every in_func of a convergence
step (L1356-1362 in argo_workflows.py):

Parameter("input-paths").value(
    compress_list([
        "argo-{{workflow.name}}/%s/{{tasks.%s.outputs.parameters.task-id}}"
        % (n, self._sanitize(n))
        for n in node.in_funcs
    ], zlibmin=inf)
)

This is unconditional. it references all branches whether they execute or not. The runtime
filter in conditional_input_paths.py then strips paths containing {{ (i.e. unresolved
references from skipped branches). This worked when Argo passed through unresolved expressions
as literals, but ReplaceStrict in v3.7.11+ rejects them before the pod starts.

The two previous approaches and why they failed:

• feature: conditionals for Argo Workflows #2550 — Argo ?. optional chaining: {{=(get(tasks['done']?.outputs?.parameters, 'task-id') ?? 'no-task')}} — handles skipped branches cleanly but breaks inside foreach/withParam nested DAGs because ?. evaluates to false in that context.
• chore: re-enable conditionals in foreach on argo workflows #2560 — Passthrough + runtime filter: relies on Argo passing unresolved expressions as strings. No longer works with ReplaceStrict.

What I'm trying to understand:
The fix needs to resolve branch outputs without relying on either ?. (broken in foreach)
or passthrough (broken in strict mode). A few questions:

1. Has anyone explored using Argo's tasks.X.status to conditionally construct the
   input-paths expression? Something like
   {{= tasks['done'].status == 'Succeeded' ? tasks['done'].outputs.parameters['task-id'] : 'skip' }}. does this also break inside foreach?
2. The PR in Fix conditional input paths handling for Argo DAG generation #3052 filters conditional nodes from input-paths during DAG generation. But
   if we remove the skipped branch references at generation time, how does the convergence
   step know which branch's output to consume? Doesn't it need at least one resolved path?
3. Is there a reason Metaflow can't emit one DAGTask per possible branch outcome at the
   convergence point, each with a when clause, rather than one task that tries to reference
   all branches?

Not proposing a solution yet — just trying to understand the constraint space before writing
any code.