Skip to content

fix(mlflow): mint each run in its own workflow's experiment - #27

Merged
HackedRico merged 1 commit into
mainfrom
fix/mlflow-experiment-mismatch
Aug 27, 2026
Merged

fix(mlflow): mint each run in its own workflow's experiment#27
HackedRico merged 1 commit into
mainfrom
fix/mlflow-experiment-mismatch

Conversation

@HackedRico

Copy link
Copy Markdown
Collaborator

Every execution after the first failed with:

Cannot start run with ID <id> because active experiment ID does not match
environment run ID.

Cause

mcp_svc minted every run into caldera-mcp-client-1 regardless of which
workflow was executing:

_exp = mlflow.get_experiment_by_name("caldera-mcp-client-1")

But author sets the process-wide active experiment to a different name,
caldera-mcp-FACTORY-client-1. MLflow refuses to resume a run whose experiment
differs from the active one (mlflow/tracking/fluent.py):

if (_active_experiment_id is not None
        and _active_experiment_id != active_run_obj.info.experiment_id):
    raise MlflowException(f"Cannot start run with ID {existing_run_id} ...")

So once an author run had switched the active experiment, every later
mlflow.start_run(run_id=...) raised.

Why it appeared to work once

Deferring set_experiment out of import time exposed it rather than causing it.
The first request runs before the switch and matches, so the failure starts on
the second request. Before the deferral both workflow modules called
set_experiment at import and whichever imported last won, which happened to
match the hardcoded name.

Fix

Workflow declares its experiment and mcp_svc mints there, so there is one
source of truth per workflow. The two experiments stay deliberately separate.

mlflow_experiment: str = "caldera-mcp-client-1"

Tests

tests/test_mlflow_experiment_binding.py covers the binding, including a test
that pins the MLflow mechanism itself so a future refactor reintroducing a
hardcoded name fails here rather than in production.

Verified the tests catch the regression: reverting the fix gives 1 failed, 4 passed, restoring it gives 5 passed.

Baseline on main is 51 passed across test_author_guards, test_config and
test_set_config_secrets; this branch is 56 with the 5 added. No regressions.

Tests need CALDERA core importable:

PYTHONPATH=<caldera-root> python -m pytest plugins/mcp/tests/test_mlflow_experiment_binding.py -v

Every execution after the first failed with "Cannot start run with ID X
because active experiment ID does not match environment run ID".

mcp_svc minted every run into caldera-mcp-client-1 regardless of
workflow, but author sets the active experiment to
caldera-mcp-FACTORY-client-1. mlflow.start_run(run_id=...) raises when
the active experiment differs from the run's, so once an author run had
switched the process-wide experiment, later runs could not resume.

Deferring set_experiment out of import time is what exposed it: the
first request runs before the switch and succeeds, so the failure starts
on the second.

Workflow now declares mlflow_experiment and mcp_svc mints there, giving
one source of truth per workflow.
@HackedRico
HackedRico merged commit 73b7088 into main Aug 27, 2026
3 checks passed
@HackedRico
HackedRico deleted the fix/mlflow-experiment-mismatch branch August 27, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant