fix(mlflow): mint each run in its own workflow's experiment - #27
Merged
Conversation
Every execution after the first failed with "Cannot start run with ID X because active experiment ID does not match environment run ID". mcp_svc minted every run into caldera-mcp-client-1 regardless of workflow, but author sets the active experiment to caldera-mcp-FACTORY-client-1. mlflow.start_run(run_id=...) raises when the active experiment differs from the run's, so once an author run had switched the process-wide experiment, later runs could not resume. Deferring set_experiment out of import time is what exposed it: the first request runs before the switch and succeeds, so the failure starts on the second. Workflow now declares mlflow_experiment and mcp_svc mints there, giving one source of truth per workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every execution after the first failed with:
Cause
mcp_svcminted every run intocaldera-mcp-client-1regardless of whichworkflow was executing:
But
authorsets the process-wide active experiment to a different name,caldera-mcp-FACTORY-client-1. MLflow refuses to resume a run whose experimentdiffers from the active one (
mlflow/tracking/fluent.py):So once an author run had switched the active experiment, every later
mlflow.start_run(run_id=...)raised.Why it appeared to work once
Deferring
set_experimentout of import time exposed it rather than causing it.The first request runs before the switch and matches, so the failure starts on
the second request. Before the deferral both workflow modules called
set_experimentat import and whichever imported last won, which happened tomatch the hardcoded name.
Fix
Workflowdeclares its experiment andmcp_svcmints there, so there is onesource of truth per workflow. The two experiments stay deliberately separate.
Tests
tests/test_mlflow_experiment_binding.pycovers the binding, including a testthat pins the MLflow mechanism itself so a future refactor reintroducing a
hardcoded name fails here rather than in production.
Verified the tests catch the regression: reverting the fix gives
1 failed, 4 passed, restoring it gives5 passed.Baseline on
mainis 51 passed acrosstest_author_guards,test_configandtest_set_config_secrets; this branch is 56 with the 5 added. No regressions.Tests need CALDERA core importable: