Skip to content

fix(history): give each MLflow run one owning task - #31

Merged
HackedRico merged 1 commit into
mitre:mainfrom
HackedRico:fix/mlflow-run-lifecycle
Aug 28, 2026
Merged

fix(history): give each MLflow run one owning task#31
HackedRico merged 1 commit into
mitre:mainfrom
HackedRico:fix/mlflow-run-lifecycle

Conversation

@HackedRico

Copy link
Copy Markdown
Collaborator

Description

Fixes #29.

mlflow.tracking.fluent._active_run_stack is a thread-local, and every
/plugin/mcp/execute request is an asyncio task on one aiohttp event-loop
thread. Concurrent runs therefore shared one active-run pointer, so a bare
end_run() terminated someone else's run, tags landed on the wrong run, and a
write with an empty stack silently minted a phantom run.

  • New app/mlflow_run.py: RunTracker binds an MlflowClient to one run id.
    Writes are truncated and best effort so observability never fails the run it
    describes; terminate() writes once.
  • _run_execution and both workflows (author, plan_execute) drop the fluent
    API. A run handed in by mcp_svc is terminated by mcp_svc; a run a workflow
    mints itself is terminated by that workflow.
  • hook.py runs a boot-time sweep reconciling any RUNNING run absent from the
    live cache to KILLED, which also clears existing residue.
  • History Model column now has a model param to read, and Result reads the
    process_result tag instead of a param nothing ever wrote.
  • History view renders KILLED and shows - for its duration, since the sweep
    timestamp is when we noticed, not when the run stopped.

Two side effects worth flagging: DSPy autolog traces are no longer attached to a
run (they still land in the experiment), and the sweep assumes one tracking
server per deployment, which is what hook.py already provisions from yaml.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

17 new tests across tests/test_mlflow_run_lifecycle.py,
tests/test_run_execution_lifecycle.py, and tests/test_author_guards.py,
covering overlapping runs, failure attribution, phantom-run absence, run
ownership, and the orphan sweep. Each was confirmed to fail against pre-fix code
and pass after. Verified against mlflow 3.15.1 and dspy 3.3.0.

Full plugin suite compared before and after in the same environment: no new
failures, 17 new passes. Remaining suite failures are pre-existing and
environmental (missing CTI extras, live-Caldera-dependent modules).

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works

mlflow's fluent active-run stack is thread-local and every concurrent
request shares it, so overlapping runs ended and retagged each other and
phantom runs appeared whenever the stack was empty. Bind every write to
its own run id, terminate each run exactly once, sweep runs stranded
RUNNING by a dead process at boot, and read Model and Result from where
they are actually written.

Closes mitre#29
@HackedRico
HackedRico force-pushed the fix/mlflow-run-lifecycle branch from 08474ad to 2671589 Compare August 28, 2026 12:31
@HackedRico
HackedRico merged commit 172ad66 into mitre:main Aug 28, 2026
3 checks passed
@HackedRico
HackedRico deleted the fix/mlflow-run-lifecycle branch August 28, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run History shows completed and dead runs as RUNNING, plus phantom "No prompt" rows

1 participant