Update MLOps Stacks to support MLflow 3.0

[davidzuma]

With the release of MLflow 3.0, the platform has shifted from a run-centric approach to a model-centric architecture. To ensure Databricks MLOps Stacks remain the "gold standard" for production best practices, the templates should be updated to leverage these new capabilities.

Proposed Changes:

• Update ML Code Templates: Refactor training notebooks to utilize the new LoggedModel entity. Specifically, move away from artifact_path in favor of the name parameter in log_model() to enable easier model searching.

• Modernize Evaluation: Replace any calls to the deprecated mlflow.evaluate with the specialized mlflow.models.evaluate or mlflow.genai.evaluate APIs.

• Streamline UC Integration: Since MLflow 3.0 defaults to databricks-uc, update the databricks.yml and resource configurations to assume Unity Catalog as the primary registry, simplifying the three-level namespace logic.

• Dependency Alignment: Update requirements.txt and cluster environment definitions in the stack to target mlflow>=3.0.0.

• Remove Legacy Refs: Ensure no references to mlflow.recipes (now removed) remain in the codebase.

Value: Updating to 3.0 will provide users of MLOps Stacks with better lineage tracking, native evaluation support, and a more intuitive model-first development loop.

[ShaylanDias]

Thanks for raising this issue @davidzuma. These are are great suggestions for creating parity between MLOps Stacks and MLflow 3.0. We are discussing how to get these efforts staffed. In the meantime if any of these would provide you immediate value please feel free to take a stab at them and file a pull request.