build: uv migration

[MattiaSarti]

Description

Purpose

This pull request addesses this issue by proposing a way of migrating to uv (for Python dependency and environment management) that is natively compatible with the ecosystem of tools already employed - namely charmcraft and tox.

Dependency Maintainability

Instead of combining .in and .txt requiements files manually for different environment via pip and derived tools such as pip-compile, uv manages dependencies for different environments in a unified way and allows to combine dependencies for some environments from others neatly. As a result, manual scripts to combine environment dependencies by compiling and orderly assembling different .in and .txt requiements files are no longer necessary.

Requirements are all listed in pyproject.toml across different extras (groups) than can be used as building blocks to combine all dependencies required by each environment (e.g.: the unit testing environment may define its dependencies as the union of the charm dependencies and some extra dependencies specific to unit tests only). Defining also the charm dependencies as such extras, instead of as mere project dependencies, allows such maintainable logic to use charm dependencies too as building blocks.

By default, uv does not allow for any dependency conflicts across environment building blocks. If you wanted to, though, uv could be configured to ignore such conflicts - as far as the conflicting blocks are never combined to build environments.

uv is adopted not only to manage dependencies of the charm itself, thanks to the uv plugin of charmcraft and its keywords to specify extras/groups, but also to manage dependencies for all the environments of the CI process with tox.

tox Environments

In fact, tox can work by referring to the environments defined in pyproject.toml. Nevertheless, while this achieves defining the dependencies of the different tox environmentsin a unified way with uv, it does not imply that the dependencies are actually built and run via uv when running tox. As a matter of fact, tox ends up internally using pip for that by default (as confirmed by old test run logs). To accomplish this latter purpose, the tox-uv plugin (a tool maintained by the official tox team) was employed, making it possible for tox to internally rely on uv for real.

An important remark: the reason why dependencies are specified by combining extras instead of dependency groups, despite the uv plugin of charmcraft allowing for both, is that the tox-uv plugin requires extras to work.

Building from Source

Also, uv can be configured to build packages from source either via pyproject.toml or via CLI. For the former option, mind the required workarounds for this temporary issue. Nevertheless, as mentioned in the same issue discussion, the latter option will soon be added to the uv plugin of charmcraft, so that charms will be made to build from source. Although this pull request does not enable any such options for building from source via charmcraft, it was locally ensured that the uv setup allows to build environment dependencies in both modes - see the test section below. Yet, even in the future, it would be beneficial not to build from source the environments run via tox or any other local command directly executed by uv, to make development and testing lighter and faster - without building dependencies and without requiring other dependencies for the build process in turn. Ideally, only the charm dependencies would be built from source and only when built via charmcraft.

Renovate

Moreover, uv is supported by Renovate for automated dependency scanning/updating, in particular via the extras of [project.optional-dependencies] that are so relevant to our use case, so the inherited Renovate configurations from upstream repositories can be updated accordingly.

Instructions for Adoption

uv is the only tool required locally.

Updating Dependencies and/or Python

To add/update/remove any dependencies and/or to upgrade Python, simply:

1. first add/update/remove such dependencies to/in/from the desired group(s) below [project.optional-dependencies] ("extras") in pyproject.toml, and/or upgrade Python itself in .python-version
   ⚠️ dependencies for the charm itself are also defined below [project.optional-dependencies] as extras, specifically in the charm section, and not below [project] as project dependencies (see above why) ⚠️
2. then run:
   • either uv lock to just update your lock file
   • or alternatively uv sync --extra <your-extra-a> --extra <your-extra-b> (or uv sync --all-extras) if you also want to update your local environment together with your lock file, so that you will be able to run Python code from your uv environment locally using uv run python3 <whatever>

By point 2., uv will let you know if there are any dependency conflicts to solve.

Running tox

To run tox environments locally, just:

1. install tox as an uv tool together with the required tox-uv plugin: uv tool install tox --with tox-uv
2. run tox as you would natively (e.g.: tox -e lint)

How Were Changes Tested?

All tests were run during CI and successfully passed. Test steps were modified to install and use uv in place of the existing dependency management solutions together with its tox plugin. To this end, some inherited tests had to be modified upstream and were tested connecting to the branch of the respective pull request.

Moreover, manual tests were locally executed by performing the following steps on an Ubuntu 24.04 machine and in the root folder of this repository checked out at the last commit of this branch:

Charm Test

test details

Notes:

The following commands aimed at:

• validating the charm build and deployment
• assessing that the charm environment was not polluted by dependencies of other environments
• checking the deployed charm allowed for operations on expected custom resource definitions

Commands:

juju bootstrap microk8s local-k8s
juju add-model demo

charmcraft pack
juju deploy --trust --resource oci-image=charmedkubeflow/pvcviewer-controller:1.10.0-09b616f ./pvcviewer-operator_ubuntu@24.04-amd64.charm

kubectl exec -it -n demo -c charm pvcviewer-operator-0 -- bash
# ls /var/lib/juju/agents/unit-pvcviewer-operator-0/charm/venv/lib/python3.12/site-packages
# source /var/lib/juju/agents/unit-pvcviewer-operator-0/charm/venv/bin/activate
# which python
# python -c "import ops"  # -> this runs successfully as "ops" is a requirement for the charm
# python -c "import cosl"  # -> this runs successfully as "cosl" is a requirement for the charm
# python -c "import black"  # -> this fails as "black" is a requirement for linting tests only, not for the charm
# exit

cat <<EOF | kubectl apply -n demo -f -
apiVersion: kubeflow.org/v1alpha1
kind: PVCViewer
metadata:
  name: pvcviewer-sample
  namespace: demo
spec:
  pvc: pvcviewer-sample
  networking:
    targetPort: 8080
    basePrefix: "/pvcviewer"
    rewrite: "/"
  rwoScheduling: true
EOF

kubectl get -n demo pvcviewers.kubeflow.org
kubectl describe pvcviewers.kubeflow.org -n demo pvcviewer-sample

uv Build Test: From Source and Not

test details

Notes:

The following commands were executed both with no-binary = true and with no-binary = false in pyproject.toml, confirming uv can successfully build the project environment(s) both from source, as charmcraft will automatically require with future releases of its uv plugin, and not from source, as currently coded and for lightweight development and testing.

Commands:

# the following build dependencies are required only when building environments from source:
sudo apt-get install -y libffi-dev libssl-dev pkg-config rustup build-essential python3-dev
rustup set profile minimal
rustup default 1.83.0

uv sync --all-extras

[mvlassis]

Let's not forget to update this to the main branch once canonical/charmed-kubeflow-workflows#110 is merged

[MattiaSarti]

Yes, let's keep this comment conversation as unresolved in order not to forget.

[mvlassis]

Excellent job with the exploration @MattiaSarti! Very thorough, and much more compact since now everything will be in a single file.

We could potentially add a CONTRIBUTING.md file that includes the information you have already written for using uv to update dependencies, and run tox environments. here is an example in the kfp-operators repo. Not sure whether this is worth the effort though, as using uv is quite straightforward.

[MattiaSarti]

Excellent job with the exploration @MattiaSarti! Very thorough, and much more compact since now everything will be in a single file.

We could potentially add a CONTRIBUTING.md file that includes the information you have already written for using uv to update dependencies, and run tox environments. here is an example in the kfp-operators repo. Not sure whether this is worth the effort though, as using uv is quite straightforward.

Thank you so much! 🥹
I have just added those instructions into such a file.