Replace Docker with uv/pixi for dependency management

[shntnu]

Docker adds unnecessary complexity for what's primarily Python dependency management. It creates friction in the development process with longer build times and steeper learning curves.

Recommendation

1. Use uv - Fast Python package installer and virtual environment manager
2. Or pixi - If uv doesn't meet needs (better GPU/scientific package support)

Docker could still be used for Hail deployment, but for local development and testing, UV/PIXI would significantly improve developer experience.

Happy to create a PR implementing this approach if helpful.

cc: @afermg

microscopy_computational_tools/Dockerfile

Lines 1 to 37 in a6aedf8

	FROM 'debian:12-slim'
	ARG JXL_VERSION="0.11.1"
	RUN apt update && \
	# tools: python3, python3-pip, git, curl, moreutils (has parallel)
	# dependencies for Pillow, opencv and libjxl: libtiff-dev zlib1g-dev libhwy-dev libgl1 libglib2.0-0
	apt install -y python3 python3-pip git curl moreutils libtiff-dev zlib1g-dev libhwy-dev libgl1 libglib2.0-0 && \
	curl -L --output jxl-linux-x86_64-static.tar.gz https://github.com/libjxl/libjxl/releases/download/v${JXL_VERSION}/jxl-linux-x86_64-static-v${JXL_VERSION}.tar.gz && \
	tar -zxvf jxl-linux-x86_64-static.tar.gz && \
	mv tools/* /usr/bin && \
	curl -L --output jxl-debs-amd64-debian-bookworm.tar.gz https://github.com/libjxl/libjxl/releases/download/v${JXL_VERSION}/jxl-debs-amd64-debian-bookworm-v${JXL_VERSION}.tar.gz && \
	tar -zxf jxl-debs-amd64-debian-bookworm.tar.gz && \
	apt install -y ./libjxl_${JXL_VERSION}_amd64.deb ./libjxl-dev_${JXL_VERSION}_amd64.deb && \
	rm /usr/lib/python*/EXTERNALLY-MANAGED && \
	python3 -m pip install --no-cache-dir --upgrade pip && \
	# torch in the debian12 repo is too old for cellpose
	# CP-CNN requires efficientnet and tensorflow
	# install all packages via pip (vs apt) since pip does not install torch if an outdated sympy is installed by apt
	# pinning to TF 2.17 due to https://github.com/tensorflow/tensorflow/issues/78784 (current version is 2.18.0)
	pip3 install --no-cache-dir torch pandas numpy scipy numba efficientnet tensorflow==2.17.1 tensorflow-cpu==2.17.1 git+https://github.com/olokelo/Pillow.git@jxl-support2 && \
	# dependencies for cellpose
	pip3 install --no-cache-dir tqdm opencv-python tifffile fastremap natsort roifile && \
	git clone https://github.com/MouseLand/cellpose.git && \
	mv cellpose/cellpose /usr/local/lib/python3.11/dist-packages/ && \
	rm -rf jxl-debs-amd64-debian-bookworm.tar.gz jxl-linux-x86_64-static.tar.gz tools LICENSE.* /var/lib/apt/lists/* *jxl*.deb cellpose
	# the nvidia-container-toolkit allows using the GPU on Hail Batch container without installing the full driver
	RUN curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && \
	curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
	sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
	tee /etc/apt/sources.list.d/nvidia-container-toolkit.list && \
	apt update && \
	apt install -y nvidia-container-toolkit && \
	rm -rf /var/lib/apt/lists/*
	COPY cellpose /scripts/cellpose
	COPY embeddings /scripts/embeddings

[shntnu]

h/t to @gnodar01 who said this:

Use pixi if there is something on conda-forge (or some conda channel) that is not available on PyPI, otherwise use uv.
Since this is used in Docker, there is one major advantage of using uv over pixi. I've logged an issue here: prefix-dev/pixi#1981
uv can compile python after installation, whereas pixi doesn't. This is something that you can do manually instead. Here is one example where we use pixi in a docker image, and then manually compile as a second step: https://github.com/bilayer-containers/bilayers/blob/main/src/algorithms/instanseg_inference/Dockerfile#L25-L26

@bgorissen said:

The Docker file is tailored to a Hail Batch deployment. Any package manager can probably work, even the built-in venv, but it'd be nice if one could automatically get the jpeg-xl libraries. If pixi or uv can't do that, do you still prefer them over venv?

---

I forgot the complexity that jpeg-xl introduces.

I was thinking that for local testing, it's ok not to use jpeg-xl, and if we skip that, uv seems sufficient. I was able to get it to run on my OSX and generate embeddings

See main...shntnu:microscopy_computational_tools:main

I get the output I'd expect (at least at the high level) which is great! Because this allows us to test stuff out locally

tree output 
output
├── cellpose_0_1.csv
├── crops.png
└── embedding.tsv

head output/cellpose_0_1.csv|cut -c-50 
file    i       j
r01c01f01p01-ch5sk1fk1fl1.tiff  [19, 4, 19, 27, 55,
r01c01f02p01-ch5sk1fk1fl1.tiff  [21, 8, 8, 11, 11, 
r01c01f03p01-ch5sk1fk1fl1.tiff  [15, 8, 8, 11, 27, 
r01c01f04p01-ch5sk1fk1fl1.tiff  [13, 15, 6, 6, 21, 

ead output/embedding.tsv|cut -c-50 
file    i       j       embedding
r01c01f01p01-ch5sk1fk1fl1.tiff  19      192     ['1.063e+00'
r01c01f01p01-ch5sk1fk1fl1.tiff  4       409     ['5.953e-01',
r01c01f01p01-ch5sk1fk1fl1.tiff  19      146     ['1.100e+00'
r01c01f01p01-ch5sk1fk1fl1.tiff  27      996     ['5.796e-01'
r01c01f01p01-ch5sk1fk1fl1.tiff  55      255     ['7.424e-01'
r01c01f01p01-ch5sk1fk1fl1.tiff  57      304     ['6.304e-01'
r01c01f01p01-ch5sk1fk1fl1.tiff  55      896     ['5.049e-01'
r01c01f01p01-ch5sk1fk1fl1.tiff  59      399     ['8.784e-01'
r01c01f01p01-ch5sk1fk1fl1.tiff  78      118     ['1.117e+00'

I'll stop here for now and report back if we come up with ideas on how to make this more streamlined.

[gnodar01]

I was thinking that for local testing, it's ok not to use jpeg-xl,

FYI: You can get jpeg-xl via conda-forge, and therefore pixi.

[shntnu]

I was thinking that for local testing, it's ok not to use jpeg-xl,

FYI: You can get jpeg-xl via conda-forge, and therefore pixi.

Neat! Then I think what's remaining is to be able to install Pillow from a fork https://github.com/olokelo/Pillow/tree/jxl-support2 and I suppose that's possible too?

[gnodar01]

Yup

You would just add this to the toml under pypi-dependencies:

pillow = { git = "https://github.com/olokelo/Pillow.git" }

OR

pillow = { git = "ssh://git@github.com:olokelo/Pillow.git" }

[shntnu]

You would just add this to the toml under pypi-dependencies:

Thank you! Over to @bgorissen to ponder all this

Meanwhile, the script in https://github.com/shntnu/microscopy_computational_tools/blob/49ff37c190f3e07df09114af1033bc8c483cd01a/example_run.sh runs fine and does everything it is supposed to (except no TF support on metal, so CPCNN is slower than it could be)

[bgorissen]

I've switched from olokelo's Pillow fork to pillow-jxl-plugin (which supports 16 bits grayscale since November). This made it easier to create the pixi.toml file. Please give it a try. The only issue I've found so far is that the cpcnn model doesn't run on the GPU. I've got a solution which I'll push in the next few days.

[bgorissen]

I've pushed another pixi.toml. It installs Torch 2.4, TF 2.17, and nvcc 12.5, which are all compatible. It lists cuda as a system requirement as otherwise it would only install cpu versions of Torch and TF, but you do not need cuda as long as you have the nvidia driver installed, since nvcc is the only dependency. If you get this error:

$ pixi run jupyter
Error: × Cannot install environment 'default'
╰─▶ Virtual package '__cuda' does not match any of the available virtual packages on your machine:
[__glibc=2.36=0, __linux=6.1.0=0, __unix=0=0, __archspec=1=haswell]
help: You can mock the virtual package by overriding the environment variable, e.g.:
'CONDA_OVERRIDE_CUDA=12.0'

you can therefore safely use the suggested override.