runtime_env pins app venvs to the declared floor, not to the installed version

[nilsmechtel]

Repo: aicell-lab/bioengine
Version: 0.11.19 (ref 4e73d9d)
Component: bioengine/utils/requirements.py

Summary

normalize_requirement rewrites >= to ==, so a dependency declared
hypha-rpc>=0.21.40 is injected into every app's runtime_env.pip list as
hypha-rpc==0.21.40. That is the floor from the specifier, not the version
the worker has installed. When pip resolved the floor to a newer release at
image build time, the two differ, and Ray pip-installs the older version into a
per-app virtualenv where it shadows system site-packages.

The docstring already states the intended behaviour:

so the runtime_env install resolves deterministically to the same version the
driver has

Reading the specifier does not achieve that. Reading
importlib.metadata.version() does.

Reproduction

Worker image with bioengine 0.11.19 installed via
pip install "bioengine[datasets,worker] @ git+...", which resolves
hypha-rpc>=0.21.40 to 0.21.46:

$ python -c "import hypha_rpc; print(hypha_rpc.__version__)"
0.21.46

$ python -c "from bioengine.utils.requirements import get_pip_requirements; \
    print(get_pip_requirements(select=['aiortc','httpx','hypha-rpc','pydantic'], extras=['worker']))"
['hypha-rpc==0.21.40', 'aiortc==1.14.0', 'pydantic==2.12.0']

Deploy any app, then look at what Ray actually installed:

$ ls .../runtime_resources/pip/<hash>/virtualenv/lib/python3.11/site-packages | grep dist-info
hypha_rpc-0.21.40.dist-info
pip-26.2.1.dist-info
setuptools-84.0.0.dist-info

$ grep include-system-site-packages .../virtualenv/pyvenv.cfg
include-system-site-packages = true

aiortc and pydantic are absent from the venv because their declarations
(==1.14.0, ~=2.12.0) have floor equal to installed, so pip sees them already
satisfied from system site-packages and skips them. hypha_rpc is the only one
that gets a private copy, and only because it is the only one declared with
>=.

So the isolation is not intentional. It is an artefact of which operator each
dependency happened to be declared with.

Impact

1. Worker and apps run different versions of the RPC layer. The worker
   process uses hypha_rpc 0.21.46, every app uses 0.21.40. Any behaviour change
   between those releases becomes a cross-process inconsistency that is not
   visible in any manifest.

2. Local modifications to system site-packages are silently discarded for
   >=-declared packages. We hit this directly. We carry local fixes to
   hypha_rpc's WebRTC path in our image. They were verified present in
   /opt/conda/lib/python3.11/site-packages/hypha_rpc/, and had no effect,
   because every app was importing an unpatched 0.21.40 from its venv. The
   symptom was a federated round dying on the first WebRTC weight transfer with

   ProxyDeployment._create_deployment_function.<locals>.deployment_function()
       missing 1 required keyword-only argument: 'context'
   

   raised from the venv's rpc.py, which is the only reason we found it.

3. Unnecessary cold-start cost. Ray builds a virtualenv and runs a pip
   install per app for a package that is already importable.

Suggested fix

Resolve the pin from the installed distribution rather than from the specifier
text:

name = split_re.split(requirement, maxsplit=1)[0].strip()
base = name.split("[", 1)[0].strip()
if base:
    try:
        return f"{name}=={md.version(base)}"
    except md.PackageNotFoundError:
        pass
# existing >=/<=/~= string rewrite as the fallback

md is already imported in the module. Falling through when the distribution is
not installed keeps a requirement naming something absent behaving as it does
today rather than raising during a deploy.

With this applied on our image, the app venv contains only pip and setuptools:
every bioengine-injected pin is satisfied by system site-packages, nothing is
installed, and worker and apps run the same code.

The TODO: Use lock files instead of modified version ranges at the top of the
module points at the same problem from a different angle. Either direction fixes
it. Resolving from importlib.metadata is the smaller change and needs no new
artefact.

Related

Separate report: pinned <workspace>/<client_id>:<service_id>@<app_id> lookups
intermittently 404 in the proxy health check, cascading into replica restarts.