Ewoks install requirements should contain the python and pip version

[poautran]

In GitLab by @woutdenolf on Jun 21, 2025, 12:29 GMT+2:

Lets do it in a way that allows expanding in the future. Unfortunately the current "requirements" leaved no room of expansion.

Assignees: @woutdenolf

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoks/-/issues/41

[poautran]

In GitLab by @woutdenolf on Jun 21, 2025, 12:38 GMT+2:

It could look something like this

graph_attrs = {
    "dependencies": {
        "schema_version": "1.0",
        "python": {
            "version": "3.11.4",
            "implementation": "CPython",
            "manager": {
                "name": "pip",
                "version": "23.1.2",
                "requirements": [
                    "requests==2.32.3",
                    "numpy==1.26.4",
                ],
            },
            # Possible in the future other package managers
            # "manager": {
            #     "name": "poetry",
            #     "version": "1.8.0",
            #     "requirements": [
            #         "flask = \"^2.3.0\"",
            #         "sqlalchemy = \"1.4.0\"",
            #         "tomlkit = {version=\"0.11.6\", extras=[\"some-extra\"]}",
            #     ],
            # },
        },
    }
}

Perhaps in the future we might want to expand to

# Probably the conda channels need to be here as well. Support conda channels and pypi mixing.
        "conda": {
            "version": "23.3.1",
            "packages": [
                    "python=23.3.1"
                    "numpy=1.26.4",
                    "pandas=2.0.1",
                    "scipy=1.11.1",
                ]
          }

or

        "node": {
          "version": "18.14.0",
          "packages": {
              "manager": {
                   "name": "npm",
                   "version": "9.6.6"
                   "requirements": ["express@4.18.2", "lodash@4.17.21"]
          }

What do you think @loichuder @poautran @payno @LudoBroche ?

[poautran]

In GitLab by @woutdenolf on Jun 21, 2025, 12:46 GMT+2:

Example

import platform
import subprocess

def get_environment_snapshot() -> dict:
    python_version = platform.python_version()
    python_impl = platform.python_implementation()

    pip_version_output = subprocess.check_output(["pip", "--version"], text=True)
    pip_version = pip_version_output.split()[1]

    freeze_output = subprocess.check_output(["pip", "freeze"], text=True)
    requirements = [line.strip() for line in freeze_output.splitlines() if line.strip()]

    return {
        "python": {
            "version": python_version,
            "implementation": python_impl,
            "manager": {
                "name": "pip",
                "version": pip_version,
                "requirements": requirements,
            },
         },
    }

[poautran]

In GitLab by @payno on Jun 23, 2025, 09:04 GMT+2:

This is a complex question.
While this approach might be the most convenient when I see that you could be looking for the python_implementation (which make sense) I wonder what about CPU architecture ? or the OS ? ...

So for me the question behind is: what should be the threshold of such a feature ?

IMO storing the result of a pip list --verbose might be enough. It list the 'installer' in the last column.

(/cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable) slurm-nice-devel2903:~ % pip list --verbose
Package                   Version        Location                                                                                                          Installer
------------------------- -------------- ----------------------------------------------------------------------------------------------------------------- ---------
alembic                   1.16.1         /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages pip
algotom                   1.6.0          /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages pip
anyio                     4.9.0          /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages conda
AnyQt                     0.2.1          /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages pip
argon2-cffi               25.1.0         /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages conda
argon2-cffi-bindings      21.2.0         /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages conda
arrow                     1.3.0          /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages conda
astra-toolbox             2.2.0          /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages pip
asttokens                 3.0.0          /cvmfs/tomo.esrf.fr/software/packages/ubuntu20.04/x86_64/tomotools/2025_06_12_stable/lib/python3.11/site-packages conda
...

[poautran]

In GitLab by @loichuder on Jun 23, 2025, 09:47 GMT+2:

I seem to remember one of the package manager (poetry, uv or hatch) did something similar.

We should try to stick to an existing standard if possible.

[poautran]

In GitLab by @loichuder on Jul 23, 2025, 15:25 GMT+2:

created branch 41-ewoks-install-requirements-should-contain-the-python-and-pip-version to address this issue

[poautran]

In GitLab by @woutdenolf on Nov 6, 2025, 10:28 GMT+1:

created branch 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 to address this issue

[poautran]

In GitLab by @woutdenolf on Jan 26, 2026, 09:45 GMT+1:

In addition, pip freeze is still very unreliable for producing requirements that are reproducible on another machine

# Editable Git install with no remote (ewoksfluo==0.6.0)
-e /users/blissadm/local/automation/dau_id16b/ewoksfluo

[poautran]

In GitLab by @woutdenolf on Jan 26, 2026, 11:14 GMT+1:

Examples

PyPi pip install

pip install ewoksid31

Path: /users/denolf/.local/wdnbashutils/virtualenvs/ewoks/lib/python3.12/site-packages/ewoksid31-0.7.1.dist-info
{}
Installer: pip
PyPi: ewoksid31==0.7.1
URL: None

Git URL pip install

pip install git+https://gitlab.esrf.fr/workflow/ewoks/ewoksdask.git

Path: /users/denolf/.local/wdnbashutils/virtualenvs/ewoks/lib/python3.12/site-packages/ewoksdask-2.0.1.dist-info
{'url': 'https://gitlab.esrf.fr/workflow/ewoks/ewoksdask.git',
 'vcs_info': {'commit_id': '81f2c2410fd7efa0781f0441f61321ba04c48bba',
              'vcs': 'git'}}
Installer: pip
PyPi: ewoksdask==2.0.1
URL: ewoksdask @ git+https://gitlab.esrf.fr/workflow/ewoks/ewoksdask.git@81f2c2410fd7efa0781f0441f61321ba04c48bba

Git local pip install

pip install -e /home/denolf/dev/pyslurmutils

Path: /users/denolf/.local/wdnbashutils/virtualenvs/ewoks/lib/python3.12/site-packages/pyslurmutils-1.3.0.dist-info
{'dir_info': {'editable': True}, 'url': 'file:///home/denolf/dev/pyslurmutils'}
Installer: pip
PyPi: pyslurmutils==1.3.0
URL: pyslurmutils @ git+https://gitlab.esrf.fr/workflow/pyslurmutils.git@40dc876b848744b1d35955b4ee833fe1b2dc3d59

Archive URL pip install

pip install https://redirect.github.com/silx-kit/silx/archive/refs/tags/v2.2.2.zip

Path: /users/denolf/.local/wdnbashutils/virtualenvs/ewoks/lib/python3.12/site-packages/silx-2.2.2.dist-info
{'archive_info': {'hash': 'sha256=f18af1891e31d0e827e5b046910b3e0419ab245b667f8a1c083498938bec5ae9',
                  'hashes': {'sha256': 'f18af1891e31d0e827e5b046910b3e0419ab245b667f8a1c083498938bec5ae9'}},
 'url': 'https://redirect.github.com/silx-kit/silx/archive/refs/tags/v2.2.2.zip'}
Installer: pip
PyPi: silx==2.2.2
URL: silx @ https://redirect.github.com/silx-kit/silx/archive/refs/tags/v2.2.2.zip --hash=sha256:f18af1891e31d0e827e5b046910b3e0419ab245b667f8a1c083498938bec5ae9

Conda install

mamba install -c conda-forge scipy==1.15.2

Path: /users/denolf/.local/wdnbashutils/virtualenvs/bliss_oda/lib/python3.10/site-packages/scipy-1.15.2.dist-info
{'archive_info': {'hash': 'sha256=9e52bad6c3294d1a5b04a13632118ca2157130603c6c018c2d710162b223b27e',
                  'hashes': {'sha256': '9e52bad6c3294d1a5b04a13632118ca2157130603c6c018c2d710162b223b27e'}},
 'url': 'file:///home/conda/feedstock_root/build_artifacts/scipy-split_1739790642651/work/dist/scipy-1.15.2-cp310-cp310-linux_x86_64.whl'}
Installer: conda
PyPi: scipy==1.15.2
URL: None

mamba install -c conda-forge orange-widget-base==4.26.0

Path: /users/denolf/.local/wdnbashutils/virtualenvs/bliss_oda/lib/python3.10/site-packages/orange_widget_base-4.26.0.dist-info
{'dir_info': {},
 'url': 'file:///home/conda/feedstock_root/build_artifacts/orange-widget-base_1749818696717/work'}
Installer: conda
PyPi: orange-widget-base==4.26.0
URL: None

Pixi install

pixi add ewoksdask

Path: /home/denolf/.local/wdnbashutils/virtualenvs/pixibox_2icfiu/.pixi/envs/default/lib/python3.12/site-packages/ewoksdask-2.0.1.dist-info
{}
Installer: uv-pixi
PyPi: ewoksdask==2.0.1
URL: None

Git local pixi install

Could not find a way to do it from the terminal so directly edited the pixi.toml file

[pypi-dependencies]
ewoks = { path = "/users/denolf/dev/ewoks", editable = true}

Seems to appear twice

Name: ewoks
Path: /home/denolf/.local/wdnbashutils/virtualenvs/pixibox_2icfiu/.pixi/envs/default/lib/python3.12/site-packages/ewoks-4.0.0.dist-info
{'dir_info': {'editable': True}, 'url': 'file:///home/denolf/dev/ewoks'}
Installer: uv-pixi
PyPi: ewoks==4.0.0
URL: ewoks @ git+https://gitlab.esrf.fr/workflow/ewoks/ewoks.git@67e10ad24797748d1747d55045c92c61e2dada56

Name: ewoks
Path: /home/denolf/dev/ewoks/src/ewoks.egg-info
{}
Installer: None
PyPi: ewoks==4.0.0
URL: None