Resolve "Ewoks install requirements should contain the python and pip version"#65

Open

poautran wants to merge 9 commits into

mainfrom

41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2

poautran commented Feb 10, 2026 •

edited by woutdenolf

Loading

Member

Closes #41

Migrated from GitLab: https://gitlab.esrf.fr/workflow/ewoks/ewoks/-/merge_requests/233

Review tip

For collapse _requirements and tests and review the rest
Then review tests
Then review the private _requirements: more details below and in github code comments

If you have experience with other package managers like uv, please check whether the package manager derived from BaseManager goes in the right direction. Only PipManager is currently enabled and tested. The others are not used, private and probably contain errors.

I think this is better to make a PR per package manager after this PR is merged.

Changes

Make a Pydantic model for graph["requirements"].
The deprecated graph["requirements"] list (pip freeze) is still parsed when doing ewoks install (converted to a dict for pip).
In addition to dependencies (what we had before) it contains other metadata such as the python version.
The model has now two lists of dependencies: one extracted from importlib.metadata and the other manage specific (like freeze for pip). This gives the installer a chance event if they do not have the same package manager.
ewoks install --python ... is replaced with the more general ewoks install --manager ...

Not supported yet but prepare for supporting other package managers in addition to pip: pixi, conda, uv, poetry, pipenv.

Private API: top-level

ewoks._requirements.__init__:

add_requirements(graph: TaskGraph) -> None: Add requirements to a workflow definition in-place
get_requirements(graph: TaskGraph) -> BaseRequirements: Extract requirements from a workflow definition.
install_requirements(requirements: BaseRequirements) -> None: Install workflow requirements.

Private API: implementation

Models

ewoks._requirements.models: the pydantic models

BaseRequirements: python version, installed distributions, ...
PipRequirements
PixiRequirements
...

Review: have a close look at the models so we avoid going through deprecation cycles because I missed obvious flaws.

Package Managers

ewoks._requirements.managers: package managers to gather requirements metadata and install requirements metadata

BaseManager
PipManager: support fallback to installing the distributions if installing the freeze list fails.
PixiManager
...

Requirements metadata

ewoks._requirements.metadata: gathering and parse requirements metadata

list of installed distributions from importlib
parse legacy pip-freeze list
generate requirements metadata from workflow node identifiers alone (ewoks install a workflow without requirements)

poautran added the gitlab merge request label

poautran assigned woutdenolf

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 6, 2025, 16:22 GMT+1:

added 1 commit

c51cf08f - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 6, 2025, 16:22 GMT+1:

added 3 commits

c51cf08f...830ab7af - 2 commits from branch main
d46af4db - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 6, 2025, 18:09 GMT+1:

added 1 commit

efe3cbaf - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 6, 2025, 18:21 GMT+1:

added 1 commit

1ddbe412 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 7, 2025, 17:01 GMT+1:

added 1 commit

932ac937 - wip

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 7, 2025, 18:26 GMT+1:

added 1 commit

3e7295e5 - wip

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Nov 8, 2025, 13:04 GMT+1:

added 1 commit

dafcd9e6 - wip

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 25, 2026, 16:23 GMT+1:

added 1 commit

d9f3cbe1 - wip

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 25, 2026, 16:26 GMT+1:

added 11 commits

d9f3cbe1...8f4a8f91 - 10 commits from branch main
3a65c2e8 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 25, 2026, 16:32 GMT+1:

added 1 commit

d3415a83 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 25, 2026, 16:37 GMT+1:

added 1 commit

1ce3da15 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 25, 2026, 16:50 GMT+1:

added 1 commit

fa70b2d7 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 25, 2026, 16:57 GMT+1:

added 1 commit

5fae8309 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 26, 2026, 08:29 GMT+1:

added 1 commit

07f1f116 - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 26, 2026, 09:15 GMT+1:

added 1 commit

df1937bc - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 26, 2026, 09:16 GMT+1:

added 1 commit

7258d94e - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran commented Feb 10, 2026

Member Author

In GitLab by @woutdenolf on Jan 26, 2026, 10:55 GMT+1:

added 1 commit

67e10ad - ewoks install: support conda, poetry and pixi

Compare with previous version (link unavailable after force-push)

poautran force-pushed the main branch 3 times, most recently from 9332091 to cfde0c6 Compare

February 10, 2026 12:48

woutdenolf force-pushed the 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 branch 5 times, most recently from 2468da5 to 99ec2e1 Compare

March 28, 2026 18:22

codecov Bot commented Mar 28, 2026 •

edited

Loading

Codecov Report

❌ Patch coverage is 36.86869% with 500 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/ewoks/_requirements/conda.py	0.00%	59 Missing ⚠️
src/ewoks/_requirements/utils/base_manager.py	55.45%	49 Missing ⚠️
src/ewoks/_requirements/pipenv.py	0.00%	46 Missing ⚠️
src/ewoks/_requirements/pixi.py	0.00%	44 Missing ⚠️
src/ewoks/_requirements/utils/pip_freeze.py	8.33%	44 Missing ⚠️
src/ewoks/_requirements/uv.py	0.00%	43 Missing ⚠️
src/ewoks/_requirements/poetry.py	0.00%	42 Missing ⚠️
src/ewoks/_requirements/__init__.py	30.90%	38 Missing ⚠️
...s/_requirements/utils/metadata/from_python/_git.py	20.00%	36 Missing ⚠️
src/ewoks/_requirements/pip.py	59.09%	27 Missing ⚠️
... and 6 more

📢 Thoughts on this report? Let us know!

woutdenolf force-pushed the 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 branch 2 times, most recently from 88b456c to 082db24 Compare

March 28, 2026 19:32

woutdenolf changed the title ~~Draft: Resolve "Ewoks install requirements should contain the python and pip version"~~ Resolve "Ewoks install requirements should contain the python and pip version"

woutdenolf requested a review from a team

March 28, 2026 20:01

woutdenolf reviewed

View reviewed changes

src/ewoks/_requirements/managers/utils/base.py Outdated

woutdenolf reviewed

View reviewed changes

src/ewoks/_requirements/models/base.py Outdated

+              class BaseRequirements(BaseModel):
+                  system: SystemInfo
+                  python: PythonInfo
+                  distributions: List[Distribution]

woutdenolf Mar 29, 2026

Member

Maybe call this python_distributions? Or put distributions in PythonInfo?

woutdenolf Apr 1, 2026

Member

We might not want the model to be too python centric although current everything happens in python.

woutdenolf requested a review from a team

April 1, 2026 15:21

loichuder reviewed

View reviewed changes

loichuder left a comment

Member

Could you explain a bit more this:

The model has now two lists of dependencies

Are both lists part of the requirements field of the workflow, now ?

When gathering requirements with ewoks convert, are both lists filled? If so, by which package manager?

And when running ewoks install on a single workflow with no requirements, what happens ? Are both lists generated ? Only one ?

src/ewoks/bindings.py

woutdenolf commented Apr 2, 2026 •

edited

Loading

Member

Could you explain a bit more this:

The model has now two lists of dependencies

Yes. We always gather the list of python distributions using importlib.metadata, regardless of the manager.

The Pip manager in addition gathers pip freeze.

The Pip manager first tried to install this pip freeze list which often goes well but can fail with editable installations (or other corners cases list dist installations), in which case the manager falls back to installing the python distributions. It does that by calling freeze_distributions which basically converts the list of distributions to a pip freeze equivalent.

For conda and pixi the second list will not be something we can fully derive from the python distributions but a similar fallback would at least give a chance of installing even if you do not have conda or pixi.

Very often we have pure python packages anyway, regardless of what package manager we use on either side (gather or install). So the fallback to deriving something from the distributions is useful.

woutdenolf requested a review from loichuder

June 4, 2026 06:39

loichuder commented Jun 4, 2026

Member

Yes. We always gather the list of python distributions using importlib.metadata, regardless of the manager.

What do you mean by "python distributions" and how is it related to Python packages?

woutdenolf commented Jun 4, 2026 •

edited

Loading

Member

What do you mean by "python distributions" and how is it related to Python packages?

A distribution (the thing you pip install) can contain several packages (the things you import).

For example ewoksxrpd has 3 packages:

ewoksxrpd
ewoksxrpd_pytest_plugin
orangecontrib/ewoksxrpd (part of the orangecontrib namespace package)

https://gitlab.esrf.fr/workflow/ewoksapps/ewoksxrpd/-/tree/main/src?ref_type=heads

Edit: sorry, your question is in the context of importlib.metadata. This does not map directly but they call it "distribution package"

https://docs.python.org/3/library/importlib.metadata.html

importlib.metadata operates on third-party distribution packages installed into Python’s site-packages directory via tools such as pip.

Edit: an example

pip install Pillow

import PIL

Distribution: Pillow
Package: PIL

loichuder approved these changes

View reviewed changes

src/ewoks/_requirements/models/distro.py Outdated

src/ewoks/_requirements/managers/utils/base.py Outdated

src/ewoks/_requirements/managers/utils/base.py Outdated

src/ewoks/_requirements/managers/utils/supported.py Outdated

src/ewoks/_requirements/utils/base_manager.py

src/ewoks/_requirements/managers/utils/detect.py Outdated

src/ewoks/_requirements/managers/utils/detect.py Outdated

src/ewoks/_requirements/managers/utils/supported.py Outdated

woutdenolf force-pushed the 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 branch from a9d0334 to bf49bdc Compare

June 4, 2026 14:22

woutdenolf added 3 commits

June 4, 2026 17:01


          ewoks install: support pixi, conda, poetry, uv and pipenv

0a33367


          merge ManagerInfo in BaseManager

e82bc4f


          fix typos

cd02d86

woutdenolf force-pushed the 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 branch from bf49bdc to cd02d86 Compare

June 4, 2026 15:01

woutdenolf added 2 commits

June 4, 2026 21:09


          use custom manager command only when targeting a specific manager

f881d06


          rearrange requirements

3878cdc

woutdenolf force-pushed the 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 branch from cd6cddb to 3878cdc Compare

June 4, 2026 19:47

woutdenolf commented Jun 4, 2026 •

edited

Loading

Member

Following your review I merged all manager specific stuff in the manager classes themselves. This was a huge refactoring so please review again.

I also harmonized the CLI

### Added

- `ewoks install`: add `--package-manager-name` and `--package-manager-command` arguments.
- `ewoks convert`: add `--package-manager-name` and `--package-manager-command` arguments.

### Removed

- `ewoks install`: remove `-p/--python` argument.

Originally I was just replacing -p/--python by -m/--manager but we need both the name and the command when we support more than pip. We still don't in this PR but the infrastructure is prepared for it so we might as well prepare the CLI for it.

woutdenolf commented Jun 4, 2026 •

edited

Loading

Member

Note that the manager classes are private API. What is public are the CLI modifications and the structure of the ewoks graph attribute requirements: list is deprecated and replaced by a pydantic model. This may need to move to ewokscore.

I just realized that since we merged everything in the modules of the manager classes, we would probably move the entire _requirements module to ewokscore, not just the models. Or perhaps we can extract the models again for that.

woutdenolf requested a review from loichuder

June 4, 2026 22:10

woutdenolf added 4 commits

June 5, 2026 00:27


          refactor manager native installation with python distribution fallback

37d089e


          ewoks convert/install arguments: package manager name and command

86b5993


          make some modules private

1c2d1de


          add distributions to last-resort requirements when installing graphs …

19dcca9

…that do not have requirements

woutdenolf force-pushed the 41-ewoks-install-requirements-should-contain-the-python-and-pip-version-2 branch from 3c8e964 to 19dcca9 Compare

June 4, 2026 22:29

loichuder reviewed

View reviewed changes

src/ewoks/_requirements/conda.py

+                  def _install_base_requirements(self, requirements: BaseRequirements) -> bool:
+                      raise NotImplementedError(f"{self.NAME} installation of python distributions")
+                  def _get_conda_command(self) -> Tuple[str, ...]:

loichuder Jun 26, 2026

Member

That seems difficult to encompass all conda executables.

I think there is also miniconda for example ?

woutdenolf Jun 26, 2026

Member

The command is always conda or mamba, except for micromamba iirc.

loichuder approved these changes

View reviewed changes

woutdenolf commented Jun 26, 2026

Member

I will wait until we release the current RC and then merge this. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gitlab merge request