Install unisolated packages into the isolated build venv and overlay their cross-build files by ryanking13 · Pull Request #21 · pyodide/pyodide-build

ryanking13 · 2024-08-21T13:30:21Z

This changes how we handle unisolated packages and the hostsitepackages directory.

The current mechanism of unisolated packages works as follows:

We have a small number of unisolated packages. NumPy and SciPy are the most important ones.
These unisolated packages are often used as build dependencies for other packages.
When a package lists one of these as a build dependency, we intercept the call that installs build dependencies and exclude that package from being installed.
Instead, we point PYTHONPATH at the WASM build of the unisolated package, so it gets used in place of the native version when building.

This mechanism caused a few issues, such as the one discussed in pyodide/pyodide#5012 (comment). In particular, build time scripts that these packages ship, like f2py and numpy-config from NumPy, never actually made it into the isolated build environment, so they weren't available on PATH during the build.

This PR changes it in the following way:

Do not exclude unisolated packages when installing build dependencies.
Instead, install the pinned cross-build versions of the packages like any other dependency, then replace only its cross-build files (headers, static libraries, .pxd files) with the WASM-compatible versions by grabbing them from the cross-build environment.

We keep the existing mechanism that installs cross-build packages into HOSTSITEPACKAGES. PYTHONPATH is no longer pointed at it, though, because Python-level access during the build (such as importing numpy, calling numpy.get_include(), running f2py or numpy-config, and so on) now comes straight from the isolated venv.

Some more info on why we kept HOSTSITEPACKAGES around is noted below, in the "Testing" section.

[EDIT: reverted this one, will split it up into its own PR]: Another change is that oldest-supported-numpy is no longer silently ignored when it appears as a build dependency and instead raises an error. It has been deprecated since NumPy 2.0, so packages that still depend on it should move to a plain numpy dependency. If this causes trouble for a specific recipe, it can still be ignored by setting ignored_build_requirements to "patchelf oldest-supported-numpy"` of course, but IMO we don't need to bother about it anymore. In pyodide/pyodide-recipes#591, only Fiona was using it. I updated its version to one that no longer uses it, and things are working fine there.

Regarding the PKG_CONFIG_LIBDIR change to find numpy.pc: this is a new issue, as SciPy 1.17.1 used to not require numpy explicitly (https://github.com/scipy/scipy/blob/527eb7fd7953a1de068f94bf8b322f249b9405ae/scipy/meson.build#L37), but when I merged the PR for SciPy 1.18.0 (pyodide/pyodide-recipes#586), it has made numpy a hard requirement: https://github.com/scipy/scipy/blob/54ef5423f2e4376230ec3bfda6912a07a50958e3/scipy/meson.build#L34 (and scipy/scipy#24119 (comment)). We use this environment variable to scan for NumPy's pkg-config file and add it to the build. PKG_CONFIG_LIBDIR is not additive as such, but we respect it if it's already set by a package. My reason for doing so is noted here: scipy/scipy#24119 (comment).

This also adds a numpy-scripts-example integration test recipe. I wrote a small extension module built with meson-python that relies on numpy-config and f2py being on PATH to exercise the new mechanism. We don't compile anything Fortran-related.

Testing

This was tested in pyodide/pyodide-recipes#591, where all 325 package builds and tests passed.

The first run turned up an issue with building RobotRaconteur, and opencv-python would have had the same issue. Their builds are CMake-based, and they need direct access to NumPy's ndarrayobject.h and other headers in numpy/_core/include/numpy/, which used to work because PYTHONPATH pointed to a WASM-patched NumPy install. With that mechanism gone, those headers were nowhere to be found, and the builds failed.

A tempting fix for us would have been to add the entire numpy/_core/include/numpy/ tree (which contains roughly 20 header files, plus the libdivide and random folders) to NumPy's cross-build-files. I decided against doing that, because those files change often between versions and more importantly are not really platform-specific, so listing them would be a maintenance burden and would stretch what our cross-build-files field is meant for (as I noted in pyodide/pyodide-recipes#591 (comment)). Instead, we kept the existing step that installs cross-build packages into HOSTSITEPACKAGES, so packages that access these files directly from the filesystem, rather than through the isolated build venv, continue to work exactly as before. I expected more packages to have these issues, but only RobotRaconteur and opencv-python did. However, more packages which are based on C/C++ libraries with Python bindings exist in the wild (such as CasADi, which I maintain, though it doesn't need NumPy headers). We need to support such types of builds for packages for better adoption of Pyodide's build machinery.

With that in place, everything builds successfully!

for more information, see https://pre-commit.ci

…build into no-host-site

for more information, see https://pre-commit.ci

hoodmane · 2024-10-14T12:11:13Z

Sounds good to me. Relying on PYTHONPATH being around does indeed seem like it could cause trouble.

hoodmane · 2024-10-14T12:13:14Z

+    return libdir, [
+        str(f.relative_to(libdir)) for f in package_dir.rglob("*") if f.is_file()
+    ]


So this is trying to get cross-build-files? How does it work in tree? Shouldn't we look at cross-build-files in meta.yaml in that case?

Also, site-packages-extras maybe should have been called site-packages-cross-files or something a bit more descriptive...

So this is trying to get cross-build-files? How does it work in tree? Shouldn't we look at cross-build-files in meta.yaml in that case?

As you can see in the buildpkg.py change, only cross-build-files are installed under the hostsitepackages on in-tree build (I think there are some room to optimize this logic though).

Also, site-packages-extras maybe should have been called site-packages-cross-files or something a bit more descriptive...

Totally agree, but the folder site-packages-extras is generated when creating the xbuildenv, so we'll need to fix there too.

site-packages-extras is well baked into the xbuildenv tarballs at this point. site-packages-cross-files would break our backwards compatibility, which we set up after this PR was created (in #167) and have surprisingly managed to maintain it well at this point. I am not opposed to changing things, but since it's separate, I guess it should be out of scope here.

hoodmane · 2024-10-14T12:16:27Z

        assert (tmp_path / version / ".installed").exists()
        assert manager.current_version == version

-    def test_install_cross_build_packages(


It feels like we could use an updated version of this test?

This comment should no longer be applicable, as the test is back again.

We also now have test_install_reqs and test_replace_unisolated_packages in test_pypabuild.py if you want to take a look.

hoodmane

Thanks @ryanking13! I made some comments but it generally looks good.

for more information, see https://pre-commit.ci

…tion]

hoodmane · 2026-06-27T10:56:27Z

+    if in_xbuildenv():
+        unisolated_packages_file = PYODIDE_ROOT / ".." / "requirements.txt"
+
+        if unisolated_packages_file.exists():


Let's assert that it exists?

I am not fond of using assert statements outside tests, but yes, we can use a FileNotFoundError instead, assuming that's close to what you suggested.

hoodmane · 2026-06-27T13:04:13Z

+    sitepackagesdir = Path(purelib)
+    for name in unisolated:
+        package_dir = get_cross_build_files_dir(name)
+        if not package_dir.is_dir():


Why would this happen? Shouldn't we assert that it exists?

Not every unisolated package has cross-build files. The package may only need to be pinned to the cross-build version (for its console scripts, for example) without any file overlay. numpy and scipy have cross-build files to be copied, but cffi and pycparser do not. Let me add a comment about this with this explanation.

Yeah, I think it is good to be extra safe here as the format of xbuildenv files can change in the future, and we can make mistakes.

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

This reverts commit a8f63ff.

We were incorrectly pulling in the oldest-supported-numpy dependency at build time despite it being marked for usage by Shapely only for Python 3.9.

agriyakhetarpal · 2026-06-27T20:38:45Z

Thanks @hoodmane and @ryanking13 for the reviews! I adjusted the CHANGELOG entries and fixed a few bugs where we were not respecting platform markers when parsing dependencies, which were exposed by raising an explicit error when encountering oldest-supported-numpy (Shapely was failing to build). I've updated pyodide/pyodide-recipes#591 and triggered one last test for the full build. We should be good to go ahead with this soon – but please feel free to take another look if you like!

agriyakhetarpal · 2026-06-27T21:14:10Z

Oops, found a problem... but it's SciPy, not us:

SciPy 1.17.1 (when the CI was green in pyodide/pyodide-recipes#591) had dependency('numpy', required: false)

SciPy 1.18.0 has dependency('numpy') and makes it explicitly required.

agriyakhetarpal · 2026-06-27T21:19:49Z

So I think we can either pass another cross file, in flight (Meson supports multiple cross files), which would look like:

[binaries]
numpy-config = '/tmp/build-env-abcdef/bin/numpy-config'
f2py = '...'

and this won't be exposed to the user (but I don't think it's an elegant solution).

Or (I chose this) we can use pkg-config and pass PKG_CONFIG_LIBDIR, since NumPy ships a .pc file. My reason for doing so is scipy/scipy#24119 (comment), but please let me know if we should ask Ralf to revisit this.

agriyakhetarpal · 2026-06-28T02:06:26Z

Updated fiona as it was using oldest-supported-numpy; hopefully there are no others. Building all recipes locally is a pain, so I am relying on CI to do this.

Edit: pyodide/pyodide-recipes#591 is all green now. Fiona was the only package using oldest-supported-numpy.

ryanking13

It looks like I cannot approve this as this was originally my PR, but you can consider this comment as my approval.

Thanks for reviving this PR!

ryanking13 · 2026-06-30T05:12:24Z

+    unisolated: set[str] = set()
+    for reqstr in reqs:
+        req = Requirement(reqstr)
+        if req.marker and not req.marker.evaluate():


could please add a comment of what not req.marker.evaulate() mean?

Yes, I'll add one. Noting it here where I encountered it: our current Shapely recipe here uses oldest-supported-numpy as a build-time dependency only for Python versions older than 3.9. So without evaluate()ing the PEP 508 markers, we would look at oldest-supported-numpy ; python_version < '3.9', find only oldest-supported-numpy and trip up, raising an error even though it shouldn't concern us..

ryanking13 · 2026-06-30T05:12:46Z

+        req = Requirement(reqstr)
+        if req.marker and not req.marker.evaluate():
+            continue
+        if canonicalize_name(req.name) == "oldest-supported-numpy":


sounds okay to me, but could you please add a comment why we are not supporting it?

The only reason for not supporting it is because @hoodmane asked 😄 #21 (comment). It is not strictly needed for this PR, though. It won't break anything with the functionality change here; it's more of a deadweight cleanup. I can split it into its own PR.

ryanking13 · 2026-06-30T05:14:24Z

+
+# So far among all packages in pyodide-recipes, only NumPy ships
+# a .pc file, but I don't want to hardcode that here as such
+def _get_unisolated_pkgconfig_dirs(venv_path: str) -> list[str]:


Makes sense to me. I am curious how numpy adds these files to PKG_CONFIG_LIBDIR during the build

Actually, NumPy's only task is to ship the .pc file. That is recognised by Meson during the dependency (dependency('numpy')) search as one of the files to look on the system for. We then point the PKG_CONFIG_LIBDIR environment variable to it ourselves.

Another way for Meson to find NumPy is to list numpy-config in the [binaries] section of the Meson cross file. Right now:

numpy-config binary missing from cross or native file, or env var undefined. Default target is not allowed for cross use numpy-config found: NO

Meson does find numpy-config, but it refuses to use it because it's a host-machine tool. Meson assumes it may contain incorrect include/library paths for WASM, and we need to alter the cross file and add numpy-config to make it not do so.

However, I felt the pkg-config approach was cleaner, and that's what was recommended, so I went ahead with it.

agriyakhetarpal · 2026-06-30T14:37:09Z

Thanks for reviewing, both of you!

ryanking13 and others added 4 commits August 19, 2024 11:03

Update cross build installation mechanism

1dc34fd

Fix symlink

ee09707

type

c806c05

[pre-commit.ci] auto fixes from pre-commit.com hooks

033e552

for more information, see https://pre-commit.ci

agriyakhetarpal mentioned this pull request Sep 17, 2024

0.29 release #31

Closed

ryanking13 and others added 4 commits September 24, 2024 12:26

Merge remote-tracking branch 'upstream/main' into no-host-site

797e605

cleanup [integration]

c8f4663

Merge branch 'no-host-site' of https://github.com/ryanking13/pyodide-…

821cef1

…build into no-host-site

[pre-commit.ci] auto fixes from pre-commit.com hooks

6521bef

for more information, see https://pre-commit.ci

ryanking13 marked this pull request as ready for review September 24, 2024 12:50

ryanking13 changed the title ~~[DRAFT] Stop excluding unisolated packages from build dependencies~~ Stop excluding unisolated packages from build dependencies Sep 24, 2024

Merge branch 'main' into no-host-site

0ec228c

hoodmane reviewed Oct 14, 2024

View reviewed changes

hoodmane approved these changes Oct 14, 2024

View reviewed changes

ryanking13 and others added 11 commits October 15, 2024 10:26

Add more test

f039816

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b49ad5

for more information, see https://pre-commit.ci

changelog

221f9ea

More integration test files [integration]

10edfb7

[pre-commit.ci] auto fixes from pre-commit.com hooks

a18357d

for more information, see https://pre-commit.ci

exclude patchfile from codespell

611ed7b

[integration]

9ec8e60

Remove integration tests from pre-commit check

f962cc4

Fix integration test [integration]

54efacc

Remove uncessery arg

2f5bc3b

Force install cross-build package even if version mismatches [integra…

0feb9aa

…tion]

ryanking13 marked this pull request as draft October 18, 2024 11:45

hoodmane mentioned this pull request Oct 22, 2024

Building and testing Python packages out of tree for Python package with C++ code produces error saying a required module (numpy) is not found. #44

Closed

agriyakhetarpal mentioned this pull request Jan 3, 2025

SciPy 1.15.0 pyodide/pyodide#5297

Closed

3 tasks

agriyakhetarpal added 4 commits June 27, 2026 06:20

Simplify Fortran package recipe

a1ea4be

Repack dummy xbuildenv without unisolated.txt

a9d4841

Update test to handle unisolated package version mismatch warning

a898657

Add line in between

53e5f91

agriyakhetarpal marked this pull request as ready for review June 27, 2026 01:16

Merge branch 'main' into no-host-site

1e82983

hoodmane reviewed Jun 27, 2026

View reviewed changes

agriyakhetarpal and others added 8 commits June 27, 2026 19:04

Explicitly reject oldest-supported-numpy

38edec2

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

Raise FileNotFoundError for non-existent xbuildenv requirements file

2c05513

Co-Authored-By: Hood Chatham <roberthoodchatham@gmail.com>

Add comment about skipping cross-build file installation

8e682f9

Shapely was using oldest-supported-numpy, bump its version

a8f63ff

Revert "Shapely was using oldest-supported-numpy, bump its version"

a5bbea2

This reverts commit a8f63ff.

Evaluate PEP 508 markers for build-time reqs

ea0f514

We were incorrectly pulling in the oldest-supported-numpy dependency at build time despite it being marked for usage by Shapely only for Python 3.9.

Same bug for removing avoided requirements

78cc998

Fix order in CHANGELOG + adjust version

86104b9

agriyakhetarpal added a commit to pyodide/pyodide-recipes that referenced this pull request Jun 27, 2026

One final test for pyodide/pyodide-build#21

8c5dcb0

Add numpy.pc through PKG_CONFIG_LIBDIR

cb3f8a8

agriyakhetarpal mentioned this pull request Jun 29, 2026

Update Fiona to drop deprecated oldest-supported-numpy usage pyodide/pyodide-recipes#608

Merged

4 tasks

ryanking13 commented Jun 30, 2026

View reviewed changes

agriyakhetarpal added 2 commits June 30, 2026 19:50

Revert oldest-supported-numpy change for now

c90969e

Add comment about PEP 508 and evaluate()

304bab0

agriyakhetarpal merged commit 9814cda into pyodide:main Jun 30, 2026
18 checks passed

This was referenced Jun 30, 2026

Explicitly reject oldest-supported-numpy as a build-time dependency #392

Merged

Use custom f2py generators for SciPy, and drop patches from six to five pyodide/pyodide-recipes#611

Merged

Uh oh!

Conversation

ryanking13 commented Aug 21, 2024 • edited by agriyakhetarpal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

hoodmane commented Oct 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hoodmane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal commented Jun 27, 2026

Uh oh!

agriyakhetarpal commented Jun 27, 2026

Uh oh!

agriyakhetarpal commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agriyakhetarpal commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanking13 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal commented Jun 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ryanking13 commented Aug 21, 2024 •

edited by agriyakhetarpal

Loading

agriyakhetarpal Jun 6, 2026 •

edited

Loading

agriyakhetarpal Jun 6, 2026 •

edited

Loading

agriyakhetarpal Jun 27, 2026 •

edited

Loading

agriyakhetarpal commented Jun 27, 2026 •

edited

Loading

agriyakhetarpal commented Jun 28, 2026 •

edited

Loading