Switch Python dependency management from pip to rules_pycross

[linzhp]

Problem

The current setup uses rules_python's pip.parse extension to manage Python dependencies (rules_ros2_pip_deps). This works for host builds
but breaks cross-compilation: pip.parse resolves wheels for the host platform only, so when targeting aarch64 the wrong (x86_64) wheels
are selected. This is the issue raised in #151 (comment).

Solution

Replace pip.parse with rules_pycross, which resolves and cross-compiles Python wheels per target
platform. Dependencies are now declared via a pyproject.toml/poetry.lock pair, and pycross handles platform-specific wheel selection and
building from source where needed.

empy is marked always_build = True because no pre-built aarch64 wheel is available on PyPI — pycross builds it from the sdist.

Changes

• MODULE.bazel — replace pip extension with pycross: add rules_pycross as a direct dep; replace pip.parse (→
  rules_ros2_pip_deps) with lock_import.import_poetry (→ rules_ros2_py_deps); configure cross-compilation environments for
  x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, aarch64-apple-darwin
• BUILD.bazel / WORKSPACE — remove now-unused compile_pip_requirements target; export pyproject.toml and poetry.lock
• pyproject.toml / poetry.lock — new files replacing requirements.txt / requirements_lock.txt
• ros2/interfaces.bzl, ros2/test.bzl — update all requirement(...) calls to use Label("@rules_ros2_py_deps//:<pkg>") now that the
  hub repo is renamed
• repositories/*.BUILD.bazel — update remaining references from rules_ros2_pip_deps to rules_ros2_py_deps
• examples/MODULE.bazel — add hermetic_cc_toolchain (Zig CC) to enable cross-compilation in the examples workspace as an e2e test. However, for this to work, I have to incorporate the change in Remove -latomic from linkopts #618 in this PR.

Test plan

# Cross-compile to aarch64                                                                                                                     
cd examples && bazelisk build --platforms @zig_sdk//platform:linux_aarch64 //chatter:chatter  

Note

This PR is created against feature/jazzy, because that is the branch we are using. I am happy to create another version for the main branch if that's preferred.

[lalten]

FYI: @oxidase's https://github.com/oxidase/ofiuco could be an interesting alternative here

[ahans]

At least when using bzlmod, standard rules_python supports cross building just fine.

[linzhp]

@ahans can you give an example bzlmod setup with rules_python that can cross compile numpy with correct .so files?

[ahans]

@ahans can you give an example bzlmod setup with rules_python that can cross compile numpy with correct .so files?

I put up a minimal example here. Hope it's useful! Let me know if you have questions!

[linzhp]

hmm, somehow numpy is pulled by rules_ros2 is not cross-compiling with pip.parse. Let me dig deeper and report back

[ahans]

hmm, somehow numpy is pulled by rules_ros2 is not cross-compiling with pip.parse. Let me dig deeper and report back

I think the issue is that here in rules_ros2 we don't use any of the parameters that make rules_python use the Bazel downloader and resolve the required wheels itself instead of relying on pip (multi-platform-support).

In a repo where I use rules_ros2, I solved this by patching rules_ros2's MODULE.bazel:

• remove all pip.parse calls
• point rules_ros2 at the main repo's Python repo: use_repo(pip, rules_ros2_pip_deps = "pypi")

In my case, the main module's Python repo is called pypi. Of course, you need to make sure that everything that rules_ros2 depends on is part of the repo (all required wheels present and in compatible versions).

If things are not clear, I can extend the minimal example from above to include rules_ros2.

[linzhp]

Thanks for the pointer. I opened #620 as an alternative. That uses the multi-platform-support you suggested.

However, like the doc pointed out:

This will not work for sdists with C extensions

So this PR may still be needed in the future if we run into that case. But #620 seems to work for now.

[linzhp]

rules_ros2 depends on psutil, which has C extension:

rules_ros2/repositories/ros2cli.BUILD.bazel

Line 15 in 97cde9a

requirement("psutil"),

And the current version in the lock doesn't have Linux aarch64 wheels. I am mildly worried....

[ahans]

rules_ros2 depends on psutil, which has C extension:

We solved this by manually building an aarch64 wheel for psutil and putting that in our private PyPI. Since the pip.parse part of rules_ros2 is patched away anyway, this works fine.

And the current version in the lock doesn't have Linux aarch64 wheels. I am mildly worried....

You should probably update #620 to also upgrade psutil to a version that has aarch64 wheels. However, I wonder if @mvukov would accept this, since promising cross-compilation compatibility also means one has to make sure it actually works, which in turn increases testing and maintaining efforts. IMHO adapting rules_ros2 by applying some small patches locally is a good enough solution.

[linzhp]

Slight worried about the major version upgrade of psutil, but I can do that if @mvukov thinks that's ok

[mvukov]

hmm, somehow numpy is pulled by rules_ros2 is not cross-compiling with pip.parse. Let me dig deeper and report back

I think the issue is that here in rules_ros2 we don't use any of the parameters that make rules_python use the Bazel downloader and resolve the required wheels itself instead of relying on pip (multi-platform-support).

In a repo where I use rules_ros2, I solved this by patching rules_ros2's MODULE.bazel:

* remove all `pip.parse` calls

* point rules_ros2 at the main repo's Python repo: `use_repo(pip, rules_ros2_pip_deps = "pypi")`

In my case, the main module's Python repo is called pypi. Of course, you need to make sure that everything that rules_ros2 depends on is part of the repo (all required wheels present and in compatible versions).

If things are not clear, I can extend the minimal example from above to include rules_ros2.

Sorry for being late to the party. Is there a way we can make this working without patches? This issue is bugging me for a long time now -- I haven't had time to look at recent rules_python... In particular, that we can inject / override rules_ros2_pip_deps while using bzlmod.

Regarding psutil -- I am totally fine we upgrade it.

One very high level question: is cross-compilation still a think these days when arm64 HW is a commodity?

[linzhp]

Is there a way we can make this working without patches?

Yes, I made it work in #620

is cross-compilation still a think these days when arm64 HW is a commodity?

Unfortunately yes for Uber... Our dev machines are still mostly macOS and Linux x86-64, while some teams need to cross-compile to Linux-aarch64

[mvukov]

Shall we then move with #620?

Upgrading psutil is fine for me. But should be properly tested. We can look into aarch64 builds as well.

In a repo where I use rules_ros2, I solved this by patching rules_ros2's MODULE.bazel:

remove all pip.parse calls
point rules_ros2 at the main repo's Python repo: use_repo(pip, rules_ros2_pip_deps = "pypi")

<--- This is still puzzling me: how to override rules_ros2_pip_deps without patching rules_ros2. Very probably unrelated to this PR.

[linzhp]

I am fine with moving with #620 and close this

[mvukov]

OKay, let's do it like that then. OK to retarget this change against the main branch?

[linzhp]

OKay, let's do it like that then. OK to retarget this change against the main branch?

Sorry, do you want this change or #620?

[mvukov]

I'd say let's go with #620 since it's much simpler. WDYT?

[linzhp]

I'd say let's go with #620 since it's much simpler. WDYT?

Sounds good. But what do you mean by this?

OK to retarget this change against the main branch?

#620 is already targeting at main

[mvukov]

My bad. I just approved #620 .