Skip to content

Enable build on aarch64#509

Open
JunHe77 wants to merge 20 commits into
TencentCloud:masterfrom
JunHe77:armbuild
Open

Enable build on aarch64#509
JunHe77 wants to merge 20 commits into
TencentCloud:masterfrom
JunHe77:armbuild

Conversation

@JunHe77

@JunHe77 JunHe77 commented Jun 10, 2026

Copy link
Copy Markdown

This PR enables build on aarch64 paltform. Tested with both make all and ./deploy/one-click/build-release-bundle-builder.sh.

Comment on lines 232 to 239
cube_hypervisor::set_runtime_seccomp_rules(vec![
#[cfg(target_arch = "x86_64")]
(libc::SYS_mkdir, vec![]),
#[cfg(target_arch = "aarch64")]
(libc::SYS_mkdirat, vec![]),
(libc::SYS_getsockopt, vec![]),
(libc::SYS_setsockopt, vec![]),
]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seccomp rule inconsistency: The snapshot launch_vmm() is missing (libc::SYS_faccessat2, vec![]) from its seccomp allowlist, while the equivalent call in cube_hypervisor.rs (line 86) includes it on both x86_64 and aarch64.

On aarch64, if Rust's standard library or any internal dependency calls faccessat2 (used for file access checks), the seccomp filter will kill the snapshot worker process. Please add SYS_faccessat2 here to match cube_hypervisor.rs.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated,

@lkml-likexu

lkml-likexu commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

@JunHe77 Thanks for your contributions.

Could you please provide more test environments and the results of the corresponding test cases?

May I ask which aarch64 hardware and host kernel versions you have actually tested
the CubeSandbox use cases on — for example, the quickstart in the examples?

Could you help check whether this PR is actually helpful for aarch64 support?

For instance, in this PR we are still missing (at least):

  • An aarch64 mvm vmlinux builder
  • Documentation updates for developers and deployers

If the default page size on the arm machine is not 4KB, will Cloud Hypervisor (CH) still work properly?

Besides, being able to run CubeSandbox on Apple M-series chips would make cube and ARM much more popular.

@lkml-likexu lkml-likexu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For commit "docker/Dockerfile.builder: support multi-arch builder image"

Overall the arch-mapping is mostly correct (Go via dpkg --print-architecture, protoc aarch_64, rustup musl target, libseccomp --host, openssl multiarch dir all map correctly). But there is one confirmed regression that defeats part of the patch's own goal, plus a few smaller items.

High — OpenSSL lib-dir override is now dead on both arches

docker/Dockerfile.builder:28-29

    UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR=/usr/lib/${HOST_ARCH}-linux-gnu \
    UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR=/usr/lib/${HOST_ARCH}-linux-gnu \

openssl-sys only reads env vars prefixed with the full uppercased target triple (e.g. X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR, AARCH64_UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR) or the unprefixed OPENSSL_LIB_DIR. The renamed UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR matches no target, so it's silently never read — on both x86_64 and aarch64. The original block already listed all four arch-prefixed vars (so it was already arch-agnostic), and hypervisor/resources/Dockerfile:87-90 still uses the correct names. Fix: revert to the four arch-prefixed vars (no HOST_ARCH interpolation needed — they already covered both arches).

Medium — World-writable Rust toolchain

docker/Dockerfile.builder:133

chmod -R a+rwX /usr/local/rustup

a+rwX makes every toolchain binary world-writable (tampering surface / least-privilege violation). a+rX is sufficient for compilation; the write bit is only needed for runtime rustup install/update. Impact is bounded (ephemeral --rm container, non-root --user), but recommend dropping the w bit unless runtime rustup mutation is intended.

Low — HOST_ARCH default can silently produce a broken mixed-arch image

HOST_ARCH defaults to x86_64, but the Go download independently uses $(dpkg --print-architecture). A bare docker build (no --build-arg) on an arm host — which is how CI invokes it (build-builder-image.yml doesn't pass HOST_ARCH) — would fetch an arm64 Go but x86_64 rustup/libseccomp/openssl. Only the make builder-image path auto-detects the arch. Consider sourcing a single arch via buildx TARGETARCH, or at least documenting that direct docker build on non-x86_64 requires --build-arg HOST_ARCH=.

Low — The aarch64 path has zero CI coverage

All workflows run on x86_64 only (no QEMU/linux/arm64/arm runners). Every new HOST_ARCH=aarch64 branch is unexercised, so the patch's primary purpose is unvalidated, and the OpenSSL rename above wouldn't be caught by any existing check. Worth an arm64 build job (QEMU + buildx) building the image with HOST_ARCH=aarch64 and at least one musl Rust component.

Low — Stale docs

  • CONTRIBUTING.md:36 hardcodes the x86_64-unknown-linux-musl target prerequisite, now arch-dependent.
  • docker/README.md is the natural place to document the new HOST_ARCH arg / multi-arch capability.

Dismissed (raised by reviewers, but not real issues)

  • Removing the blank line between the two RUNs does not merge layers (continuation depends on trailing \).
  • The chmod -R does not create a duplicate/oversized layer — it's in the same RUN as the rustup install.
  • Unverified downloads (go/protoc/libseccomp/rustup) and unquoted HOST_ARCH interpolation are pre-existing / trusted-input, not introduced or worsened here.

Recommendation: address the High OpenSSL finding before merge; the Medium chmod is a quick safe improvement; the rest are optional follow-ups.

Comment thread CubeNet/cubevs/localgw_x86_bpfel.o Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add .gitignore entries for these auto-generated files.

@chenhengqi chenhengqi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Networking-related changes looks good to me. Thanks.

JunHe77 and others added 18 commits June 29, 2026 12:51
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I8148fa1c3eddff106f5e4223404ed1b3f5c37a52
The base image is downgraded to ubuntu 20.04 which doesn't provide clang-14 packages.
To generate cubevs arch-dependent bpf objects, clang 14 (minimum) is needed. Add the installation in dockerfile.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I29d85a0d2d7d593e988e08ed27fd6528b45eea3b
CubeMaster previously pulled in both the unversioned gomonkey module
and gomonkey/v2 at the same time. The v1 module is unmaintained and
only kept alive in go.sum because a single integration test helper
still imported it, which inflated the dependency graph and made the
two versions easy to mix up in future contributions.

This change consolidates the codebase on gomonkey/v2, the version
already used by the rest of CubeMaster, so that only one monkey-
patching library is shipped and audited.

No functional or test-behavior change is intended.

Assisted-by: Anthropic:claude-opus-4-7
Signed-off-by: Like Xu <likexu@tencent.com>
Change-Id: If8a836df9f6e040977d433f8ccee44f9d27416cf
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I499da5f135ae4dba76ec542e88e54cd440fb7ef8
The guest agent was hard-wired to x86_64 in two places: the Makefile
always built against x86_64-unknown-linux-musl, and the RPC startup
path used a PIO write to signal the shim that the vsock server is
ready, which is not available on aarch64. As a result the agent
could neither be built for nor run inside an ARM64 sandbox.

This commit lifts both assumptions. The musl build now follows the
host architecture by default while still allowing an explicit TRIPLE
override, and the readiness notification on aarch64 is delivered via
the SysCtrl MMIO region exposed by the hypervisor, matching what the
shim already listens for. A small print-target-path helper is added
so the surrounding build scripts can locate the produced binary
without duplicating the triple logic.

The x86_64 path is unchanged.

Assisted-by: Anthropic:claude-opus-4-7
Signed-off-by: Like Xu <likexu@tencent.com>
Change-Id: Iac42d7f405fe0faa8c2c3fe42200f3d0ee67a01f
Refactor the way to generate architecture dependent localgw/mvmtap/nodenic bpf code.
Add header file, vmlinux.h, for arm64. Update build deps and steps accrodingly.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: Ia712c5515f4e67208edc05dfe935406277b72ffe
After rebasing the hypervisor on a newer cloud-hypervisor revision,
the aarch64 path accumulated a number of regressions that prevented
MicroVMs from being built and booted on ARM64 hosts.

This commit consolidates the aarch64-only fixes required to bring
the platform back to a working state. It realigns the KVM, snapshot
and migration call sites with the upstream API changes, exposes the
SysCtrl device over MMIO so the guest can still signal shutdown and
reboot to the shim, and tightens the cross-arch cfg gates so the
crate compiles cleanly without warnings.

The x86_64 behavior is intentionally left unchanged.

Assisted-by: Anthropic:claude-opus-4-7
Signed-off-by: Like Xu <likexu@tencent.com>
Change-Id: Ied3d75b301a06869bddfd20d4cd022485ebfdd41
CubeShim was previously assuming an x86_64 host when launching the
guest VM and when installing seccomp rules for the hypervisor and
snapshot workers. As a result the shim could neither boot a guest
nor pass syscall filtering on aarch64 hosts.

This change generalizes the host-architecture assumptions so that
the shim works on both x86_64 and aarch64. The default guest kernel
cmdline now picks an appropriate console and drops x86-only mitiga-
tion knobs on ARM64, and the seccomp allow-lists account for the
syscall numbering differences between the two architectures.

No functional change is intended on x86_64.

Assisted-by: Anthropic:claude-opus-4-7
Signed-off-by: Like Xu <likexu@tencent.com>
Change-Id: Ie65a897403ad1b8c1d2710855c9bc51c3fb6e1bf
Change hardcoded architecture name to dynamically detect one. Add aarch64 dependent machine name and bios params.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I8053c73b3466382a154b36926551f42e8564f861
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: Idb1d099a00e8b1afa9bbf1e6b7be59bd73685dd4
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: If395689ca16cb5a213098c1df82b51babceddc97
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I2402cdb8c046a0fb1700cf88fae67255b924576a
As based image has downgraded to Ubuntu 20.04, the linker here is updated to align with this change

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I9c3b0b956233be5d9ccffcde569729568ace3a73
In cube_hypervisor.rs, SYS_faccessat2 is included in launch_vmm, while it is not in mod.rs of snapshot.
This could lead to snapshot worker issue due to allowlist mismatch.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I7a6e242ad6033eaaa6f8acc9f69622e63e10cfed
Replaced hardcode amd64 build with TARGETARCH from container build.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: Iecbf371bfe895eafa1e2ac0d802a21c06c31783a
Remove hardcoded arch string. Add dep generation for cubevs.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I4ceeb2f752a9eb1a3dc758786cdf077152517131
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: If97d0534d5f8c5c1c98fb6070f6808777aea9678
Add canonical CUBE_PROXY_COREDNS_IMAGE / CUBE_PROXY_BASE_IMAGE / CUBE_SANDBOX_MYSQL_IMAGE / CUBE_SANDBOX_REDIS_IMAGE / WEB_UI_IMAGE override variables in deploy/one-click/env.example and consume them from the one-click and systemd scripts.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: Ia2ad43860267728e57b2285b28a8cd9526b8eedd
JunHe77 added 2 commits June 29, 2026 12:51
Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: I598f2da9055d38b4778eeaa8b521cda35df1ab7e
Add a `make kernel KERNEL_SRC=/path/to/linux` target that builds a guest
kernel vmlinux from an external kernel source tree, using the in-tree config
(configs/kernel-oc9.<arch>.config).
The build runs inside the same toolchain builder image.
Supports native builds (x86_64 or aarch64) and cross builds
(x86_64 <-> aarch64). KERNEL_TARGET_ARCH selects the target; the matching
ARCH / CROSS_COMPILE values are derived automatically (override with
KERNEL_CROSS_COMPILE). The kernel is built out-of-tree (O=) so KERNEL_SRC
is left untouched; vmlinux lands in _output/kernel/<arch>/.

Signed-off-by: Jun He <jun.he@arm.com>
Change-Id: Ib0767143c13354ab0276c05c2c3331a3a2d01181
@JunHe77

JunHe77 commented Jun 29, 2026

Copy link
Copy Markdown
Author

@JunHe77 Thanks for your contributions.

Could you please provide more test environments and the results of the corresponding test cases?

May I ask which aarch64 hardware and host kernel versions you have actually tested the CubeSandbox use cases on — for example, the quickstart in the examples?

Could you help check whether this PR is actually helpful for aarch64 support?

For instance, in this PR we are still missing (at least):

  • An aarch64 mvm vmlinux builder
  • Documentation updates for developers and deployers

If the default page size on the arm machine is not 4KB, will Cloud Hypervisor (CH) still work properly?

Besides, being able to run CubeSandbox on Apple M-series chips would make cube and ARM much more popular.
Hi @lkml-likexu , thanks for reviewing this PR.
I tested it with Arm N2 and V2 platforms followed self-build-deploy.md, bare-metal-deploy.md. All works as expected.
Arm vmlinux (Image) builder and doc updated are added (782ce21, 4a0039a). Please review. Thanks.

@lkml-likexu

Copy link
Copy Markdown
Collaborator

@JunHe77 To fix Hypervisor Tests (x86-64) regression,

Could you review this diff to verify if it introduces any regressions on ARM?

diff --git a/hypervisor/test_infra/src/lib.rs b/hypervisor/test_infra/src/lib.rs
index 49e0e17..e0878a3 100644
--- a/hypervisor/test_infra/src/lib.rs
+++ b/hypervisor/test_infra/src/lib.rs
@@ -15,7 +15,7 @@ use std::io::{Read, Write};
 use std::net::TcpListener;
 use std::net::TcpStream;
 use std::os::unix::fs::PermissionsExt;
-use std::os::unix::io::{AsRawFd, FromRawFd};
+use std::os::unix::io::{AsRawFd, FromRawFd, RawFd};
 use std::path::Path;
 use std::process::{Child, Command, ExitStatus, Output, Stdio};
 use std::str::FromStr;
@@ -755,6 +755,49 @@ pub fn kill_child(child: &mut Child) {
 
 pub const PIPE_SIZE: i32 = 32 << 20;
 
+// Smallest pipe size we are willing to fall back to (1 MiB). Below this the
+// child can block on a full buffer while emitting verbose logs, so we keep a
+// generous floor.
+const MIN_PIPE_SIZE: i32 = 1 << 20;
+
+/// Enlarge a pipe's kernel buffer on a best-effort basis.
+///
+/// `F_SETPIPE_SZ` rounds the request up and asks the kernel for a contiguous
+/// `pipe_buffer` array. For large sizes (e.g. 32 MiB => ~320 KiB, order-7) this
+/// is a high-order allocation that can fail with `ENOMEM` under memory
+/// fragmentation/pressure, which is common when `common_parallel` spawns many
+/// hypervisor children at once (each wanting a big stdout *and* stderr pipe).
+///
+/// Rather than aborting the whole test on a transient allocation failure, we
+/// retry with progressively smaller sizes down to `MIN_PIPE_SIZE`, returning
+/// the size the kernel actually granted. Only a failure at the floor is fatal.
+fn set_pipe_size_best_effort(fd: RawFd) -> io::Result<i32> {
+    let mut desired = PIPE_SIZE;
+    loop {
+        let actual = unsafe { libc::fcntl(fd, libc::F_SETPIPE_SZ, desired) };
+        if actual != -1 {
+            return Ok(actual);
+        }
+
+        let err = io::Error::last_os_error();
+        // Only back off for allocation-related failures; anything else (e.g.
+        // EPERM, EINVAL) won't be cured by a smaller size.
+        let retryable = matches!(
+            err.raw_os_error(),
+            Some(libc::ENOMEM) | Some(libc::EAGAIN)
+        );
+        if !retryable || desired <= MIN_PIPE_SIZE {
+            return Err(err);
+        }
+
+        let next = (desired / 2).max(MIN_PIPE_SIZE);
+        eprintln!(
+            "warning: F_SETPIPE_SZ({desired}) failed ({err}); retrying with {next} bytes"
+        );
+        desired = next;
+    }
+}
+
 static NEXT_VM_ID: Lazy<Mutex<u8>> = Lazy::new(|| Mutex::new(1));
 
 pub struct Guest {
@@ -1296,17 +1339,11 @@ impl<'a> GuestCommand<'a> {
                 .unwrap();
 
             let fd = child.stdout.as_ref().unwrap().as_raw_fd();
-            let pipesize = unsafe { libc::fcntl(fd, libc::F_SETPIPE_SZ, PIPE_SIZE) };
-            if pipesize == -1 {
-                return Err(io::Error::last_os_error());
-            }
+            let pipesize = set_pipe_size_best_effort(fd)?;
             let fd = child.stderr.as_ref().unwrap().as_raw_fd();
-            let pipesize1 = unsafe { libc::fcntl(fd, libc::F_SETPIPE_SZ, PIPE_SIZE) };
-            if pipesize1 == -1 {
-                return Err(io::Error::last_os_error());
-            }
+            let pipesize1 = set_pipe_size_best_effort(fd)?;
 
-            if pipesize >= PIPE_SIZE && pipesize1 >= PIPE_SIZE {
+            if pipesize >= MIN_PIPE_SIZE && pipesize1 >= MIN_PIPE_SIZE {
                 Ok(child)
             } else {
                 Err(std::io::Error::new(

@lkml-likexu

lkml-likexu commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

@JunHe77 Pls note, the "deploy/one-click/build-release-bundle-builder.sh" does not work on oc9:

[one-click] packaging one-click release bundle on host with prebuilt artifacts
/data/likexu/CubeSandbox/deploy/one-click/build-release-bundle.sh: line 31: dpkg: command not found

You may need this fix:

diff --git a/deploy/one-click/build-release-bundle.sh b/deploy/one-click/build-release-bundle.sh
index 1b17f37..1ec440a 100755
--- a/deploy/one-click/build-release-bundle.sh
+++ b/deploy/one-click/build-release-bundle.sh
@@ -28,7 +28,7 @@ CUBE_PROXY_SOURCE_DIR="${ONE_CLICK_CUBE_PROXY_SOURCE_DIR:-${ROOT_DIR}/CubeProxy}
 CUBE_EGRESS_SOURCE_DIR="${ONE_CLICK_CUBE_EGRESS_SOURCE_DIR:-${ROOT_DIR}/CubeEgress}"
 WEB_SOURCE_DIR="${ONE_CLICK_WEB_SOURCE_DIR:-${ROOT_DIR}/web}"
 WEB_DIST_OVERRIDE="${ONE_CLICK_WEB_DIST_DIR:-}"
-MKCERT_BIN_ASSET="${ONE_CLICK_MKCERT_BIN:-${SCRIPT_DIR}/assets/bin/mkcert-v1.4.4-linux-$(dpkg --print-architecture)}"
+MKCERT_BIN_ASSET="${ONE_CLICK_MKCERT_BIN:-${SCRIPT_DIR}/assets/bin/mkcert-v1.4.4-linux-$(dpkg_arch)}"
 CUBE_KERNEL_VMLINUX="${ONE_CLICK_CUBE_KERNEL_VMLINUX:-${RAW_ARTIFACTS_DIR}/vmlinux}"
 KERNEL_ARTIFACT_ZIP="${WORK_ROOT}/cube-kernel-scf.zip"
 
diff --git a/deploy/one-click/lib/common.sh b/deploy/one-click/lib/common.sh
index 15b9ec1..902ff72 100755
--- a/deploy/one-click/lib/common.sh
+++ b/deploy/one-click/lib/common.sh
@@ -40,6 +40,20 @@ require_cmd() {
   command -v "${cmd}" >/dev/null 2>&1 || die "required command not found: ${cmd}"
 }
 
+# Print the Debian/dpkg-style architecture name (amd64/arm64) without depending
+# on `dpkg`, which is absent on RPM-based hosts (OpenCloudOS, RHEL, ...). Bundle
+# assets (e.g. mkcert-v1.4.4-linux-amd64) follow the dpkg naming convention, so
+# we map the kernel machine name from `uname -m` to it.
+dpkg_arch() {
+  local machine
+  machine="$(uname -m)"
+  case "${machine}" in
+    x86_64 | amd64) printf 'amd64\n' ;;
+    aarch64 | arm64) printf 'arm64\n' ;;
+    *) die "unsupported architecture: ${machine}" ;;
+  esac
+}
+
 _version_trim_leading_zeroes() {
   local value="$1"
   value="${value#${value%%[!0]*}}"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants