hl.1

.\" Copyright 2025 Moritz Angermann <moritz@zw3rk.com>, zw3rk pte. ltd.
.\" SPDX-License-Identifier: Apache-2.0
.Dd March 15, 2026
.Dt HL 1
.Os
.Sh NAME
.Nm hl
.Nd run aarch64-linux and x86_64-linux ELF binaries on macOS Apple Silicon
.Sh SYNOPSIS
.Nm
.Op Fl hvV
.Op Fl t Ar seconds
.Op Fl -timeout Ar seconds
.Op Fl -sysroot Ar path
.Op Fl -fs-mode Ar legacy|rooted
.Op Fl -bind Ar host:guest
.Op Fl -guest-home Ar path
.Op Fl -guest-cwd Ar path
.Op Fl -isolated
.Op Fl -app-profile
.Op Fl -trace Ar categories
.Op Fl -audio-backend Ar name
.Op Fl -verbose
.Op Fl -gdb Ar port
.Op Fl -gdb-stop-on-entry
.Op Fl -
.Ar elf-path
.Op Ar args ...
.Sh DESCRIPTION
.Nm
executes aarch64-linux and x86_64-linux ELF binaries on macOS Apple
Silicon using Apple's Hypervisor.framework.
It creates a lightweight virtual machine with a minimal EL1 shim that
provides exception vectors and forwards Linux syscalls (SVC #0) to the
host via HVC, where they are translated to macOS equivalents.
.Pp
aarch64-linux binaries run natively via HVF.
x86_64-linux binaries are JIT-translated to ARM64 by Apple's Rosetta
Linux translator
.Pq loaded automatically from Pa /Library/Apple/usr/libexec/oah/RosettaLinux/rosetta .
The architecture is detected from the ELF header's
.Va e_machine
field.
.Pp
Both statically-linked and dynamically-linked ELF binaries are
supported.
For dynamic binaries, a sysroot must be provided via
.Fl -sysroot
(musl or glibc).
.Pp
Guest memory is identity-mapped with 2MB block page tables.
The address space size is determined at runtime by querying the
maximum IPA (Intermediate Physical Address) size and capping it at
40 bits: a 36-bit VM has a 64GB primary mapping and a 40-bit VM has
a 1TB primary mapping.
Both guest architectures use the negotiated host width.
High Rosetta virtual addresses are translated into the available primary
GPA span and do not require a 48-bit VM.
.Sh OPTIONS
.Bl -tag -width indent
.It Fl h , Fl -help
Display this manual page.
Falls back to a brief usage summary if
.Xr man 1
is not available.
.It Fl V , Fl -version
Print the version string and exit.
.It Fl v , Fl -verbose
Enable verbose output on stderr.
Shows ELF loading details, page table layout, vCPU register
configuration, and syscall-level diagnostics.
.It Fl t Ar seconds , Fl -timeout Ar seconds
Opt-in hang watchdog for the main vCPU.
Default: 0 (off).
When
.Ar seconds
is greater than 0, the guest is terminated if it runs that long without a
VM exit (a coarse hang detector: the alarm is armed per
.Fn hv_vcpu_run
iteration).
A value of 0 disables it; other values must be 0..86400.
This is
.Em not
a total-execution-time limit and does not fire for a guest that keeps
making syscalls.
It is off by default because a compute-bound or
.Pa vDSO Ns -time-polling
guest makes no VM exits and was killed by the old 10-second default.
It covers the main vCPU only and is inherited by
.Fl -fork-child
processes.
.It Fl -sysroot Ar path
Path to a sysroot directory containing the dynamic linker and shared
libraries (musl or glibc).
When set,
.Nm
loads the ELF interpreter (e.g.,
.Pa ld-musl-aarch64.so.1
or
.Pa ld-linux-aarch64.so.1 )
from this sysroot and transparently redirects absolute path opens
through it.
.It Fl -fs-mode Ar legacy|rooted
Filesystem namespace mode.
Default:
.Ar rooted
(deterministic mount table, virtual guest CWD, no host
.Xr chdir 2 ).
.Ar legacy
preserves prior sysroot/host path behavior for tools and tests.
.It Fl -bind Ar host:guest
Add a bind mount from host path to guest path.
Also accepts
.Ar guest=host .
Longest guest prefix wins.
By default (unless
.Fl -isolated ),
.Nm
binds
.Pa $HOME
to
.Pa /home/user .
.It Fl -guest-home Ar path
Set the guest home directory path used by profiles.
.It Fl -guest-cwd Ar path
Set the initial virtual guest working directory
.Pq rooted mode .
Default when the home bind is active:
.Pa /home/user/Music
if
.Pa $HOME/Music
exists on the host, otherwise
.Pa /home/user .
GTK file dialogs list the working directory, and a full macOS home can stall
on cloud placeholders, so the media root is preferred when present.
.It Fl -isolated
Do not auto-bind
.Pa $HOME
to
.Pa /home/user
and do not set the default guest CWD.
Useful for hermetic runs or explicit
.Fl -bind
profiles only.
.It Fl -app-profile
Install the fuller rooted app profile (home, Music, Desktop,
Documents, Downloads, /Volumes, virtual /dev and /proc).
.It Fl -trace Ar categories
Enable category tracing.
Categories:
.Ar fs,fd,dev,audio,proc,fork,sys
(or
.Ar all ).
Also set via
.Ev HL_TRACE .
Unknown categories are an error.
Off by default.
.It Fl -audio-backend Ar name
Select PCM backend:
.Ar null ,
.Ar null-realtime ,
.Ar wav ,
or
.Ar coreaudio
(default).
Override with this flag or
.Ev HL_AUDIO_BACKEND .
Deterministic tests use null/wav.
.It Fl -gdb Ar port
Start a GDB Remote Serial Protocol stub listening on TCP
.Ar port .
GDB or LLDB can then attach with
.Ql target remote localhost: Ns Ar port .
Only supported for aarch64 guests; ignored for x86_64 (Rosetta).
Supports up to 16 hardware breakpoints and 16 watchpoints.
.It Fl -gdb-stop-on-entry
When used with
.Fl -gdb ,
stop the guest at the ELF entry point and wait for GDB to attach
before executing any guest instructions.
Without this flag, the guest runs immediately and only stops on
breakpoints or Ctrl+C.
.It Fl -
Stop processing
.Nm
options.
All subsequent arguments are passed to the guest binary.
.El
.Sh MEMORY LAYOUT
The guest address space is organized as follows:
.Pp
.Bl -column "0x200000000" "varies" "Description" -compact
.It Sy Start Ta Sy End Ta Sy Description
.It 0x010000 Ta 0x0FFFFF Ta Page table pool (960KB)
.It 0x100000 Ta 0x1FFFFF Ta Shim code (2MB, RX)
.It 0x200000 Ta 0x3FFFFF Ta Shim data/stack (2MB, RW)
.It 0x400000 Ta varies Ta ELF LOAD segments
.It 0x1000000 Ta varies Ta brk heap (grows up)
.It 0x7800000 Ta 0x7800FFF Ta Stack guard page (PROT_NONE, dynamic)
.It 0x7801000 Ta 0x7FFFFFF Ta Stack (8MB, RW, grows down)
.It 0x10000000 Ta 0x1FFFFFFF Ta mmap RX region (256MB initial)
.It 0x200000000 Ta 0x20FFFFFFF Ta mmap RW region (256MB initial, at 8GB)
.El
.Pp
Additional mmap pages beyond the initial regions are mapped dynamically
via page table extension with TLB invalidation.
PIE executables (ET_DYN) are loaded at 0x400000.
.Sh SYSCALL SUPPORT
.Nm
translates 172 Linux aarch64 syscalls to macOS equivalents.
.Ss Basic I/O
read, readv, write, writev, openat, close, lseek, pread64, pwrite64,
preadv, pwritev, preadv2, pwritev2, ioctl, fstat, newfstatat.
.Ss Process and Memory
exit, exit_group, brk, mmap, munmap, mprotect, madvise, mincore,
set_tid_address, getpid, gettid, getppid, getuid, geteuid, getgid,
getegid, uname, sethostname, getrandom, umask, prctl.
.Ss Filesystem
getcwd, chdir, fchdir, faccessat, readlinkat, unlinkat, mkdirat,
mknodat, symlinkat, linkat, renameat, renameat2, getdents64, dup, dup3,
fcntl, pipe2, ftruncate, truncate, statfs, fstatfs, statx, flock,
close_range, fchmod, fchmodat, fchownat, fchown, utimensat, chroot,
memfd_create.
.Ss Extended Attributes
getxattr, lgetxattr, setxattr, lsetxattr, listxattr, llistxattr,
removexattr, lremovexattr, fgetxattr, fsetxattr, flistxattr,
fremovexattr.
.Ss Signals
rt_sigaction, rt_sigprocmask, rt_sigsuspend, rt_sigpending,
rt_sigreturn, rt_tgsigqueueinfo, sigaltstack, kill, tgkill.
.Ss Time and Timers
clock_gettime, clock_getres, clock_nanosleep, nanosleep, gettimeofday,
setitimer, getitimer, timerfd_create, timerfd_settime, timerfd_gettime.
.Ss Process Management
clone, clone3 (stub, returns -ENOSYS), execve, execveat, wait4, waitid,
setuid, setgid, setreuid, setregid, setresuid, getresuid, setresgid,
getresgid, setpriority, getpriority, setpgid, getpgid, setsid,
getgroups, getrusage, getrlimit, setrlimit, sched_getaffinity,
sched_setaffinity, prlimit64, sysinfo.
.Ss I/O Optimization and Sync
fallocate, sendfile, sync, fsync, fdatasync, msync, sched_yield,
copy_file_range, splice, vmsplice, tee, fadvise64.
.Ss I/O Multiplexing
pselect6, ppoll, epoll_create1, epoll_ctl, epoll_pwait.
.Ss Special File Descriptors
eventfd2, signalfd4, timerfd_create.
.Ss Networking
socket, socketpair, bind, listen, accept, accept4, connect,
getsockname, getpeername, sendto, recvfrom, setsockopt, getsockopt,
shutdown, sendmsg, recvmsg, sendmmsg, recvmmsg.
.Ss inotify (via kqueue)
inotify_init1, inotify_add_watch, inotify_rm_watch.
.Ss Futex
FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET,
FUTEX_REQUEUE, FUTEX_CMP_REQUEUE, FUTEX_WAKE_OP, FUTEX_LOCK_PI,
FUTEX_UNLOCK_PI, FUTEX_TRYLOCK_PI.
.Ss Ptrace (for Rosetta JIT)
PTRACE_SEIZE, PTRACE_CONT, PTRACE_INTERRUPT, PTRACE_GETREGSET,
PTRACE_SETREGSET.
.Ss Stubs (no-op, return 0)
mlock, munlock, membarrier, set_robust_list.
.Ss /proc and /dev Emulation
The following virtual paths are intercepted in openat and readlinkat:
.Pa /proc/self/exe ,
.Pa /proc/self/cwd ,
.Pa /proc/self/fd/ Ns Ar N ,
.Pa /proc/self/maps ,
.Pa /proc/self/status ,
.Pa /proc/self/stat ,
.Pa /proc/self/cmdline ,
.Pa /proc/self/mountinfo ,
.Pa /proc/self/mounts ,
.Pa /proc/cpuinfo ,
.Pa /proc/meminfo ,
.Pa /proc/stat ,
.Pa /proc/mounts ,
.Pa /proc/uptime ,
.Pa /proc/loadavg ,
.Pa /proc/version ,
.Pa /proc/filesystems ,
.Pa /proc/sys/vm/mmap_min_addr ,
.Pa /proc/sys/kernel/randomize_va_space ,
.Pa /etc/mtab ,
.Pa /etc/passwd ,
.Pa /etc/group ,
.Pa /var/run/utmp ,
.Pa /run/utmp ,
.Pa /dev/null ,
.Pa /dev/zero ,
.Pa /dev/urandom ,
.Pa /dev/random ,
.Pa /dev/tty ,
.Pa /dev/stdin ,
.Pa /dev/stdout ,
.Pa /dev/stderr ,
.Pa /dev/fd/ Ns Ar N .
.Sh DYNAMIC LINKING
For dynamically-linked ELF binaries, use
.Fl -sysroot
to point to a directory containing the dynamic linker and shared
libraries (musl or glibc):
.Bd -literal -offset indent
hl --sysroot /path/to/musl-sysroot ./my-dynamic-program
hl --sysroot /path/to/glibc-sysroot ./my-glibc-program
.Ed
.Pp
.Nm
parses PT_INTERP from the ELF to find the interpreter path (e.g.,
.Pa /lib/ld-musl-aarch64.so.1
or
.Pa /lib/ld-linux-aarch64.so.1 ) ,
loads it from the sysroot, and sets the entry point to the dynamic
linker.
Absolute path opens are transparently redirected through the sysroot.
.Pp
The sysroot is inherited by fork children and honored across execve.
.Pp
For x86_64 dynamically-linked binaries, Rosetta handles the x86_64
dynamic linker internally; the
.Fl -sysroot
provides the shared libraries that Rosetta's internal linker resolves.
.Sh THREADING
.Nm
supports multi-threaded guest binaries.
Guest threads map 1:1 to host pthreads, each with its own HVF vCPU
sharing the same guest physical memory.
.Pp
The clone syscall with CLONE_THREAD creates new vCPUs.
Futex operations (FUTEX_WAIT, FUTEX_WAKE, and bitset/requeue variants)
are fully implemented.
Per-thread signal masks are maintained.
Up to 64 concurrent guest threads are supported.
.Sh x86_64 SUPPORT (ROSETTA)
.Nm
transparently runs x86_64-linux ELF binaries via Apple's Rosetta Linux
translator.
When
.Nm
detects
.Va e_machine
== EM_X86_64 in the ELF header, it loads the Rosetta binary from
.Pa /Library/Apple/usr/libexec/oah/RosettaLinux/rosetta
and configures it as a binfmt_misc-style interpreter.
.Pp
Rosetta JIT-translates x86_64 instructions to ARM64 at runtime.
All syscalls appear as ARM64 SVC #0 instructions, so
.Nm Ap s
existing syscall handlers work transparently.
.Pp
Rosetta's ahead-of-time (AOT) translation is supported via a built-in
rosettad handler thread with persistent caching at
.Pa ~/.cache/hl-rosettad/ .
.Pp
x86_64 mode requires the Rosetta Linux binary to be installed
.Pq typically via Xr softwareupdate 8 .
.Sh FORK
macOS HVF allows only one VM per process.
Fork is implemented via
.Xr posix_spawn 2
and IPC.
For native aarch64 binaries, the parent uses copy-on-write via
file-backed shared memory: the backing fd is sent to the child via
SCM_RIGHTS, and the child maps it MAP_PRIVATE for instant COW semantics
with zero data copy.
For x86_64 (Rosetta) binaries, the parent serializes VM state
(registers, memory regions, file descriptors, signal state) over a Unix
socket to a new
.Nm
process running in
.Fl -fork-child
mode.
.Pp
CLOEXEC semantics follow POSIX: all file descriptors are inherited
across fork; O_CLOEXEC takes effect only at execve.
.Sh ENVIRONMENT
.Nm
reads the following host environment variables:
.Bl -tag -width "HL_GUEST_LANG_OFF"
.It Ev HL_TRACE
Trace categories, same syntax as
.Fl -trace .
Applied before option parsing, so it also covers startup events.
.It Ev HL_TRACE_REDACT
Redact host home paths in trace and crash-report output.
.It Ev HL_AUDIO_BACKEND
Default audio backend.
An explicit
.Fl -audio-backend
takes precedence.
.It Ev HL_AUDIO_WAV
Output path for the
.Cm wav
backend.
.It Ev HL_SYSCALL_STATS
Enable syscall volume statistics
.Pq also enabled by Fl -trace Ns = Ns Cm sys .
.It Ev HL_GUEST_LD_PRELOAD , Ev HL_GTK_THEME , Ev HL_GTK_RC
Injected into the guest environment when set.
.It Ev HL_GUEST_LANG , Ev HL_GUEST_LANG_OFF
Override or disable the guest locale default.
.El
.Pp
The guest does
.Em not
inherit the host environment verbatim in rooted mode.
The following keys are
set or rewritten for the guest:
.Ev LD_LIBRARY_PATH ,
.Ev LD_PRELOAD ,
.Ev HOME ,
.Ev XAUTHORITY ,
.Ev DISPLAY ,
.Ev TMPDIR ,
.Ev TMP ,
.Ev GTK_RC_FILES ,
.Ev XLOCALEDIR ,
.Ev GTK_EXE_PREFIX
and
.Ev LANG ,
which is set to
.Ev C.UTF-8
unless
.Ev HL_GUEST_LANG_OFF
is set.
.Sh EXIT STATUS
The exit status of
.Nm
is the exit code of the guest binary.
If the guest calls
.Fn exit 42 ,
.Nm
exits with status 42.
.Sh EXAMPLES
Run a statically-linked binary:
.Bd -literal -offset indent
hl ./hello-static
.Ed
.Pp
Run with verbose output and a 30-second timeout:
.Bd -literal -offset indent
hl --verbose --timeout 30 ./my-program arg1 arg2
.Ed
.Pp
Run a dynamically-linked binary with a musl sysroot:
.Bd -literal -offset indent
hl --sysroot ./musl-sysroot ./my-dynamic-program
.Ed
.Pp
Run jq on a JSON file:
.Bd -literal -offset indent
hl /path/to/aarch64-linux/jq '.name' data.json
.Ed
.Pp
Run an interactive bash shell:
.Bd -literal -offset indent
hl /path/to/aarch64-linux/bash
.Ed
.Pp
Run an x86_64-linux binary (automatically detected and run via Rosetta):
.Bd -literal -offset indent
hl ./x86_64-linux-binary arg1 arg2
.Ed
.Pp
Pass
.Fl -
to separate
.Nm
options from guest arguments starting with
.Sq - :
.Bd -literal -offset indent
hl -- ./my-program --guest-flag
.Ed
.Sh LIMITATIONS
.Bl -bullet -compact
.It
Apple HVF enforces W^X: memory regions cannot be simultaneously
writable and executable.
Code and data in the same 2MB block require L3 page table splitting.
.It
Job control syscalls (TIOCSPGRP, TIOCSCTTY) are stubs.
Interactive job control in guest shells is limited.
.It
MAP_SHARED is treated as MAP_PRIVATE.
The guest is single-process from the VM's perspective, so shared
memory semantics are not observable.
.It
No 32-bit ARM or 32-bit x86 support.
.It
x86_64 mode requires Apple's Rosetta Linux binary to be installed.
Some x86_64 features (SA_RESETHAND, TLS edge cases) are limited by
Rosetta's internal signal/thread state shadowing.
.It
Requires macOS on Apple Silicon with the
.Li com.apple.security.hypervisor
entitlement.
.El
.Sh SEE ALSO
.Lk https://developer.apple.com/documentation/hypervisor "Apple Hypervisor.framework"
.Sh AUTHORS
.An Moritz Angermann Aq Mt moritz@zw3rk.com ,
zw3rk pte. ltd.