Guest execution architecture: reaching "any Linux tool works" beyond proot's ptrace ceiling

[mega123-art]

Summary

Every guest-execution bug we've hit (#112 git corruption, #115 link false-success, bun install failure, the failed glibc-native T0) traces to one architectural root: proot uses ptrace, a lossy syscall-translation layer forced on us by unrooted Android (untrusted_app blocks namespaces/root). We've fixed the widespread cases with targeted patches, but ptrace has a ceiling some tools sit above. This issue frames the long-term "any Linux tool works" architecture decision so it's made deliberately, not one band-aid at a time.

What's already fixed (proot patches, shipped/in-flight)

• Run the engine natively (no proot in the hot path): structural fix for the untrusted_app transport corruption #115 (PR fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116) — dropped --link2symlink. In untrusted_app the kernel denies hardlinks; l2s faked success (result=0) via dangling symlinks, defeating callers' fallbacks → silent git object loss + broken pnpm. Removing it makes linkat fail honestly → git falls back to rename, pnpm to copy. Fixes the whole link()-with-fallback class (git, pnpm, npm was never affected). Verified on-device (SM-A356E): git 50/50 objects, pnpm 589/589 files, node server HTTP 200.
• Fix #112: git clone in the proot guest under targetSdk-35 #114 — clone-only dulwich shim (superseded by fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116).

What proot CANNOT fix (the ceiling)

• bun — its installer opens node_modules as a directory fd then forks ~14 workers doing openat(dirfd, …)/linkat relative to that fd. proot can't correctly track dirfd→path across parallel ptraced processes → workers get ENOENT on an existing dir → Failed to install N packages, empty node_modules, misleading exit 0. Not the Run the engine natively (no proot in the hot path): structural fix for the untrusted_app transport corruption #115 bug (copyfile backend fails identically); not config-fixable. Parallel + fd-relative ops are exactly where ptrace sandboxes break down. (Even native Termux struggles to run bun.)
• glibc-native (T0, #115 comment) — running our glibc rootfs binaries directly under Android's bionic linker is impossible (ET_EXEC rejected; glibc rtld aborts; PIE glibc still segfaults). So "just drop proot for the current rootfs" is off the table.
• General ceiling: io_uring, complex fd-relative flows, some namespaces — anything ptrace can't faithfully mediate.

Reference: what Termux does

Termux's native environment is not proot — its ~2000 packages are compiled against bionic and run as real app processes (no ptrace) → real syscalls, hardlinks, fd semantics all work. When Termux needs glibc (real Ubuntu) it uses proot-distro — the same proot, the same bugs we have. So Termux "wins" only by avoiding proot for its own tools, at the cost of a fully bionic userland. Their Play-Store build execs those bionic PIE binaries via /system/bin/linker64 (our T0 confirmed that path works for bionic PIE, not for glibc).

The decision: how do we reach "any Linux tool works"?

Option	What	Fixes bun / exotic tools?	Cost / risk
A. Stay on proot, patch per-tool	keep glibc rootfs; fix tools as they break (l2s, git/pnpm config, …)	No — ceiling stays	Low per fix, but unbounded tail; can't cover parallel/fd-relative/io_uring tools
B. Native bionic userland (Termux model)	rebuild engine (node/git/…) as bionic PIE, run native via linker64, no proot in hot path; proot only for optional glibc	Most tools yes; bun still uncertain (flaky even on native Termux)	High — adopt/маintain a bionic userland, drop glibc rootfs; claude/codex ride native node (npm pkgs, OK once node runs)
C. microVM (Android AVF / pKVM)	real Linux kernel in a VM; everything runs natively incl. glibc	Yes — everything (real kernel: hardlinks, io_uring, fd, all)	High + device-gated (needs AVF/pKVM: Android 13+, specific SoCs; MediaTek/Exynos support varies), perf/battery/size cost; fall back to proot where AVF absent

Recommendation

• Short term: ship fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116 (fixes the real, common problem) and treat bun as unsupported-under-proot (document it; agents use npm which works). Don't rebuild the world for bun.
• Long term: pick B or C deliberately if "arbitrary Linux tooling must work" is a real product requirement. C (AVF microVM) is the only path that fixes everything including glibc + exotic tools, but it's device-gated and needs a proot fallback. B (native bionic) removes ptrace for the engine but keeps a long tail (and doesn't clearly win bun).

Asks

1. Decide whether "any arbitrary Linux tool works on-device" is a hard requirement or best-effort.
2. If hard requirement → spike AVF availability across the target device matrix (Seeker/MediaTek, Snapdragon, Exynos) — that gates option C.
3. Otherwise → adopt "proot + targeted fixes, bun/exotic unsupported" as the documented stance.

Related: #112, #114, #115 (+ PR #116). Evidence (probe harness, on-device traces) in debug/issue-115/ and the #115 comments.

[mega123-art]

References + likely next bug class (proot-improvement track)

Capturing prior art for the incremental "keep improving the translator" path (option A), separate from the bigger B/C architecture bets.

Upstream / forks to reference

• termux/proot — upstream of the proot build we ship. The link-faking extension is src/extension/link2symlink/ — this is the file to patch if we ever do the deeper proot-level fix instead of just dropping the flag (e.g. make l2s honest: don't poke result=0 when the underlying rename/symlink didn't land). For now fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116 removes the flag entirely, which is simpler and correct for our domain.
• termux/proot-distro — the full launch flag set + fake /proc bindings. When Android blocks /proc entries, they bind-mount fakes for /proc/loadavg, /proc/stat, /proc/uptime, /proc/version, etc. This is probably our next bug class (see below).
• UserLAnd — closest app to our setup (proot + rootfs in an untrusted_app). Launch flow in UserLAnd-Assets-Support/execInProot.sh; they run their own proot fork (CypherpunkArmory/proot). Note: they kept --link2symlink and patched around it rather than removing it — but they target SDK 30, so they never hit our stricter untrusted_app (SDK 35) domain where the kernel denies hardlinks. That's why removal is right for us and wasn't needed for them.

Likely NEXT bug class: fake /proc

Under untrusted_app, Android restricts several /proc files. proot-distro pre-empts this by binding fake versions. Tools that will trip on it:

• node's os.loadavg() / os.cpus() → reads /proc/loadavg, /proc/stat
• process/uptime libraries → /proc/uptime
• version sniffers / vite & other file watchers → /proc/version, and inotify//proc/sys/fs/inotify/* limits
• anything reading /proc/meminfo, /proc/self/* we don't already bind

Approach when we hit it: mirror proot-distro — generate small fake files in the rootfs and -b <fake>:/proc/<name> bind them at launch (in DirectProotExec.prootCommand), same pattern as our existing -b /dev /proc /sys binds. Cheap, per-file, no proot rebuild.

Status of the incremental track

• fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116 — dropped --link2symlink (link-fake bug). ✅
• chore/115-remove-dulwich-shim — remove the now-redundant git clone always fails in proot (Android) guest — "remote did not send all necessary objects" (suspected Ubuntu 24.04 rseq regression) #112 dulwich shim (gated on real-git-clone verification).
• Next — fake /proc binds when os.loadavg()/watchers break (not yet observed; watch for it).

[mega123-art]

Update (2026-07-14, SM-A356E / Android 16, all on-device in untrusted_app via the guest-runner)

The central claim of this issue — "bun sits above proot's ptrace ceiling; parallel + fd-relative ops are where ptrace breaks down" — is wrong. Tested it directly instead of assuming, and proot handles every mechanism the ceiling story blamed. bun works under our proot with a thin config nudge, exactly like git did.

What actually happens under proot (proven, one variable each)

proot handles parallel dirfd-relative work fine. A python probe doing a real dirfd-relative openat (create relative to a dir fd):

A single dirfd-openat:      OK
B forked dirfd-openat:      OK
C parallel(14) dirfd-openat: rc_ok=14  files_on_disk=14   <- all 14 parallel forked workers succeed
D dirfd-linkat:             FAIL EACCES                    <- only failure, and it's a HARDLINK (the #115 kernel denial), not dirfd

proot resolves dirfd -> path via readlink("/proc/<pid>/fd/<n>") (src/path/path.c:287); the tracees are proot's own descendants (same uid) so that readlink works even for forked workers. The "can't track dirfd across parallel procs" theory does not survive contact with the device.

proot handles openat2 + every RESOLVE_ flag (bun is Zig, uses openat2):

abs/rel openat2  none / RESOLVE_NO_SYMLINKS / RESOLVE_BENEATH / RESOLVE_IN_ROOT  -> all fd OK

(Our shipped proot converts openat2->openat and drops the resolve flags, src/syscall/enter.c:2184.)

bun's real failure (both backends, identical)

warn: Error caching manifest for dunder-proto: EACCES        <- hardlink into ~/.bun cache (the #115 denial)
ENOENT: could not open the "node_modules" directory          <- fatal; bun won't create node_modules under proot

Two bun-specific defaults, not a proot limitation:

1. Default hardlink backend hits the kernel hardlink denial (EACCES) → "Failed to install N packages", empty node_modules, misleading exit 0.
2. bun won't create node_modules itself under proot (ENOENT when it's absent).

The fix (proven end-to-end through the shipped Installer)

Pre-create node_modules + --backend=copyfile:

precreate + copyfile : 66 packages installed, files=594, express PRESENT, require() OK   ✅
precreate + symlink  : installs but require() FAILS (dangling into cache)
precreate + hardlink : Failed to install 67 packages, files=0                            ✗

Shipped as a /usr/local/bin/bun wrapper (PATH-ahead of /usr/bin/bun, same pattern as the old git shim) that does exactly those two things for install commands. Plain bun add express through the wrapper → 66 packages, 594 files, require() OK (was 0 files unwrapped). bunfig.toml backend="copyfile" was not honored (fell back to hardlink) — only the CLI flag works, hence the wrapper rather than a config file.

Revised picture of the "any Linux tool works" question

The basement is in better shape than this issue assumed. Failures cluster into two fixable classes, both now handled:

Class	Mechanism	Fix	Universal?
link()-lie	kernel denies hardlinks; --link2symlink faked success	drop l2s (#116) → link fails honestly → tools fall back	Yes (whole class)
/proc denial	untrusted_app blocks /proc/loadavg, inotify, cap_last_cap, …	fake-file binds (this branch)	Yes (whole class)
tool link-strategy defaults	git default link; bun default hardlink + won't mkdir node_modules	git core.createObject=rename (#116); bun wrapper (this branch)	per-tool nudge on a now-solid basement

No evidence of a genuine ptrace ceiling in anything we've tested. git ✓ pnpm ✓ npm ✓ bun ✓ (now). cargo untested but is a hardlink-heavy tool → expect it to fall in the same "force copy/rename" bucket, not a ceiling.

Recommendation shift: Option A (stay on proot + targeted fixes) is carrying more weight than this issue credited it with. The B/C (native-bionic / AVF microVM) bets are still the answer for a hard "arbitrary kernel feature" requirement (io_uring, real namespaces), but nothing in the current tool set forces that bet. Keeping this issue open as the architecture tracker; downgrading bun from "ceiling / unsupported" to "supported via wrapper."

Branch: feat/115-proot-hardening (fake /proc + bun wrapper). Guest-runner probes on feat/115-native-exec-probe.

[mega123-art]

cargo tested + the deepest layer (2026-07-14, on-device, untrusted_app)

cargo works under proot — and it's the model citizen. Installed cargo/rustc 1.75 via apt, cargo new + a crates.io dep (itoa), cargo build:

• crates.io index update ✓, crate download ✓, registry unpack (~/.cargo/registry/src/…/itoa-1.0.18) ✓, rustc compile ✓.
• When cargo hit the hardlink denial it handled it gracefully: warning: hard linking files in the incremental compilation cache failed. copying files instead. → it retries with a copy and continues. This is exactly the fallback bun's default backend lacked. cargo needs no fix.
• The build only failed on error: linker 'cc' not found — a missing/unconfigured C linker, which traces to the finding below, not to cargo.

New finding: apt/dpkg is broken by the same hardlink denial (and dpkg has no fallback)

ls: cannot access '/var/lib/dpkg/status-old': No such file or directory
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Permission denied
E: Sub-process /usr/bin/dpkg returned an error code (2)
… E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.

status-old doesn't exist, yet dpkg fails "Permission denied" creating it → that's link(status, status-old) (dpkg's atomic status backup) → EACCES (the #115 kernel hardlink denial). Packages still unpack, but dpkg aborts at the status commit and leaves itself "interrupted"; dpkg --configure -a fails the same way. Net effect: apt install <anything> half-works and leaves the guest inconsistent (e.g. gcc unpacked but its cc alternative symlink never configured → cargo can't link).

Unlike git/pnpm/cargo, dpkg does not fall back to copy on link() failure — so honest-EACCES (our correct l2s removal) breaks it. This is the one place removing l2s regressed vs faking success.

This is the case for a real proot source patch = the "whole solution" for the link class

Per-tool nudges (git config, bun wrapper) work but don't scale to dpkg/coreutils/etc. The universal fix is in the basement — patch our proot's link/linkat translation (we already build-flag our proot; GPLv2, publish the patch):

• Option 1 — copy-on-link: on link()/linkat(), if the kernel denies the hardlink (EACCES), transparently perform a real file copy instead. Semantically fine for the write-once consumers that break today (git objects, dpkg status backup, bun cache, package installs). This is the "make link2symlink honest" the original brief asked for — but copy the bytes instead of leaving a dangling symlink. Fixes every link-using tool with or without its own fallback, no per-tool config.
• Option 2 — errno remap (EACCES→EXDEV): cheaper. Many tools already fall back to copy when link() returns EXDEV (cross-filesystem), but not on EACCES — dpkg and coreutils mv/cp -l are in this camp. Returning EXDEV makes them trigger their own copy path. (git/pnpm/cargo fall back on any errno, so they're already fine.) One-line change, but relies on each tool having an EXDEV fallback.

Both need building proot from source (we currently ship the prebuilt Termux .deb). Recommend Option 1 (copy-on-link) as the durable basement fix; it's the closest thing to "every link()-using tool just works." Tracking as the next basement item.

Scoreboard under proot (untrusted_app, on-device)

tool	status	note
git	✅	l2s off + core.createObject=rename (#116)
pnpm	✅	l2s off → copy fallback
npm	✅	never affected
bun	✅	wrapper: copyfile + precreate node_modules (#122)
cargo	✅	graceful hardlink→copy fallback; needs a C toolchain in the rootfs to link
apt/dpkg	⚠️	link()-EACCES, no fallback → half-configured installs. Wants the copy-on-link proot patch.

Still no genuine ptrace ceiling in anything tested — every failure is the hardlink-denial class or a missing package, both addressable.

[mega123-art]

✅ copy-on-link built + VERIFIED on-device — every tool now works (2026-07-14, SM-A356E, untrusted_app)

The --copy-on-link PRoot patch is built from source (CI android-assets, run 29325627361 → patched libproot.so carries the flag) and verified through the guest-runner on the patched binary:

raw `ln src dst`   : ln OK; inodes differ (1419993 vs 672435)  → copy fallback fired (independent copy, not a fake symlink)
dpkg               : `dpkg --configure -a` self-heals; /usr/bin/cc now configured  → apt/dpkg FIXED (was the last broken tool)
cargo build        : Finished dev; ./target/debug/cargotest → "Hello, world!"       → fully green (was blocked on cc)
git clone express  : 0 fsck errors, 240 files
bun add express    : 66 packages, 594 files, require() typeof=function
server             : http://localhost:4317 → HTTP 200 (patched proot boots clean)

Final scoreboard under proot (untrusted_app, on-device): git ✅ pnpm ✅ npm ✅ bun ✅ cargo ✅ apt/dpkg ✅. No genuine ptrace ceiling in anything tested — the whole hardlink class is now handled in the basement.

How it fits together (layers, most-universal first)

1. --copy-on-link (PRoot source patch) — on link()/linkat() EACCES, PRoot byte-copies a regular file to a fresh O_EXCL destination and reports success. Fixes the whole class in one place: tools with a fallback (git/pnpm/cargo) never even reach it; tools without one (dpkg, coreutils) now just work. This is the real "any Linux tool" basement fix.
2. Fake /proc binds — the other class: /proc/loadavg, inotify limits, cap_last_cap, etc. that untrusted_app denies.
3. Thin per-tool belt-and-suspenders — git core.createObject=rename (fix(android): #115 root cause — drop proot --link2symlink (fixes git, pnpm, all link() tools) #116) and the bun wrapper (--backend=copyfile + pre-create node_modules). Both still correct with copy-on-link present, and they keep the tool working even if an older (pre-patch) android-assets artifact is reused (DirectProotExec gates the flag on the binary advertising it, so an old binary is never handed an unknown option).

Build is reproducible: pinned termux-packages commit + package-builder image digest, GPLv2 patch published in surfaces/android/proot/. PR #122.