From 20c89a52dde78a9509e363733b968e0110f41a10 Mon Sep 17 00:00:00 2001 From: sugarforever Date: Tue, 20 Oct 2026 21:38:32 +0100 Subject: [PATCH 1/4] feat(ios): add executable iSH host and native feasibility probes --- .gitignore | 2 + README.md | 2 + docs/ios-ish-execution.md | 86 +++++++++++++ docs/superpowers/plans/2026-10-20-ios-ish.md | 27 ++++ native/ish/host.c | 31 +++++ native/ish/probe.c | 63 ++++++++++ native/ish/probe.h | 6 + native/ish/simulator.m | 125 +++++++++++++++++++ scripts/ish/audit_guest_boundary.py | 30 +++++ scripts/ish/build_probe.py | 106 ++++++++++++++++ scripts/ish/host_runner.py | 48 +++++++ scripts/ish/test_guest.py | 69 ++++++++++ scripts/ish/test_probe_controls.py | 17 +++ scripts/ish/test_simulator.py | 41 ++++++ 14 files changed, 653 insertions(+) create mode 100644 docs/ios-ish-execution.md create mode 100644 docs/superpowers/plans/2026-10-20-ios-ish.md create mode 100644 native/ish/host.c create mode 100644 native/ish/probe.c create mode 100644 native/ish/probe.h create mode 100644 native/ish/simulator.m create mode 100644 scripts/ish/audit_guest_boundary.py create mode 100644 scripts/ish/build_probe.py create mode 100644 scripts/ish/host_runner.py create mode 100644 scripts/ish/test_guest.py create mode 100644 scripts/ish/test_probe_controls.py create mode 100644 scripts/ish/test_simulator.py diff --git a/.gitignore b/.gitignore index a01a4ba..7ac80bc 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,5 @@ src/generated/ .DS_Store *.log + +scripts/ish/__pycache__/ diff --git a/README.md b/README.md index 0359b7f..b2de977 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,8 @@ Skill 支持 SKILL.md 直链、GitHub 文件、目录和仓库根目录 URL。 文件环境使用 npm 发布的 `pi-durable-expo-env@0.1.0`，版本与完整性由锁文件固定，不依赖同级目录的源码或本地 tarball。更新步骤、受控原生测试入口和验证记录见 [文件环境集成说明](docs/mobile-files.md)。 +iOS iSH 原生执行仍处于独立可行性探针阶段，尚未接入应用。构建步骤、验证结果和阻塞项见 [iOS iSH 实验](docs/ios-ish-execution.md)。 + ## 验证 ```sh diff --git a/docs/ios-ish-execution.md b/docs/ios-ish-execution.md new file mode 100644 index 0000000..8c6c886 --- /dev/null +++ b/docs/ios-ish-execution.md @@ -0,0 +1,86 @@ +# iOS iSH execution design and acceptance + +Issue: https://github.com/sugarforever/moose/issues/6 + +## Intended product + +An optional iOS ExecutionEnv runs real i386 Linux programs inside the Moose process through iSH's usermode CPU interpreter and syscall layer. File tools and shell must see the same persistent guest workspace. No model credentials, SQLite, SecureStore values, or host home directory may be mounted or inherited. Node execution environments must never enter Hermes. + +## Evidence boundary + +First build and run a pinned upstream kernel independently of React Native. A standalone native probe is a feasibility artifact, not an enabled Moose feature. Keep the existing Expo file environment until embedding satisfies all acceptance criteria. Do not ship a nonfunctional bash tool or silently substitute a host shell. + +Upstream source: https://github.com/ish-app/ish at c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c. README describes usermode x86 emulation, syscall translation, and precompiled threaded interpreter gadgets rather than generated executable code. `meson.build` separates static emulator, fakefs, and kernel libraries from the app frontend. `kernel/task.c` owns a global PID table; `fs/mount.c` owns a global mount list. Thread-local `current` is a task pointer, not an independent kernel instance. `kernel/exit.c` halts/unmounts the kernel when PID 1 dies. `xX_main_Xx.h` adds a host `exit` callback. Embedding must replace that callback and maintain a supervisor init; spawning another kernel per conversation in the same process is unsafe without upstream changes. + +## Licensing and redistribution + +`LICENSE.md` specifies GPLv3 plus `LICENSE.IOS`, with additional GPLv2 licensing for contributions after the identified commit and named earlier authors. Do not assume every historical contribution can be linked under GPLv2 alone. `LICENSE.IOS` permits otherwise GPL-compliant App Store distribution despite App Store terms; it explicitly retains source and license-text obligations. Before distributing a linked Moose binary, select a compatible whole-program license, include license texts/notices, and provide complete corresponding source and reproducible build instructions for the exact kernel, modifications, frontend and rootfs packages. Source-only feasibility scripts do not make that distribution decision. Rootfs package licenses and sources require a separate manifest. No prebuilt kernel/rootfs is committed here. + +## Required architecture + +One native kernel supervisor per app process, serialized initialization and shutdown. Each conversation gets an independent persistent guest filesystem with a non-root guest identity and restricted mount/syscall policy. A guest chroot alone is not sufficient if the guest retains root or can mount host paths. iSH's emulator shares Moose's native privileges: it is not an OS security boundary. Disable dangerous host mounts and unnecessary network/devices before model-directed commands are enabled. + +The bridge must allocate per-command stdin/stdout/stderr descriptors, encode argv and env as NUL-separated strings with length validation, set guest cwd, stream bounded output with stream identity and omitted-byte accounting, and return exit status. Only explicit environment variables are passed. Cancellation/timeout must signal the entire guest process group, reap descendants, close descriptors, and leave the supervisor and other conversations alive. Never use host process termination as cancellation inside Moose. + +Persist workspace files and command intent/status, not CPU registers or credentials. After process death, pending commands become interrupted; do not replay commands with possible side effects. Reconnect by stable command ID where the native process still exists. iOS suspension pauses execution; foreground recovery must reconcile state. No background execution guarantee. + +## Acceptance gates + +- Real shell pipeline, fork/wait, argv with metacharacters, cwd/env, separate streaming stdout/stderr, and nonzero status in host harness and iOS simulator. +- File read/write/edit and shell share files; reopen preserves them; two conversations and symlink/absolute-path attacks cannot expose another conversation or host data. +- Timeout, cancellation, output flood, descendant cleanup, repeated commands, supervisor survival, and initialization races verified natively. +- Pi Durable real Harness integration and environment conformance with controlled model transport. +- Native device build and simulator interaction; physical-device evidence reported separately. No JIT entitlement required. +- Runtime bundle, tests, types, portable smoke, both resource exports, Expo doctor; native compilation is distinct from resource export. + +Until all security/lifecycle gates pass, deliver as draft with actual feasibility results and explicit missing implementation. Do not change the product's advertised capabilities. + +## Running the executable probes + +These scripts currently target Apple Silicon macOS. Prerequisites: Xcode with an installed iOS simulator runtime, Python 3, Meson, Ninja, LLVM **and LLD** (Homebrew packages are separate), libarchive, pkg-config, git. The scripts do not install dependencies. LLVM is only used to cross-compile the guest i386 VDSO; Apple Clang builds host and iOS code. For example, put Homebrew LLD on PATH before building: + +```sh +PATH="$(brew --prefix lld)/bin:$PATH" python3 scripts/ish/build_probe.py \ + --cache /tmp/moose-ish-probe --simulator +xcrun simctl install booted /tmp/moose-ish-probe/MooseISHProbe.app +python3 scripts/ish/test_simulator.py +python3 scripts/ish/audit_guest_boundary.py \ + --runner /tmp/moose-ish-probe/host/moose-ish-probe \ + --root /tmp/moose-ish-probe/root/data +``` + +Use a dedicated cache outside the repository. The build script fetches the exact iSH commit without changing it and verifies the Alpine 3.23.0 x86 minirootfs SHA-256 against the pinned value. Upstream submodules are not needed for these three static libraries. Rootfs checksum source: https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86/alpine-minirootfs-3.23.0-x86.tar.gz.sha256. + +`native/ish/probe.c` directly boots the real kernel with bounded argv/env packing. `host.c` is a standalone process adapter; its exit callback may terminate that executable. `simulator.m` instead owns a dedicated pthread and records status without terminating UIKit. iSH tasks call `pthread_exit`; running them on GCD worker threads caused the original native survival regression. A guest PID 1 exit still shuts down the one kernel, so the probe deliberately boots only once per app launch. It is not the required long-lived supervisor. + +Host capture in `scripts/ish/host_runner.py` streams separate bytes, limits combined captured output to 1 MiB, and kills/reaps the **entire standalone host process** on timeout or a callback failure. The flood test uses a 64 KiB bound. This is not a solution for per-command mobile cancellation. Simulator commands are fixed, finite probes; they write stdout/stderr to app-private files and do not implement a bounded streaming bridge. Do not put arbitrary model commands into this app. + +The simulator test launches the native app twice, checks stream contents/status, verifies the host remains alive after guest exit, and checks persistence on the second launch. The app remains installed under `dev.moose.ishprobe`, separate from Moose. Test artifacts are in its `Documents/probe`. Remove the disposable app with `xcrun simctl uninstall booted dev.moose.ishprobe` when finished. The cache contains local GPL-linked test binaries and rootfs; do not publish that cache without satisfying distribution obligations. + +## Actual results, 2026-10-20 + +| Evidence | Result | +| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Fresh scripted host + arm64 iOS simulator build | Passed from a new external cache; no upstream source patches | +| Real Alpine x86 guest tests | 8 passed: argv/cwd/env/exit, fork/pipeline/persistence, empty inherited environment, host timeout, early streaming, output bound, oversize argv, invalid cwd | +| Simulator native execution, iPhone 17 Pro / iOS 26.2, Xcode 26.2 | Passed: shell, literal argv, cwd/env, separate stream files, exit 7, fork/pipeline, UIKit host survival, relaunch persistence | +| Probe tooling regression | Passed: runner failure produces nonzero inconclusive status, never isolation success | +| Host boundary audit | **FAILED / BLOCKER**: guest mounted a newly created host temporary directory and read only its controlled sentinel | +| Project `npm run verify` | Passed: runtime bundle, 41 tests, TypeScript, portable smoke, Android and iOS Hermes resource exports | +| `npx expo-doctor` | 19/20; existing react-native-tcp-socket New Architecture metadata warning | +| Native compiler warnings | Upstream syscall function-pointer casts and assembly gadget alignment warnings; builds completed, warnings are not claimed resolved | +| Physical iPhone, signed device build, app background/foreground recovery | Not performed | +| Pi Durable integration, native bridge, real model calls | Not implemented / not performed; no credentials or billable requests used | + +The audit is an intentionally red product acceptance gate, separate from the functional probe tests. It exits nonzero when the sentinel becomes visible. It must not be inverted into a passing security test. `fs/mount.c` registers `realfs`, and `sys_mount` passes guest-provided source paths to `realfs_mount`, which resolves and opens host paths. Dropping the guest uid alone is not a demonstrated fix: the inspected syscall does not perform the expected privilege check. This reproducer is why unmodified upstream must not be wired to Moose's model tools. + +## Remaining work before #6 can be completed + +1. Fork/pin an embedded core policy that denies host filesystem mounts and unintended syscall/device/network capabilities; run native adversarial isolation tests, including other conversations and Moose files. Re-audit path and socket boundaries, not only `mount`. +2. Implement one surviving init/supervisor and per-conversation guest identities/root directories. Keep all shared mounts and tasks behind a serialized native owner. Verify repeated commands and independent process-group cancellation without host exit or kernel shutdown. +3. Implement native descriptor-based streaming with byte bounds, output omission accounting, exit-status events, timeouts, cancellation and idempotent cleanup. Handle callback errors and descendants holding output descriptors. +4. Implement Pi 1.3.0's actual `ExecutionEnv` file contract (`BinaryReader`, line scanning, directory reads, mutation and flush semantics) against the same guest filesystem. The fakefs metadata database means exposing its `data` directory through Expo file APIs is not equivalent to guest file operations. Run Pi's environment conformance and real Harness tests with controlled SSE. +5. Persist stable command IDs/status and reconcile interrupted commands on app restart; do not transparently rerun side effects. Exercise background/foreground and termination on native devices. +6. Decide a compatible linked-app distribution license and corresponding-source delivery for Moose + iSH + bundled packages before distributing binaries. Keep this PR draft until runtime gates are met; do not close #6 as implemented on the strength of these probes. + +Apple's background-execution guidance remains applicable to an embedded interpreter: https://developer.apple.com/documentation/uikit/extending-your-app-s-background-execution-time. No probe result establishes continuous execution while iOS suspends the application. diff --git a/docs/superpowers/plans/2026-10-20-ios-ish.md b/docs/superpowers/plans/2026-10-20-ios-ish.md new file mode 100644 index 0000000..715c254 --- /dev/null +++ b/docs/superpowers/plans/2026-10-20-ios-ish.md @@ -0,0 +1,27 @@ +# iOS iSH Implementation Plan + +**Goal:** Prove native iSH execution and implement a safe optional Pi ExecutionEnv when the kernel boundary permits it. + +**Architecture:** Pin upstream core outside the committed tree; build its static libraries and exercise guest binaries. Keep experimental native code outside the shipping dependency graph until lifecycle/isolation gates pass. + +**Tech Stack:** C, Objective-C, iSH, Meson/Ninja, Python test runner, TypeScript/Pi Durable. + +**Spec:** `docs/ios-ish-execution.md` + +## Constraints + +No host-shell substitution, Node in Hermes, real model calls, unrelated checkout changes, or binary redistribution. Preserve GPL/source obligations. One global upstream kernel is not one isolated instance per conversation. + +## Review focus + +Host-process exit; guest-root host access; background descendants; unbounded output; interrupted command replay. These require native evidence before enabling product integration. + +## Tasks + +- [x] Write executable feasibility assertions first: guest shell, streams/status, persistence, fork, argv/cwd/env, bounded host timeout. +- [x] Observe failure without a built guest kernel, then build pinned upstream and add the minimum native entrypoint needed. +- [x] Build/run a standalone simulator application with the same core; record evidence and differences from host execution. +- [x] Resolve supervisor, credentials, filesystem isolation, cancellation and recovery before connecting Pi ExecutionEnv. If unresolved, leave product disabled and enumerate blocking gates in draft PR. +- [ ] Run project verification and Expo doctor, inspect diff, commit, push, create draft/ready PR according to evidence, attach PR. + +Status: executable host/native probes complete. Product integration remains blocked by the red host-mount boundary audit and missing supervisor/cancellation/file contract. See the spec for the exact remaining gates. diff --git a/native/ish/host.c b/native/ish/host.c new file mode 100644 index 0000000..fb7843d --- /dev/null +++ b/native/ish/host.c @@ -0,0 +1,31 @@ +#include "kernel/calls.h" +#include "kernel/task.h" +#include "probe.h" +#include +#include +#include + +static void exited(struct task *task, int status) { + if (task->parent) + return; + /* Deliberately only valid in a standalone harness, never in Moose. */ + fflush(NULL); + _Exit((status & 0xff) ? 128 + (status & 0x7f) : status >> 8); +} +int main(int argc, const char *argv[]) { + if (argc < 5) + return 64; + int separator = 3; + while (separator < argc && strcmp(argv[separator], "--")) { + if (!strchr(argv[separator], '=')) + return 64; + separator++; + } + if (separator >= argc - 1) + return 64; + exit_hook = exited; + int err = moose_ish_probe_boot(argv[1], argv[2], argc - separator - 1, + argv + separator + 1, separator - 3, argv + 3); + fprintf(stderr, "iSH probe boot failed: %d\n", err); + return 70; +} diff --git a/native/ish/probe.c b/native/ish/probe.c new file mode 100644 index 0000000..2cef029 --- /dev/null +++ b/native/ish/probe.c @@ -0,0 +1,63 @@ +// Preserve upstream header prerequisites (fs/path.h is not standalone). +// clang-format off +#include +#include +#include "kernel/calls.h" +#include "kernel/init.h" +#include "kernel/task.h" +#include "fs/real.h" +#include "fs/path.h" +#include "probe.h" + +// clang-format on + +/* Avoid upstream CLI's unbounded 4096-byte argv copy and host env lookup. */ +static int pack(int count, const char *const values[], char *out, size_t size) { + size_t offset = 0; + for (int i = 0; i < count; i++) { + size_t len = strlen(values[i]) + 1; + if (len >= size - offset) + return -1; + memcpy(out + offset, values[i], len); + offset += len; + } + out[offset] = 0; + return 0; +} + +int moose_ish_probe_boot(const char *root, const char *cwd, int argc, + const char *const argv[], int envc, + const char *const env[]) { + if (argc < 1 || envc < 0 || cwd[0] != '/') + return -1; + char arguments[32768] = {0}, environment[32768] = {0}; + if (pack(argc, argv, arguments, sizeof(arguments)) || + pack(envc, env, environment, sizeof(environment))) + return -1; + int err = mount_root(&fakefs, root); + if (err < 0) + return err; + err = become_first_process(); + if (err < 0) + return err; + current->thread = pthread_self(); + struct fd *pwd = generic_open(cwd, O_RDONLY_, 0); + if (IS_ERR(pwd)) + return PTR_ERR(pwd); + fs_chdir(current->fs, pwd); + err = create_piped_stdio(); + if (err < 0) + return err; + create_some_device_nodes(); + err = do_mount(&procfs, "proc", "/proc", "", 0); + if (err < 0) + return err; + err = do_mount(&devptsfs, "devpts", "/dev/pts", "", 0); + if (err < 0) + return err; + err = do_execve(argv[0], argc, arguments, environment); + if (err < 0) + return err; + task_run_current(); + return -1; +} diff --git a/native/ish/probe.h b/native/ish/probe.h new file mode 100644 index 0000000..9e17568 --- /dev/null +++ b/native/ish/probe.h @@ -0,0 +1,6 @@ +#pragma once +/* Feasibility only: one kernel boot per host process. Not a multi-tenant API. + */ +int moose_ish_probe_boot(const char *root, const char *cwd, int argc, + const char *const argv[], int envc, + const char *const env[]); diff --git a/native/ish/simulator.m b/native/ish/simulator.m new file mode 100644 index 0000000..3b155bc --- /dev/null +++ b/native/ish/simulator.m @@ -0,0 +1,125 @@ +// UIKit must precede iSH headers: iSH defines macros used by Apple headers. +// clang-format off +#import +#include +#include +#include "kernel/task.h" +#include "kernel/calls.h" +#include "probe.h" +// clang-format on + +/* Standalone controlled probe; never registered as a Moose native module. */ +static NSString *results; +static NSString *guestRoot; +static void exited(struct task *task, int status) { + if (task->parent) + return; + @autoreleasepool { + [@{ + @"waitStatus" : @(status), + @"exitCode" : @(status >> 8) + } writeToFile:[results stringByAppendingPathComponent:@"exit.plist"] + atomically:YES]; + } +} +static void *run_guest(void *unused) { + @autoreleasepool { + int input = open("/dev/null", O_RDONLY); + int output = open([[results stringByAppendingPathComponent:@"stdout"] + fileSystemRepresentation], + O_WRONLY | O_CREAT | O_TRUNC, 0600); + int errors = open([[results stringByAppendingPathComponent:@"stderr"] + fileSystemRepresentation], + O_WRONLY | O_CREAT | O_TRUNC, 0600); + if (input < 0 || output < 0 || errors < 0) + return NULL; + dup2(input, 0); + dup2(output, 1); + dup2(errors, 2); + close(input); + close(output); + close(errors); + exit_hook = exited; + const char *argv[] = { + "/bin/sh", "-c", + "printf 'cwd=%s env=%s arg=%s\\n' \"$PWD\" \"$PROBE\" \"$1\"; " + "if test -f native-saved; then printf 'reopened='; cat native-saved; " + "fi; " + "printf persistent > native-saved; (printf 'pipeline\\n' | cat) & " + "wait; " + "printf 'stderr-probe\\n' >&2; exit 7", + "probe", "literal; $(false)"}; + const char *env[] = {"PROBE=explicit"}; + int error = + moose_ish_probe_boot([[guestRoot stringByAppendingPathComponent:@"data"] + fileSystemRepresentation], + "/workspace", 5, argv, 1, env); + [@(error).description + writeToFile:[results stringByAppendingPathComponent:@"boot-error"] + atomically:YES + encoding:NSUTF8StringEncoding + error:nil]; + } + return NULL; +} +@interface ProbeDelegate : UIResponder +@property(nonatomic, strong) UIWindow *window; +@end +@implementation ProbeDelegate +- (BOOL)application:(UIApplication *)application + didFinishLaunchingWithOptions:(NSDictionary *)options { + self.window = [[UIWindow alloc] initWithFrame:UIScreen.mainScreen.bounds]; + UIViewController *view = [UIViewController new]; + view.view.backgroundColor = UIColor.systemBackgroundColor; + UILabel *label = [[UILabel alloc] initWithFrame:CGRectMake(20, 80, 350, 150)]; + label.text = @"iSH native feasibility probe\nResults in Documents/probe\nNo " + @"model connection"; + label.numberOfLines = 0; + [view.view addSubview:label]; + self.window.rootViewController = view; + [self.window makeKeyAndVisible]; + NSString *documents = NSSearchPathForDirectoriesInDomains( + NSDocumentDirectory, NSUserDomainMask, YES) + .firstObject; + results = [documents stringByAppendingPathComponent:@"probe"]; + NSFileManager *fm = NSFileManager.defaultManager; + [fm createDirectoryAtPath:results + withIntermediateDirectories:YES + attributes:nil + error:nil]; + [fm removeItemAtPath:[results stringByAppendingPathComponent:@"exit.plist"] + error:nil]; + NSString *root = [documents stringByAppendingPathComponent:@"root"]; + NSError *copyError = nil; + if (![fm fileExistsAtPath:root] && + ![fm copyItemAtPath:[NSBundle.mainBundle.bundlePath + stringByAppendingPathComponent:@"root"] + toPath:root + error:©Error]) { + [copyError.description + writeToFile:[results stringByAppendingPathComponent:@"boot-error"] + atomically:YES + encoding:NSUTF8StringEncoding + error:nil]; + return YES; + } + guestRoot = root; + pthread_t thread; + int error = pthread_create(&thread, NULL, run_guest, NULL); + if (!error) + pthread_detach(thread); + else + [@(error).description + writeToFile:[results stringByAppendingPathComponent:@"boot-error"] + atomically:YES + encoding:NSUTF8StringEncoding + error:nil]; + return YES; +} +@end +int main(int argc, char **argv) { + @autoreleasepool { + return UIApplicationMain(argc, argv, nil, + NSStringFromClass(ProbeDelegate.class)); + } +} diff --git a/scripts/ish/audit_guest_boundary.py b/scripts/ish/audit_guest_boundary.py new file mode 100644 index 0000000..8e957c7 --- /dev/null +++ b/scripts/ish/audit_guest_boundary.py @@ -0,0 +1,30 @@ +#!/usr/bin/env python3 +"""Fail closed if unmodified iSH can mount a controlled host directory. + +Only a newly created temporary sentinel is read. No app/user data is inspected. +A failure is an integration blocker, not an expected-safe test pass. +""" +import argparse +from pathlib import Path +import subprocess +import tempfile + +p = argparse.ArgumentParser(description=__doc__) +p.add_argument('--runner', required=True, type=Path) +p.add_argument('--root', required=True, type=Path) +a = p.parse_args() +with tempfile.TemporaryDirectory(prefix='moose-ish-boundary-') as directory: + sentinel = b'controlled-host-sentinel' + Path(directory, 'marker').write_bytes(sentinel) + result = subprocess.run([str(a.runner.resolve()), str(a.root.resolve()), '/', '--', + '/bin/sh', '-c', 'printf "MOOSE_AUDIT_READY\\n"; mkdir -p /mnt/boundary || exit 92; ' + 'if mount -t real "$1" /mnt/boundary; then cat /mnt/boundary/marker; ' + 'else printf "MOOSE_AUDIT_DENIED\\n"; exit 17; fi', + 'audit', directory], env={}, capture_output=True, timeout=10) + if not result.stdout.startswith(b'MOOSE_AUDIT_READY\n'): + raise SystemExit('INCONCLUSIVE: guest did not confirm readiness') + if sentinel in result.stdout: + raise SystemExit('BLOCKED: guest mounted a host directory outside its root; do not expose this kernel to models') + if result.returncode != 17 or result.stdout != b'MOOSE_AUDIT_READY\nMOOSE_AUDIT_DENIED\n': + raise SystemExit('INCONCLUSIVE: guest did not confirm a completed mount-denial probe') + print('Mount probe denied; this single check does not establish complete isolation') diff --git a/scripts/ish/build_probe.py b/scripts/ish/build_probe.py new file mode 100644 index 0000000..df71d81 --- /dev/null +++ b/scripts/ish/build_probe.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Build pinned upstream iSH, not a host-shell replacement. macOS/Apple Silicon.""" +import argparse +import hashlib +from pathlib import Path +import plistlib +import shutil +import subprocess +import urllib.request + +REVISION = 'c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c' +ROOTFS_URL = 'https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86/alpine-minirootfs-3.23.0-x86.tar.gz' +ROOTFS_SHA256 = '855408b4682ce72b098e87106a4456eea11b6b816b6bc0ea5910ac3a8bd50a8c' +REPO = Path(__file__).resolve().parents[2] +NATIVE = REPO / 'native' / 'ish' + + +def run(*argv, **kwargs): + return subprocess.run([str(arg) for arg in argv], check=True, **kwargs) + + +def capture(*argv): + return subprocess.check_output([str(arg) for arg in argv], text=True).strip() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cache', required=True, type=Path, + help='Dedicated disposable build directory outside the repository') + parser.add_argument('--simulator', action='store_true') + args = parser.parse_args() + cache = args.cache.resolve() + if cache == REPO or REPO in cache.parents: + parser.error('--cache must be outside the repository') + cache.mkdir(parents=True, exist_ok=True) + source = cache / 'ish' + if not source.exists(): + run('git', 'init', source) + run('git', '-C', source, 'remote', 'add', 'origin', 'https://github.com/ish-app/ish.git') + run('git', '-C', source, 'fetch', '--depth', '1', 'origin', REVISION) + run('git', '-C', source, 'checkout', '--detach', REVISION) + if capture('git', '-C', source, 'rev-parse', 'HEAD') != REVISION: + raise RuntimeError('Unexpected upstream revision; use a fresh cache directory') + if capture('git', '-C', source, 'status', '--porcelain', '--untracked-files=no'): + raise RuntimeError('Refusing a modified upstream tree') + host = cache / 'host' + common = ['-Dbuildtype=release', '-Dno_crlf=true'] + if not (host / 'build.ninja').exists(): + run('meson', 'setup', host, source, *common) + run('ninja', '-C', host, 'libish.a', 'libish_emu.a', 'libfakefs.a', 'tools/fakefsify') + libraries = [host / name for name in ('libish.a', 'libish_emu.a', 'libfakefs.a')] + compiler_flags = ['-O2', '-DENGINE_ASBESTOS=1', f'-I{source}', f'-I{host}', f'-I{NATIVE}'] + runner = host / 'moose-ish-probe' + run('clang', *compiler_flags, NATIVE / 'host.c', NATIVE / 'probe.c', + *libraries, '-lsqlite3', '-lpthread', '-o', runner) + tarball = cache / 'alpine.tar.gz' + if not tarball.exists(): + with urllib.request.urlopen(ROOTFS_URL, timeout=60) as response: + # Reject oversized/error downloads before writing a rootfs. + data = response.read(16 * 1024 * 1024 + 1) + if len(data) > 16 * 1024 * 1024: + raise RuntimeError('Rootfs download exceeds bound') + tarball.write_bytes(data) + if hashlib.sha256(tarball.read_bytes()).hexdigest() != ROOTFS_SHA256: + raise RuntimeError('Rootfs SHA-256 mismatch') + root = cache / 'root' + if not root.exists(): + run(host / 'tools/fakefsify', tarball, root) + run(runner, root / 'data', '/', '--', '/bin/mkdir', '-p', '/workspace') + run('python3', REPO / 'scripts/ish/test_guest.py', '--runner', runner, '--root', root / 'data') + if not args.simulator: + return + sdk = capture('xcrun', '--sdk', 'iphonesimulator', '--show-sdk-path') + cross = cache / 'simulator.cross' + # Meson's cross-file syntax is Python-like; repr preserves spaces in SDK paths. + cross.write_text("[binaries]\nc = " + repr(['/usr/bin/clang', '-target', + 'arm64-apple-ios15.0-simulator', '-isysroot', sdk]) + + "\nar = '/usr/bin/ar'\nstrip = '/usr/bin/strip'\n[host_machine]\n" + "system = 'darwin'\ncpu_family = 'aarch64'\ncpu = 'arm64'\nendian = 'little'\n") + sim = cache / 'simulator' + if not (sim / 'build.ninja').exists(): + run('meson', 'setup', sim, source, '--cross-file', cross, *common) + run('ninja', '-C', sim, 'libish.a', 'libish_emu.a', 'libfakefs.a') + app = cache / 'MooseISHProbe.app' + app.mkdir(exist_ok=True) + run('xcrun', '--sdk', 'iphonesimulator', 'clang', '-target', 'arm64-apple-ios15.0-simulator', + '-O2', '-fobjc-arc', '-DENGINE_ASBESTOS=1', f'-I{source}', f'-I{sim}', f'-I{NATIVE}', + NATIVE / 'probe.c', NATIVE / 'simulator.m', + *[sim / name for name in ('libish.a', 'libish_emu.a', 'libfakefs.a')], + '-framework', 'UIKit', '-framework', 'Foundation', '-lsqlite3', '-lpthread', + '-o', app / 'MooseISHProbe') + with (app / 'Info.plist').open('wb') as output: + plistlib.dump(dict(CFBundleExecutable='MooseISHProbe', CFBundleIdentifier='dev.moose.ishprobe', + CFBundleName='Moose iSH Probe', CFBundleVersion='1', CFBundleShortVersionString='0.1', + CFBundlePackageType='APPL', LSRequiresIPhoneOS=True, MinimumOSVersion='15.0', + UIDeviceFamily=[1, 2], UILaunchScreen={}), output) + if (app / 'root').exists(): + shutil.rmtree(app / 'root') + shutil.copytree(root, app / 'root') + # Local ad-hoc signing only. Does not publish or produce a distributable IPA. + run('codesign', '--force', '--sign', '-', app) + print(f'Simulator app built at {app}; install with xcrun simctl install booted ') + + +if __name__ == '__main__': + main() diff --git a/scripts/ish/host_runner.py b/scripts/ish/host_runner.py new file mode 100644 index 0000000..55c49fa --- /dev/null +++ b/scripts/ish/host_runner.py @@ -0,0 +1,48 @@ +"""Bounded streaming capture for the standalone HOST feasibility executable. + +Killing this process terminates all guest threads. This mechanism cannot be used +inside Moose: native per-command process-group cancellation remains unimplemented. +""" +import os +import selectors +import subprocess +import time + + +class OutputLimit(RuntimeError): + pass + + +def execute(argv, *, timeout=15, max_bytes=1024 * 1024, on_output=None): + output = {'stdout': bytearray(), 'stderr': bytearray()} + total = 0 + deadline = time.monotonic() + timeout + with subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, env={}) as process: + try: + with selectors.DefaultSelector() as selector: + for stream in output: + descriptor = getattr(process, stream) + os.set_blocking(descriptor.fileno(), False) + selector.register(descriptor, selectors.EVENT_READ, stream) + while selector.get_map(): + remaining = deadline - time.monotonic() + if remaining <= 0: + raise subprocess.TimeoutExpired(argv, timeout) + for key, _ in selector.select(min(remaining, 0.1)): + chunk = os.read(key.fd, min(16384, max_bytes - total + 1)) + if not chunk: + selector.unregister(key.fileobj) + continue + total += len(chunk) + if total > max_bytes: + raise OutputLimit(f'guest output exceeded {max_bytes} bytes') + output[key.data].extend(chunk) + if on_output: + on_output(key.data, chunk, process) + status = process.wait(timeout=max(0.001, deadline - time.monotonic())) + finally: + if process.poll() is None: + process.kill() + process.wait() + return subprocess.CompletedProcess(argv, status, bytes(output['stdout']), bytes(output['stderr'])) diff --git a/scripts/ish/test_guest.py b/scripts/ish/test_guest.py new file mode 100644 index 0000000..4a92052 --- /dev/null +++ b/scripts/ish/test_guest.py @@ -0,0 +1,69 @@ +#!/usr/bin/env python3 +"""Acceptance tests against a real iSH executable; never substitutes host sh.""" +import argparse +import os +from pathlib import Path +import subprocess +import unittest +from host_runner import execute, OutputLimit + +parser = argparse.ArgumentParser() +parser.add_argument('--runner', required=True, type=Path) +parser.add_argument('--root', required=True, type=Path) +args, rest = parser.parse_known_args() + +class GuestTests(unittest.TestCase): + def run_guest(self, *argv, timeout=15, cwd='/workspace', variables=()): + # Empty host environment deliberately prevents secret inheritance. + return execute([str(args.runner.resolve()), str(args.root.resolve()), cwd, + *variables, '--', *argv], timeout=timeout) + + def test_argv_cwd_env_streams_and_exit(self): + result = self.run_guest('/bin/sh', '-c', + 'printf "%s|%s|%s" "$PWD" "$PROBE" "$1"; printf error >&2; exit 7', + 'probe', 'literal; $(false)', variables=('PROBE=value with spaces',)) + self.assertEqual(result.stdout, b'/workspace|value with spaces|literal; $(false)') + self.assertEqual(result.stderr, b'error') + self.assertEqual(result.returncode, 7) + + def test_real_fork_pipeline_and_persistent_files(self): + first = self.run_guest('/bin/sh', '-c', + 'printf persistent > saved; (printf child | cat) & wait; cat saved') + self.assertEqual((first.returncode, first.stdout), (0, b'childpersistent')) + second = self.run_guest('/bin/cat', '/workspace/saved') + self.assertEqual((second.returncode, second.stdout), (0, b'persistent')) + + def test_no_inherited_host_credentials(self): + result = self.run_guest('/bin/sh', '-c', 'printf "%s" "$HOME$OPENAI_API_KEY$ANTHROPIC_API_KEY"') + self.assertEqual((result.returncode, result.stdout), (0, b'')) + + def test_host_harness_timeout(self): + # This terminates the standalone host process. It is NOT mobile cancellation. + with self.assertRaises(subprocess.TimeoutExpired): + self.run_guest('/bin/sh', '-c', 'while :; do :; done', timeout=0.5) + + def test_streams_arrive_before_process_exit(self): + observed = [] + result = execute([str(args.runner.resolve()), str(args.root.resolve()), '/', '--', + '/bin/sh', '-c', 'printf first; sleep 1; printf second >&2'], + on_output=lambda stream, chunk, process: observed.append((stream, chunk, process.poll()))) + self.assertEqual(result.returncode, 0) + self.assertTrue(any(stream == 'stdout' and b'first' in chunk and status is None + for stream, chunk, status in observed)) + self.assertTrue(any(stream == 'stderr' and b'second' in chunk + for stream, chunk, _ in observed)) + + def test_output_flood_is_bounded(self): + with self.assertRaises(OutputLimit): + execute([str(args.runner.resolve()), str(args.root.resolve()), '/', '--', + '/bin/sh', '-c', 'yes flood'], max_bytes=65536) + + def test_oversized_argv_rejected_without_crash(self): + result = self.run_guest('/bin/echo', 'x' * 40000) + self.assertEqual(result.returncode, 70) + self.assertIn(b'boot failed', result.stderr) + + def test_invalid_cwd_fails(self): + self.assertNotEqual(self.run_guest('/bin/true', cwd='/does-not-exist').returncode, 0) + +unittest.main(argv=[__file__, *rest]) diff --git a/scripts/ish/test_probe_controls.py b/scripts/ish/test_probe_controls.py new file mode 100644 index 0000000..a55103b --- /dev/null +++ b/scripts/ish/test_probe_controls.py @@ -0,0 +1,17 @@ +#!/usr/bin/env python3 +"""Regression tests for feasibility tooling failures, without claiming guest evidence.""" +from pathlib import Path +import subprocess +import sys +import unittest + +class AuditControls(unittest.TestCase): + def test_runner_failure_is_not_reported_as_isolation(self): + result = subprocess.run([sys.executable, str(Path(__file__).with_name('audit_guest_boundary.py')), + '--runner', '/usr/bin/false', '--root', '/does-not-exist'], + capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0, result.stdout) + self.assertIn('INCONCLUSIVE', result.stderr) + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/ish/test_simulator.py b/scripts/ish/test_simulator.py new file mode 100644 index 0000000..935f225 --- /dev/null +++ b/scripts/ish/test_simulator.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Assert simulator guest execution AND survival of its UIKit host.""" +import argparse +import os +import plistlib +from pathlib import Path +import subprocess +import time + +p = argparse.ArgumentParser() +p.add_argument('--device', default='booted') +a = p.parse_args() +bundle = 'dev.moose.ishprobe' +def sim(*args, check=True): + return subprocess.run(['xcrun', 'simctl', *args], check=check, capture_output=True, text=True).stdout.strip() + +def launch(): + data = Path(sim('get_app_container', a.device, bundle, 'data')) / 'Documents' / 'probe' + (data / 'exit.plist').unlink(missing_ok=True) + launched = sim('launch', a.device, bundle) + pid = int(launched.rsplit(':', 1)[1]) + deadline = time.monotonic() + 20 + while not (data / 'exit.plist').exists(): + if time.monotonic() > deadline: + raise AssertionError('guest did not finish in 20s') + time.sleep(0.1) + with (data / 'exit.plist').open('rb') as f: + assert plistlib.load(f)['exitCode'] == 7 + output = (data / 'stdout').read_text() + assert 'cwd=/workspace env=explicit arg=literal; $(false)\n' in output, output + assert 'pipeline\n' in output, output + assert (data / 'stderr').read_text() == 'stderr-probe\n' + time.sleep(0.5) + os.kill(pid, 0) # PID 1's guest exit must not kill the UIKit host. + return output + +sim('terminate', a.device, bundle, check=False) +launch() +sim('terminate', a.device, bundle) +assert 'reopened=persistent' in launch() +print('PASS: native shell, streams, status, fork/pipeline, host survival, persistent reopen') From 45fb890b9ec879ad15ce3e5ce90be0600309fbb1 Mon Sep 17 00:00:00 2001 From: sugarforever Date: Tue, 20 Oct 2026 21:38:59 +0100 Subject: [PATCH 2/4] docs: record draft iSH probe delivery --- docs/superpowers/plans/2026-10-20-ios-ish.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-10-20-ios-ish.md b/docs/superpowers/plans/2026-10-20-ios-ish.md index 715c254..201773c 100644 --- a/docs/superpowers/plans/2026-10-20-ios-ish.md +++ b/docs/superpowers/plans/2026-10-20-ios-ish.md @@ -21,7 +21,9 @@ Host-process exit; guest-root host access; background descendants; unbounded out - [x] Write executable feasibility assertions first: guest shell, streams/status, persistence, fork, argv/cwd/env, bounded host timeout. - [x] Observe failure without a built guest kernel, then build pinned upstream and add the minimum native entrypoint needed. - [x] Build/run a standalone simulator application with the same core; record evidence and differences from host execution. -- [x] Resolve supervisor, credentials, filesystem isolation, cancellation and recovery before connecting Pi ExecutionEnv. If unresolved, leave product disabled and enumerate blocking gates in draft PR. -- [ ] Run project verification and Expo doctor, inspect diff, commit, push, create draft/ready PR according to evidence, attach PR. +- [x] Evaluate supervisor, credentials, filesystem isolation, cancellation and recovery before connecting Pi ExecutionEnv. Unresolved gates leave the product disabled and are enumerated in draft PR #7. +- [x] Run project verification and Expo doctor, inspect diff, commit, push, create draft/ready PR according to evidence, attach PR. Status: executable host/native probes complete. Product integration remains blocked by the red host-mount boundary audit and missing supervisor/cancellation/file contract. See the spec for the exact remaining gates. + +Delivery: https://github.com/sugarforever/moose/pull/7 (draft, attached to this chat). From dff1b1372a25d4df4bea666d1f9971e9ee34b02f Mon Sep 17 00:00:00 2001 From: sugarforever Date: Tue, 20 Oct 2026 22:49:36 +0100 Subject: [PATCH 3/4] feat: integrate optional native iSH Pi execution environment --- .gitattributes | 2 + .gitignore | 4 +- README.md | 6 +- docs/ios-ish-execution.md | 111 ++- docs/mobile-feasibility.md | 4 + docs/superpowers/plans/2026-10-20-ios-ish.md | 12 +- modules/moose-ish/.gitignore | 3 + modules/moose-ish/MooseISH.podspec | 15 + modules/moose-ish/README.md | 75 ++ modules/moose-ish/index.js | 2 + modules/moose-ish/ios/MooseISH.m | 286 +++++++ modules/moose-ish/package.json | 14 + modules/moose-ish/react-native.config.js | 16 + modules/moose-ish/tests/bridge_harness.m | 104 +++ .../tests/react/React/RCTBridgeModule.h | 6 + .../tests/react/React/RCTEventEmitter.h | 5 + modules/moose-ish/tests/run_bridge.py | 11 + .../moose-ish/tests/run_bridge_simulator.py | 41 + modules/moose-ish/tests/test_package.py | 13 + native/ish/NOTICE.md | 40 + native/ish/embedded-policy.patch | 539 +++++++++++++ native/ish/embedded.c | 265 +++++++ native/ish/embedded.h | 40 + native/ish/embedded_guest_test.c | 86 +++ native/ish/embedded_host.c | 32 + native/ish/embedded_simulator.m | 67 ++ native/ish/embedded_test.c | 303 ++++++++ package-lock.json | 9 + package.json | 5 +- scripts/ish/audit_guest_boundary.py | 4 +- scripts/ish/build_embedded.py | 91 +++ scripts/ish/prepare_native.py | 76 ++ scripts/ish/test_embedded.py | 42 ++ scripts/ish/verify.mjs | 31 + src/dev/FilesSmoke.tsx | 7 +- src/dev/files-smoke.ts | 31 +- src/platform/ish.ts | 41 + src/platform/open.ts | 5 +- src/runtime/engine.ts | 28 +- src/runtime/files.ts | 5 +- src/runtime/ish.ts | 707 ++++++++++++++++++ src/runtime/portable-entry.ts | 2 + src/skills/skills.ts | 14 +- tests/ish-native.test.ts | 330 ++++++++ tests/ish.test.ts | 108 +++ 45 files changed, 3547 insertions(+), 91 deletions(-) create mode 100644 .gitattributes create mode 100644 modules/moose-ish/.gitignore create mode 100644 modules/moose-ish/MooseISH.podspec create mode 100644 modules/moose-ish/README.md create mode 100644 modules/moose-ish/index.js create mode 100644 modules/moose-ish/ios/MooseISH.m create mode 100644 modules/moose-ish/package.json create mode 100644 modules/moose-ish/react-native.config.js create mode 100644 modules/moose-ish/tests/bridge_harness.m create mode 100644 modules/moose-ish/tests/react/React/RCTBridgeModule.h create mode 100644 modules/moose-ish/tests/react/React/RCTEventEmitter.h create mode 100644 modules/moose-ish/tests/run_bridge.py create mode 100644 modules/moose-ish/tests/run_bridge_simulator.py create mode 100644 modules/moose-ish/tests/test_package.py create mode 100644 native/ish/NOTICE.md create mode 100644 native/ish/embedded-policy.patch create mode 100644 native/ish/embedded.c create mode 100644 native/ish/embedded.h create mode 100644 native/ish/embedded_guest_test.c create mode 100644 native/ish/embedded_host.c create mode 100644 native/ish/embedded_simulator.m create mode 100644 native/ish/embedded_test.c create mode 100644 scripts/ish/build_embedded.py create mode 100644 scripts/ish/prepare_native.py create mode 100644 scripts/ish/test_embedded.py create mode 100644 scripts/ish/verify.mjs create mode 100644 src/platform/ish.ts create mode 100644 src/runtime/ish.ts create mode 100644 tests/ish-native.test.ts create mode 100644 tests/ish.test.ts diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..ff64ada --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +# Unified diff context lines intentionally contain a leading space. +native/ish/*.patch -whitespace diff --git a/.gitignore b/.gitignore index 7ac80bc..c8bef2b 100644 --- a/.gitignore +++ b/.gitignore @@ -2,8 +2,8 @@ node_modules/ .expo/ dist/ coverage/ -ios/ -android/ +/ios/ +/android/ .env* *.sqlite* src/generated/ diff --git a/README.md b/README.md index b2de977..b1e72b9 100644 --- a/README.md +++ b/README.md @@ -84,11 +84,11 @@ Skill 支持 SKILL.md 直链、GitHub 文件、目录和仓库根目录 URL。 检查点保存对话历史，不保存文件快照。恢复检查点会创建新对话和空工作区，不复制原对话文件，也不回滚磁盘内容。原对话的文件保持不变。 -单个文件限制为 8 MiB。只提供 read、write、edit，不支持 Shell、脚本执行或搜索。读取的文件内容会作为工具结果进入模型上下文，可能发送给当前模型服务商。 +单个文件限制为 8 MiB。默认的 Expo 环境只提供 read、write、edit，不支持 Shell、脚本执行或搜索。读取的文件内容会作为工具结果进入模型上下文，可能发送给当前模型服务商。 文件环境使用 npm 发布的 `pi-durable-expo-env@0.1.0`，版本与完整性由锁文件固定，不依赖同级目录的源码或本地 tarball。更新步骤、受控原生测试入口和验证记录见 [文件环境集成说明](docs/mobile-files.md)。 -iOS iSH 原生执行仍处于独立可行性探针阶段，尚未接入应用。构建步骤、验证结果和阻塞项见 [iOS iSH 实验](docs/ios-ish-execution.md)。 +iOS 可选用实验性的 iSH 本地 Linux 环境，让 Pi 文件工具和真实 POSIX shell 共用同一对话工作区。先运行 `npm run prepare:ish` 生成原生依赖，再用 `EXPO_PUBLIC_MOOSE_ISH=1 npm run ios` 重编开发客户端并启用。默认仍使用 Expo 文件环境；两种环境的文件不自动迁移。iSH 网络访问关闭，命令受输出、时间和资源限制，应用挂起时可能暂停。构建前提、验证结果、容量限制和 GPL 再分发义务见 [iOS iSH 执行环境](docs/ios-ish-execution.md)。 ## 验证 @@ -97,7 +97,7 @@ npm run verify npx expo-doctor ``` -`verify` 包含运行包生成、41 项单元与集成测试、TypeScript 检查、无 Node 全局变量的 Pi Durable / SQLite / Codex 模拟链路，以及 Android 和 iOS 的 Hermes 发布资源导出。 +`verify` 包含运行包生成、45 项单元与集成测试、TypeScript 检查、无 Node 全局变量的 Pi Durable / SQLite / Codex 模拟链路，以及 Android 和 iOS 的 Hermes 发布资源导出。 当前验证范围： diff --git a/docs/ios-ish-execution.md b/docs/ios-ish-execution.md index 8c6c886..6932341 100644 --- a/docs/ios-ish-execution.md +++ b/docs/ios-ish-execution.md @@ -1,86 +1,83 @@ -# iOS iSH execution design and acceptance +# Optional iOS iSH execution environment -Issue: https://github.com/sugarforever/moose/issues/6 +Issue: https://github.com/sugarforever/moose/issues/6 · implementation: https://github.com/sugarforever/moose/pull/7 -## Intended product +Moose can opt into a real local iSH-backed Pi Durable `ExecutionEnv` on iOS. The Hermes runtime calls a local React Native module; the module links the pinned iSH CPU/syscall core. The existing Expo filesystem remains the default, including on Android. There is no remote host, host-shell substitute, Node execution environment, or automatic credential inheritance. -An optional iOS ExecutionEnv runs real i386 Linux programs inside the Moose process through iSH's usermode CPU interpreter and syscall layer. File tools and shell must see the same persistent guest workspace. No model credentials, SQLite, SecureStore values, or host home directory may be mounted or inherited. Node execution environments must never enter Hermes. +## Build and enable -## Evidence boundary +Requires Apple Silicon macOS, Xcode with an installed iOS simulator runtime, Python 3, Meson, Ninja, Clang/LLVM **and LLD**, libarchive, pkg-config and git. Scripts do not install dependencies. LLD is a separate Homebrew package. -First build and run a pinned upstream kernel independently of React Native. A standalone native probe is a feasibility artifact, not an enabled Moose feature. Keep the existing Expo file environment until embedding satisfies all acceptance criteria. Do not ship a nonfunctional bash tool or silently substitute a host shell. +```sh +npm ci +PATH="$(brew --prefix lld)/bin:$PATH" npm run prepare:ish +EXPO_PUBLIC_MOOSE_ISH=1 npm run ios +``` + +Preparation downloads iSH at `c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c`, verifies Alpine 3.23.0 x86 minirootfs SHA-256, and builds arm64 device/simulator static libraries and an XCFramework. Generated binaries, rootfs and native projects are ignored. The local module autolinks only when prepared artifacts exist (or `MOOSE_ISH_NATIVE=1` explicitly forces it). Rebuild the development client after preparation or native changes; Expo Go cannot load this module. An enabled backend with a missing module fails explicitly instead of silently falling back. -Upstream source: https://github.com/ish-app/ish at c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c. README describes usermode x86 emulation, syscall translation, and precompiled threaded interpreter gadgets rather than generated executable code. `meson.build` separates static emulator, fakefs, and kernel libraries from the app frontend. `kernel/task.c` owns a global PID table; `fs/mount.c` owns a global mount list. Thread-local `current` is a task pointer, not an independent kernel instance. `kernel/exit.c` halts/unmounts the kernel when PID 1 dies. `xX_main_Xx.h` adds a host `exit` callback. Embedding must replace that callback and maintain a supervisor init; spawning another kernel per conversation in the same process is unsafe without upstream changes. +The flag selects a different persistent filesystem. Existing Expo workspace files are not migrated into iSH. Switching it off returns to the original Expo files. Within iSH each conversation owns a full private guest root; `/workspace` is shared by Pi read/write/edit and shell commands. Checkpoint branches get new roots and do not snapshot or roll back files. -## Licensing and redistribution +## Runtime ownership -`LICENSE.md` specifies GPLv3 plus `LICENSE.IOS`, with additional GPLv2 licensing for contributions after the identified commit and named earlier authors. Do not assume every historical contribution can be linked under GPLv2 alone. `LICENSE.IOS` permits otherwise GPL-compliant App Store distribution despite App Store terms; it explicitly retains source and license-text obligations. Before distributing a linked Moose binary, select a compatible whole-program license, include license texts/notices, and provide complete corresponding source and reproducible build instructions for the exact kernel, modifications, frontend and rootfs packages. Source-only feasibility scripts do not make that distribution decision. Rootfs package licenses and sources require a separate manifest. No prebuilt kernel/rootfs is committed here. +`native/ish/embedded.c` owns one neutral supervisor and serializes guest commands. `embedded-policy.patch` adds an inherited task domain and selects the filesystem root by domain, instead of pretending process-global iSH PID/mount tables are separate kernel instances. The guest runs as uid/gid 501; setuid/setgid exec elevation is suppressed. Host mounts/unmounts, sockets, ptrace, privileged device/chroot operations, cross-domain signals, and cross-process resource-limit changes are denied. No procfs or Moose/SecureStore/SQLite directory is mounted. Only explicit argv/env and three command descriptors cross the boundary. -## Required architecture +Cancellation targets every task in the domain, including descendants which changed session or process group. The supervisor survives command exit. Queued cancellation is checked before admission, timeouts include queue time, and completion waits for descendant teardown/reaping. The core uses dedicated pthreads; guest `pthread_exit` must never run on a GCD worker. The original probe caught that lifecycle mistake natively. -One native kernel supervisor per app process, serialized initialization and shutdown. Each conversation gets an independent persistent guest filesystem with a non-root guest identity and restricted mount/syscall policy. A guest chroot alone is not sufficient if the guest retains root or can mount host paths. iSH's emulator shares Moose's native privileges: it is not an OS security boundary. Disable dangerous host mounts and unnecessary network/devices before model-directed commands are enabled. +The original unmodified iSH probe reproduced `mount -t real` exposing a controlled host directory. The embedded policy denies that operation; the executable boundary audit remains available for comparison. Review and tests also found upstream PID-boundary, negative process-group locking, and truncate64 host-abort paths; the patch includes their regression fixes. This is an experimental embedded interpreter sharing the app's OS process, not a separately sandboxed Linux VM. -The bridge must allocate per-command stdin/stdout/stderr descriptors, encode argv and env as NUL-separated strings with length validation, set guest cwd, stream bounded output with stream identity and omitted-byte accounting, and return exit status. Only explicit environment variables are passed. Cancellation/timeout must signal the entire guest process group, reap descendants, close descriptors, and leave the supervisor and other conversations alive. Never use host process termination as cancellation inside Moose. +`modules/moose-ish/ios/MooseISH.m` clones the complete fakefs seed atomically per workspace, pumps stdin/stdout/stderr concurrently, emits byte-stream events before promise settlement and persists a host-side command journal. A process-wide coordinator owns jobs, admission and the journal across React Native bridge generations; invalidating a generation cancels only its commands and prevents new admission through that module. Guest files and fakefs metadata always change through iSH. Pi's portable file adapter is backed by controlled guest filesystem commands, never by writing directly into fakefs `data` through Expo. -Persist workspace files and command intent/status, not CPU registers or credentials. After process death, pending commands become interrupted; do not replay commands with possible side effects. Reconnect by stable command ID where the native process still exists. iOS suspension pauses execution; foreground recovery must reconcile state. No background execution guarantee. +`src/runtime/ish.ts` implements shell strings/argv, cwd/env, independent UTF-8 stream decoders, cancellation, spill files, binary file tools, no-follow reader checks, flush and bounded polling watchers. Complete file operations serialize with shell operations. Polling watches can miss changes that occur and are undone between polls, and wait behind a running guest command. Pi `bash` uses the guest POSIX `/bin/sh`; Bash extensions and unsupported Linux syscalls are not promised. Network access is unavailable. -## Acceptance gates +## Durability and limits -- Real shell pipeline, fork/wait, argv with metacharacters, cwd/env, separate streaming stdout/stderr, and nonzero status in host harness and iOS simulator. -- File read/write/edit and shell share files; reopen preserves them; two conversations and symlink/absolute-path attacks cannot expose another conversation or host data. -- Timeout, cancellation, output flood, descendant cleanup, repeated commands, supervisor survival, and initialization races verified natively. -- Pi Durable real Harness integration and environment conformance with controlled model transport. -- Native device build and simulator interaction; physical-device evidence reported separately. No JIT entitlement required. -- Runtime bundle, tests, types, portable smoke, both resource exports, Expo doctor; native compilation is distinct from resource export. +Pi shell tools are explicitly `replay: unsafe`. Before shell effects, the tool stores a stable native command ID with Pi's durable `api.memo`. The native journal stores only ID/workspace/status/exit code, never argv, stdin, environment or model credentials. IDs are never evicted or automatically re-executed. A running record becomes interrupted when the process-wide coordinator first opens after process death; malformed/unreadable journals disable execution. File contents persist, but CPU state and a killed process cannot be resumed. The existing Pi transcript records interrupted unsafe tools; the user can deliberately rerun with a new submission. -Until all security/lifecycle gates pass, deliver as draft with actual feasibility results and explicit missing implementation. Do not change the product's advertised capabilities. +- Pi file transfers: 8 MiB per file, inherited from the portable file adapter. +- Native capture: 16 MiB combined stdout/stderr; overflow cancels and reports an error. Input: 16 MiB per command, 32 MiB across admitted jobs. +- Native admission: four jobs, including queued jobs and pending promise settlements. Busy rejection happens before guest effects or journaling. +- Journal and pre-registration cancellation sets: 10,000 IDs each, fail closed at capacity. Internal filesystem helper executions also consume journal IDs. This is a finite experimental ledger, not an indefinitely growing history; there is no automatic destructive reset or eviction. +- Native core: 32 task slots per active domain, including threads and unreaped zombies; 16 MiB per regular file; 256 MiB combined guest mapping backing; 16 MiB syscall transfer buffers and at most 1,024 vector entries. Checks run in actual allocation/I/O paths. Mapping backing is not a bound on total app RSS, emulator metadata or every host allocation. Unsupported `RLIMIT` metadata is not treated as enforcement. +- Commands default to 120 seconds in the Pi adapter; explicit timeout is at most 86,400 seconds. -## Running the executable probes +These bounds do not impose an aggregate on-disk workspace quota. iOS suspension can pause the entire app; foreground recovery reconciles persisted Pi/native state and does not promise background execution. No keychain credential is passed into the guest or stored in its environment. -These scripts currently target Apple Silicon macOS. Prerequisites: Xcode with an installed iOS simulator runtime, Python 3, Meson, Ninja, LLVM **and LLD** (Homebrew packages are separate), libarchive, pkg-config, git. The scripts do not install dependencies. LLVM is only used to cross-compile the guest i386 VDSO; Apple Clang builds host and iOS code. For example, put Homebrew LLD on PATH before building: +## Verification ```sh -PATH="$(brew --prefix lld)/bin:$PATH" python3 scripts/ish/build_probe.py \ - --cache /tmp/moose-ish-probe --simulator -xcrun simctl install booted /tmp/moose-ish-probe/MooseISHProbe.app -python3 scripts/ish/test_simulator.py -python3 scripts/ish/audit_guest_boundary.py \ - --runner /tmp/moose-ish-probe/host/moose-ish-probe \ - --root /tmp/moose-ish-probe/root/data -``` +# Host: builds actual patched iSH, same-kernel lifecycle/resource tests, +# production Objective-C bridge tests, and real Pi/Harness/conformance tests. +PATH="$(brew --prefix lld)/bin:$PATH" npm run test:ish + +# Full app JS/portable/Hermes exports; native tests are separate. +npm run verify +npx expo-doctor -Use a dedicated cache outside the repository. The build script fetches the exact iSH commit without changing it and verifies the Alpine 3.23.0 x86 minirootfs SHA-256 against the pinned value. Upstream submodules are not needed for these three static libraries. Rootfs checksum source: https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86/alpine-minirootfs-3.23.0-x86.tar.gz.sha256. +# Controlled model transport inside the real iOS app; no real account/request. +EXPO_PUBLIC_MOOSE_ISH=1 EXPO_PUBLIC_MOOSE_FILES_SMOKE=1 npm run ios +``` -`native/ish/probe.c` directly boots the real kernel with bounded argv/env packing. `host.c` is a standalone process adapter; its exit callback may terminate that executable. `simulator.m` instead owns a dedicated pthread and records status without terminating UIKit. iSH tasks call `pthread_exit`; running them on GCD worker threads caused the original native survival regression. A guest PID 1 exit still shuts down the one kernel, so the probe deliberately boots only once per app launch. It is not the required long-lived supervisor. +Use `MOOSE_ISH_CACHE=/absolute/external/cache` to override the host verifier cache. `prepare:ish` uses `/tmp/moose-ish-embedded`; the lower-level builder also accepts explicit external paths. The host test transport launches the real embedded CLI per call; it does not itself establish same-process lifecycle or mobile cancellation. Those are independently checked by the C suite and the production Objective-C bridge on host/simulator. -Host capture in `scripts/ish/host_runner.py` streams separate bytes, limits combined captured output to 1 MiB, and kills/reaps the **entire standalone host process** on timeout or a callback failure. The flood test uses a 64 KiB bound. This is not a solution for per-command mobile cancellation. Simulator commands are fixed, finite probes; they write stdout/stderr to app-private files and do not implement a bounded streaming bridge. Do not put arbitrary model commands into this app. +Additional native commands are available in `scripts/ish/test_embedded.py`, `modules/moose-ish/tests/run_bridge.py` and `run_bridge_simulator.py`. The last two use a thin React registration shim for standalone harnesses while compiling/executing the actual production bridge. The Moose app smoke additionally validates the real React Native/Hermes bridge with controlled OpenAI SSE, Pi tools, SQLite reopen and checkpoint isolation. Temporary simulator bundles use distinct `dev.moose.ish*test` identifiers. Test artifacts and generated rootfs are not APKs/IPAs and are not committed. -The simulator test launches the native app twice, checks stream contents/status, verifies the host remains alive after guest exit, and checks persistence on the second launch. The app remains installed under `dev.moose.ishprobe`, separate from Moose. Test artifacts are in its `Documents/probe`. Remove the disposable app with `xcrun simctl uninstall booted dev.moose.ishprobe` when finished. The cache contains local GPL-linked test binaries and rootfs; do not publish that cache without satisfying distribution obligations. +## Validation record, 2026-10-20 -## Actual results, 2026-10-20 +- Real host adapter/Harness suite: 29 passed, including the 24 applicable official Pi environment conformance cases (optional `listFiles` is not implemented) and five integration cases, including a 9 MiB spilled shell result. +- Same-kernel native suite: initialization races, repeated commands, isolated roots, argv/env/stdio, timeout/cancel and changed-session descendants; independent-process file reopen. +- Production bridge suite: separate streaming/exit, 8 MiB binary transfer, event ordering, timeout/flood cancellation, stable IDs/journal, early cancellation, resource admission and overlapping bridge generations/invalidation. +- Real Moose Debug iOS build and iPhone 17 Pro / iOS 26.2 simulator: controlled OpenAI SSE → Pi write/edit/bash/read → iSH passed, as did transcript persistence, separate checkpoint workspace, file/SQLite reopen and traversal rejection. +- Standard `npm test`: 45 passed; the 29 native adapter tests are skipped unless the explicit native verifier supplies the real core. TypeScript and runtime bundle passed. +- Expo Doctor: 19/20; React Native Directory reports the existing TCP library as untested and the private local `moose-ish` module as unknown. These metadata warnings are not runtime test results. +- Upstream syscall-cast/gadget-alignment and Xcode script warnings remain. Physical-device operation, signed device app/IPA, background/foreground device behavior and real-account/model requests are not claimed. -| Evidence | Result | -| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Fresh scripted host + arm64 iOS simulator build | Passed from a new external cache; no upstream source patches | -| Real Alpine x86 guest tests | 8 passed: argv/cwd/env/exit, fork/pipeline/persistence, empty inherited environment, host timeout, early streaming, output bound, oversize argv, invalid cwd | -| Simulator native execution, iPhone 17 Pro / iOS 26.2, Xcode 26.2 | Passed: shell, literal argv, cwd/env, separate stream files, exit 7, fork/pipeline, UIKit host survival, relaunch persistence | -| Probe tooling regression | Passed: runner failure produces nonzero inconclusive status, never isolation success | -| Host boundary audit | **FAILED / BLOCKER**: guest mounted a newly created host temporary directory and read only its controlled sentinel | -| Project `npm run verify` | Passed: runtime bundle, 41 tests, TypeScript, portable smoke, Android and iOS Hermes resource exports | -| `npx expo-doctor` | 19/20; existing react-native-tcp-socket New Architecture metadata warning | -| Native compiler warnings | Upstream syscall function-pointer casts and assembly gadget alignment warnings; builds completed, warnings are not claimed resolved | -| Physical iPhone, signed device build, app background/foreground recovery | Not performed | -| Pi Durable integration, native bridge, real model calls | Not implemented / not performed; no credentials or billable requests used | +Final verification after native resource hardening is recorded in the PR validation notes. Earlier standalone `probe.*` and `simulator.m` files preserve the original one-shot feasibility reproducer; they are not linked into Moose. -The audit is an intentionally red product acceptance gate, separate from the functional probe tests. It exits nonzero when the sentinel becomes visible. It must not be inverted into a passing security test. `fs/mount.c` registers `realfs`, and `sys_mount` passes guest-provided source paths to `realfs_mount`, which resolves and opens host paths. Dropping the guest uid alone is not a demonstrated fix: the inspected syscall does not perform the expected privilege check. This reproducer is why unmodified upstream must not be wired to Moose's model tools. +## Source, licensing and redistribution -## Remaining work before #6 can be completed +The pin comes from [official iSH](https://github.com/ish-app/ish); [its license notice](https://github.com/ish-app/ish/blob/c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c/LICENSE.md) specifies GPLv3 plus [LICENSE.IOS](https://github.com/ish-app/ish/blob/c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c/LICENSE.IOS), with additional GPLv2 grants for specified contributions. Do not infer a GPLv2-only license for all historical source. The App Store exception retains other GPL obligations. -1. Fork/pin an embedded core policy that denies host filesystem mounts and unintended syscall/device/network capabilities; run native adversarial isolation tests, including other conversations and Moose files. Re-audit path and socket boundaries, not only `mount`. -2. Implement one surviving init/supervisor and per-conversation guest identities/root directories. Keep all shared mounts and tasks behind a serialized native owner. Verify repeated commands and independent process-group cancellation without host exit or kernel shutdown. -3. Implement native descriptor-based streaming with byte bounds, output omission accounting, exit-status events, timeouts, cancellation and idempotent cleanup. Handle callback errors and descendants holding output descriptors. -4. Implement Pi 1.3.0's actual `ExecutionEnv` file contract (`BinaryReader`, line scanning, directory reads, mutation and flush semantics) against the same guest filesystem. The fakefs metadata database means exposing its `data` directory through Expo file APIs is not equivalent to guest file operations. Run Pi's environment conformance and real Harness tests with controlled SSE. -5. Persist stable command IDs/status and reconcile interrupted commands on app restart; do not transparently rerun side effects. Exercise background/foreground and termination on native devices. -6. Decide a compatible linked-app distribution license and corresponding-source delivery for Moose + iSH + bundled packages before distributing binaries. Keep this PR draft until runtime gates are met; do not close #6 as implemented on the strength of these probes. +Local preparation bundles exact upstream notices and a source hash/rebuild manifest. That manifest is explicitly not complete corresponding source. Before distributing **any binary that links iSH**, even with the backend flag disabled, choose a compatible whole-program license and supply applicable license texts, complete corresponding app/kernel/package source and reproducible build instructions. Alpine's package licenses/sources require their own inventory. This PR does not publish binaries, choose the linked app's distribution license or establish App Store approval. -Apple's background-execution guidance remains applicable to an embedded interpreter: https://developer.apple.com/documentation/uikit/extending-your-app-s-background-execution-time. No probe result establishes continuous execution while iOS suspends the application. +Rootfs checksum source: [Alpine 3.23.0 x86 checksum](https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86/alpine-minirootfs-3.23.0-x86.tar.gz.sha256). iOS lifecycle guidance: [Apple background execution](https://developer.apple.com/documentation/uikit/extending-your-app-s-background-execution-time). diff --git a/docs/mobile-feasibility.md b/docs/mobile-feasibility.md index ec74144..a5d045a 100644 --- a/docs/mobile-feasibility.md +++ b/docs/mobile-feasibility.md @@ -55,3 +55,7 @@ ## 本地 Expo 文件环境集成 Moose 现通过 npm 发布的 `pi-durable-expo-env` 注入设备文件系统，复用 Pi 的 read、write、edit 工具。平台工厂位于 `src/platform/open.ts`，运行时只接收便携接口。工作区按对话隔离，检查点不保存或回滚文件。详细语义、原生配置和本次验证见 [文件环境集成说明](mobile-files.md)。 + +## 可选 iOS iSH 环境 + +`EXPO_PUBLIC_MOOSE_ISH=1` 可启用原生 iSH 后端，通过便携 `ExecutionEnv` 提供文件工具和 POSIX shell。核心通过显式桥接注入，不引入 Node 执行环境；Android 和默认 Expo 文件环境不启用 shell。原生编译、受控 SSE 工具链、权限和资源边界见 [iOS iSH 执行环境](ios-ish-execution.md)。 diff --git a/docs/superpowers/plans/2026-10-20-ios-ish.md b/docs/superpowers/plans/2026-10-20-ios-ish.md index 201773c..fe19b3d 100644 --- a/docs/superpowers/plans/2026-10-20-ios-ish.md +++ b/docs/superpowers/plans/2026-10-20-ios-ish.md @@ -24,6 +24,16 @@ Host-process exit; guest-root host access; background descendants; unbounded out - [x] Evaluate supervisor, credentials, filesystem isolation, cancellation and recovery before connecting Pi ExecutionEnv. Unresolved gates leave the product disabled and are enumerated in draft PR #7. - [x] Run project verification and Expo doctor, inspect diff, commit, push, create draft/ready PR according to evidence, attach PR. -Status: executable host/native probes complete. Product integration remains blocked by the red host-mount boundary audit and missing supervisor/cancellation/file contract. See the spec for the exact remaining gates. +## Same-PR implementation continuation + +- [x] Add a neutral native supervisor, per-workspace roots and restricted task domains; cover cancellation, descendants, host-access boundaries and resource limits with real guest regressions. +- [x] Add an optional React Native bridge with bounded binary streams, command IDs and a persistent interrupted-command journal. +- [x] Implement the actual Pi ExecutionEnv file and shell contract and durable unsafe bash tool; keep the default Expo environment unchanged. +- [x] Run the official applicable Pi conformance tests and controlled model/Harness integration against the actual core. +- [x] Build and operate the actual React Native/Hermes simulator application through controlled SSE, file tools and bash. +- [x] Complete the native bridge reload/journal ownership regression and fix; host and UIKit tests plus final project verification pass. +- [x] Complete the final Moose native rebuild/simulator smoke and update the same draft PR #7 with final evidence and remaining device/distribution limits. + +Status: actual optional Pi ExecutionEnv integration and the cross-generation native coordinator lifecycle fix are implemented and verified, including the final full-app rebuild and simulator smoke. The updated spec records exact experimental resource limits and untested physical-device/distribution paths. Delivery: https://github.com/sugarforever/moose/pull/7 (draft, attached to this chat). diff --git a/modules/moose-ish/.gitignore b/modules/moose-ish/.gitignore new file mode 100644 index 0000000..6603dc3 --- /dev/null +++ b/modules/moose-ish/.gitignore @@ -0,0 +1,3 @@ +vendor/ +__pycache__/ +*.pyc diff --git a/modules/moose-ish/MooseISH.podspec b/modules/moose-ish/MooseISH.podspec new file mode 100644 index 0000000..2601852 --- /dev/null +++ b/modules/moose-ish/MooseISH.podspec @@ -0,0 +1,15 @@ +Pod::Spec.new do |s| + s.name = 'MooseISH' + s.version = '0.1.0' + s.summary = 'Optional in-process iSH execution for Moose' + s.homepage = 'https://github.com/sugarforever/moose' + s.license = { :type => 'See bundled iSH licenses; redistribution requires whole-program review' } + s.author = 'Moose contributors' + s.source = { :git => 'https://github.com/sugarforever/moose.git' } + s.platform = :ios, '15.1' + s.source_files = 'ios/*.{h,m}' + s.vendored_frameworks = 'vendor/MooseISHCore.xcframework' + s.resource_bundles = { 'MooseISHResources' => ['vendor/root', 'vendor/licenses', 'vendor/source-manifest.json'] } + s.libraries = 'sqlite3', 'pthread' + s.dependency 'React-Core' +end diff --git a/modules/moose-ish/README.md b/modules/moose-ish/README.md new file mode 100644 index 0000000..1163d61 --- /dev/null +++ b/modules/moose-ish/README.md @@ -0,0 +1,75 @@ +# Optional iOS embedded execution + +This private local React Native package registers `NativeModules.MooseISH` on iOS. +Android has no module. The application chooses whether to use it; adding the JS +package does not produce native artifacts or enable the execution backend. + +From the repository root, with Xcode, Meson, Ninja, LLVM/LLD, pkg-config and +libarchive installed: + +```sh +python3 scripts/ish/prepare_native.py --cache /absolute/path/outside/repository +npx expo run:ios +``` + +Preparation downloads the pinned iSH source and checksum-verified Alpine rootfs, +builds device and arm64 simulator static libraries, creates an XCFramework, and +copies the complete fakefs seed into ignored `vendor/`. Native dependency changes +require rebuilding the development client. The pod bundles the seed and licenses; +first use clones the entire fakefs tree into app-private Application Support. +Cleanup stops a workspace's commands and preserves its files. + +`prepare(workspaceId)`, `execute(workspaceId, commandId, argv, cwd, env, +inputBase64, timeoutMs)`, `cancel(commandId)`, `cleanup(workspaceId)`, and +`commandStatus(commandId)` return promises. `execute` resolves to `{exitCode, +stdout, stderr, omittedBytes, cancelled, timedOut}`; stdout/stderr are base64. +`MooseISHOutput` events contain `{commandId, stream, data}` where data is base64. +Events precede settlement. The caller must register its listener before execute. +Capture is bounded at 16 MiB combined; overflow cancels the command and reports +omitted bytes. Stdin accepts up to 16 MiB and is pumped concurrently with output. +Environment variables are explicit, never inherited from the app. + +Admission allows at most four running/queued commands and 32 MiB combined +retained stdin. Capacity rejection uses `spawn_error`, starts no guest command, +and does not record the rejected ID. Retained stdin is released before new +admission after completion. The journal retains at most 10,000 lifetime command +IDs, without eviction; reaching that bound disables new commands while keeping +existing duplicate protection and status queries. Unreadable, malformed or +oversized journals fail closed. Pending cancellation IDs are also bounded at +10,000. These are explicit evaluation limits rather than a journal pruning +policy. + +A process-wide coordinator owns workspace storage, journal, jobs and admission +limits across React Native module generations. Reloading the JavaScript bridge +does not reclassify live native commands as interrupted or create a second +journal writer. Invalidating a module cancels its own generation’s jobs, stops +its output events and rejects further commands from that generation. + +IDs contain only ASCII letters, digits, `_`, and `-`, up to 128 bytes. The journal +contains IDs, workspace, status and exit code only. Recorded IDs are never +executed twice, including failed/interrupted commands; use a fresh ID for an +intentional rerun. On process restart running records become interrupted. Final +output is returned to the live caller rather than persisted by this module. + +Local native verification (build the real host core first): + +```sh +python3 scripts/ish/build_embedded.py --cache /absolute/external/cache --sdk macosx --output /tmp/moose-core +python3 modules/moose-ish/tests/run_bridge.py --core /tmp/moose-core +python3 modules/moose-ish/tests/test_package.py +python3 modules/moose-ish/tests/run_bridge_simulator.py --core /tmp/moose-core-simulator --output /tmp/moose-bridge-test +``` + +The host harness compiles the production Objective-C bridge against Foundation +and the actual embedded library. Its React shim substitutes registration/events, +not the kernel. The separate simulator harness installs a disposable UIKit application under +`dev.moose.ishbridgetest`; remove it with `xcrun simctl uninstall booted +dev.moose.ishbridgetest`. Neither harness establishes physical-device operation. + +Generated binaries are for local evaluation. `vendor/source-manifest.json` +records pins and local build-input hashes; it is not complete corresponding +source. Before redistributing any linked app, provide exact whole-app source, +build instructions, all kernel changes, upstream license texts, and rootfs +package licenses/sources, and resolve a compatible whole-program license. See +`docs/ios-ish-execution.md` and `native/ish/NOTICE.md`. Nothing here publishes an +app or waives those obligations. diff --git a/modules/moose-ish/index.js b/modules/moose-ish/index.js new file mode 100644 index 0000000..03cfaff --- /dev/null +++ b/modules/moose-ish/index.js @@ -0,0 +1,2 @@ +// NativeModules.MooseISH is consumed only by the opt-in iOS adapter. +module.exports = {}; diff --git a/modules/moose-ish/ios/MooseISH.m b/modules/moose-ish/ios/MooseISH.m new file mode 100644 index 0000000..97c6cbc --- /dev/null +++ b/modules/moose-ish/ios/MooseISH.m @@ -0,0 +1,286 @@ +#import +#import +#import +#include "embedded.h" +#include +#include +#include +#include +#include +#include + +static const NSUInteger CaptureLimit = 16 * 1024 * 1024; +static const NSUInteger ActiveJobLimit = 4; +static const NSUInteger RetainedInputLimit = 32 * 1024 * 1024; +static const NSUInteger JournalRecordLimit = 10000; +@interface MooseISHJob : NSObject +@property NSString *identifier; +@property NSString *workspace; +@property NSString *generation; +@property NSData *input; +@property moose_ish_command *command; +@property NSLock *lock; +@property BOOL finished; +@property moose_ish_result result; +@property int inputFD; +@property int outputFD; +@property int errorFD; +@property RCTPromiseResolveBlock resolve; +@property RCTPromiseRejectBlock reject; +@end +@implementation MooseISHJob +@end + +@interface MooseISHCoordinator : NSObject +@property NSLock *lock; +@property NSMutableDictionary *jobs; +@property NSMutableDictionary *journal; +@property NSMutableSet *cancelledIDs; +@property NSMutableSet *closing; +@property NSString *base; +@property NSString *journalPath; +@property BOOL journalUsable; +- (instancetype)initWithBase:(NSString *)base; +- (BOOL)saveJournal; +@end +@interface MooseISH : RCTEventEmitter +@property MooseISHCoordinator *coordinator; +@property NSString *generation; +@property BOOL invalidated; +- (instancetype)initWithBase:(NSString *)base; +- (void)runJob:(MooseISHJob *)job; +@end + +static BOOL validID(id value) { + if (![value isKindOfClass:NSString.class] || [value length] < 1 || [value length] > 128) return NO; + static NSCharacterSet *invalid; + static dispatch_once_t once; + dispatch_once(&once, ^{ invalid = [[NSCharacterSet characterSetWithCharactersInString:@"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-"] invertedSet]; }); + return [value rangeOfCharacterFromSet:invalid].location == NSNotFound; +} +static BOOL validString(id value, NSUInteger limit) { + return [value isKindOfClass:NSString.class] && [value lengthOfBytesUsingEncoding:NSUTF8StringEncoding] <= limit && [value rangeOfString:@"\0"].location == NSNotFound; +} +static void *runCore(void *raw) { + @autoreleasepool { + MooseISHJob *job = (__bridge_transfer MooseISHJob *)raw; + moose_ish_result result = {0}; + int rc = moose_ish_command_run(job.command, &result); + if (rc && !result.error) result.error = rc; + [job.lock lock]; job.result = result; job.finished = YES; [job.lock unlock]; + } + return NULL; +} +@interface MooseISHRunner : NSObject +@property MooseISH *bridge; +@property MooseISHJob *job; +@end +@implementation MooseISHRunner +@end +static void *runBridge(void *raw) { + @autoreleasepool { + MooseISHRunner *runner = (__bridge_transfer MooseISHRunner *)raw; + [runner.bridge runJob:runner.job]; + } + return NULL; +} + +@implementation MooseISHCoordinator +- (instancetype)initWithBase:(NSString *)base { + if ((self = [super init])) { + _lock = [NSLock new]; _jobs = [NSMutableDictionary new]; _cancelledIDs = [NSMutableSet new]; _closing = [NSMutableSet new]; + _base = [base copy]; + [[NSFileManager defaultManager] createDirectoryAtPath:_base withIntermediateDirectories:YES attributes:@{NSFileProtectionKey:NSFileProtectionCompleteUntilFirstUserAuthentication} error:nil]; + _journalPath = [_base stringByAppendingPathComponent:@"commands.json"]; + NSNumber *journalBytes = [[[NSFileManager defaultManager] attributesOfItemAtPath:_journalPath error:nil] objectForKey:NSFileSize]; + NSData *data = journalBytes.unsignedLongLongValue <= 8 * 1024 * 1024 ? [NSData dataWithContentsOfFile:_journalPath] : nil; + id decoded = data ? [NSJSONSerialization JSONObjectWithData:data options:NSJSONReadingMutableContainers error:nil] : nil; + _journalUsable = [decoded isKindOfClass:NSDictionary.class] || ![[NSFileManager defaultManager] fileExistsAtPath:_journalPath]; + _journal = [decoded isKindOfClass:NSDictionary.class] ? [decoded mutableCopy] : [NSMutableDictionary new]; + if (_journal.count > JournalRecordLimit) _journalUsable = NO; + for (NSString *key in [_journal allKeys]) { + NSDictionary *record = _journal[key]; + if (!validID(key) || ![record isKindOfClass:NSDictionary.class] || !validID(record[@"workspace"]) || ![@[@"running",@"interrupted",@"completed",@"failed"] containsObject:record[@"status"] ?: @""]) { _journalUsable = NO; continue; } + for (id field in record) if (![@[@"workspace",@"status",@"exitCode"] containsObject:field]) _journalUsable = NO; + if ([record[@"status"] isEqual:@"running"]) _journal[key] = @{@"workspace":record[@"workspace"] ?: @"", @"status":@"interrupted"}; + } + if (_journalUsable) [self saveJournal]; + } + return self; +} +- (BOOL)saveJournal { + NSData *data = [NSJSONSerialization dataWithJSONObject:self.journal options:0 error:nil]; + return data && [data writeToFile:self.journalPath options:NSDataWritingAtomic error:nil]; +} +@end +static MooseISHCoordinator *coordinatorForBase(NSString *base) { + static NSLock *registryLock; + static NSMutableDictionary *registry; + static dispatch_once_t once; + dispatch_once(&once, ^{ registryLock=[NSLock new]; registry=[NSMutableDictionary new]; }); + NSString *key=[base stringByStandardizingPath]; + [registryLock lock]; + MooseISHCoordinator *coordinator=registry[key]; + if(!coordinator){coordinator=[[MooseISHCoordinator alloc] initWithBase:key];registry[key]=coordinator;} + [registryLock unlock]; + return coordinator; +} + +@implementation MooseISH +RCT_EXPORT_MODULE(MooseISH) ++ (BOOL)requiresMainQueueSetup { return NO; } +- (NSArray *)supportedEvents { return @[@"MooseISHOutput"]; } +- (instancetype)init { + NSString *support=NSSearchPathForDirectoriesInDomains(NSApplicationSupportDirectory, NSUserDomainMask, YES).firstObject; + return [self initWithBase:[support stringByAppendingPathComponent:@"MooseISH"]]; +} +// Internal initializer lets native tests isolate storage while exercising the same coordinator. +- (instancetype)initWithBase:(NSString *)base { + if((self=[super init])){_coordinator=coordinatorForBase(base);_generation=NSUUID.UUID.UUIDString;} + return self; +} +- (void)invalidate { + [self.coordinator.lock lock]; + self.invalidated=YES; + for(MooseISHJob *job in self.coordinator.jobs.allValues) + if([job.generation isEqual:self.generation] && job.command)moose_ish_command_cancel(job.command); + [self.coordinator.lock unlock]; + [super invalidate]; +} +- (BOOL)saveJournal { return [self.coordinator saveJournal]; } +- (NSString *)rootFor:(NSString *)workspace { + return [[self.coordinator.base stringByAppendingPathComponent:workspace] stringByAppendingPathComponent:@"root"]; +} +- (BOOL)prepareRoot:(NSString *)workspace error:(NSError **)error { + NSString *destination = [self rootFor:workspace]; + NSFileManager *fm = NSFileManager.defaultManager; + if ([fm fileExistsAtPath:[destination stringByAppendingPathComponent:@"data"]]) return YES; + NSBundle *bundle = [NSBundle bundleWithPath:[[NSBundle bundleForClass:self.class] pathForResource:@"MooseISHResources" ofType:@"bundle"]]; + NSString *seed = [bundle pathForResource:@"root" ofType:nil]; + if (!seed) { if (error) *error = [NSError errorWithDomain:@"MooseISH" code:1 userInfo:@{NSLocalizedDescriptionKey:@"Embedded rootfs is missing; run prepare:ish and rebuild iOS."}]; return NO; } + NSString *parent = [destination stringByDeletingLastPathComponent]; + if (![fm createDirectoryAtPath:parent withIntermediateDirectories:YES attributes:nil error:error]) return NO; + NSString *temporary = [parent stringByAppendingPathComponent:NSUUID.UUID.UUIDString]; + if (![fm copyItemAtPath:seed toPath:temporary error:error]) { [fm removeItemAtPath:temporary error:nil]; return NO; } + if (![fm moveItemAtPath:temporary toPath:destination error:error]) { [fm removeItemAtPath:temporary error:nil]; return NO; } + return YES; +} +RCT_REMAP_METHOD(prepare, prepare:(NSString *)workspace resolver:(RCTPromiseResolveBlock)resolve rejecter:(RCTPromiseRejectBlock)reject) { + if (!validID(workspace)) { reject(@"invalid_argument", @"Invalid workspace ID", nil); return; } + [self.coordinator.lock lock]; if(self.invalidated){[self.coordinator.lock unlock];reject(@"spawn_error",@"Native bridge generation is invalidated",nil);return;} NSError *error = nil; BOOL ok = [self prepareRoot:workspace error:&error]; [self.coordinator.lock unlock]; + if (ok) resolve(nil); else reject(@"prepare_failed", error.localizedDescription, error); +} +RCT_REMAP_METHOD(commandStatus, commandStatus:(NSString *)identifier resolver:(RCTPromiseResolveBlock)resolve rejecter:(RCTPromiseRejectBlock)reject) { + if (!validID(identifier)) { reject(@"invalid_argument", @"Invalid command ID", nil); return; } + [self.coordinator.lock lock]; NSDictionary *record = [self.coordinator.journal[identifier] copy]; [self.coordinator.lock unlock]; resolve(record); +} +RCT_REMAP_METHOD(execute, execute:(NSString *)workspace commandId:(NSString *)identifier argv:(NSArray *)argv cwd:(NSString *)cwd env:(NSDictionary *)env inputBase64:(NSString *)inputBase64 timeoutMs:(NSNumber *)timeout resolver:(RCTPromiseResolveBlock)resolve rejecter:(RCTPromiseRejectBlock)reject) { + BOOL valid = validID(workspace) && validID(identifier) && [argv isKindOfClass:NSArray.class] && argv.count > 0 && argv.count <= 256 && validString(cwd, 4096) && [cwd hasPrefix:@"/"] && [env isKindOfClass:NSDictionary.class] && env.count <= 256 && [timeout isKindOfClass:NSNumber.class] && isfinite(timeout.doubleValue) && timeout.doubleValue >= 0 && timeout.doubleValue <= 86400000 && floor(timeout.doubleValue) == timeout.doubleValue && validString(inputBase64, 24 * 1024 * 1024); + if (!valid) { reject(@"invalid_argument", @"Invalid command arguments", nil); return; } + NSUInteger bytes = 0; + for (id arg in argv) { valid = valid && validString(arg, 131072); if ([arg isKindOfClass:NSString.class]) bytes += [arg lengthOfBytesUsingEncoding:NSUTF8StringEncoding] + 1; } + NSMutableArray *environment = [NSMutableArray new]; + for (id key in env) { + id value = env[key]; + if (![key isKindOfClass:NSString.class] || ![value isKindOfClass:NSString.class]) { valid = NO; continue; } + valid = valid && validString(key, 4096) && [key length] > 0 && [key rangeOfString:@"="].location == NSNotFound && validString(value, 131072); + if ([key isKindOfClass:NSString.class] && [value isKindOfClass:NSString.class]) { NSString *entry = [NSString stringWithFormat:@"%@=%@",key,value]; [environment addObject:entry]; bytes += [entry lengthOfBytesUsingEncoding:NSUTF8StringEncoding] + 1; } + } + NSData *input = [inputBase64 isKindOfClass:NSString.class] ? [[NSData alloc] initWithBase64EncodedString:inputBase64 options:0] : nil; + if (!valid || bytes > 1024 * 1024 || !input || input.length > CaptureLimit) { reject(@"invalid_argument", @"Invalid command arguments, environment, input or timeout", nil); return; } + [self.coordinator.lock lock]; + if(self.invalidated){[self.coordinator.lock unlock];reject(@"spawn_error",@"Native bridge generation is invalidated",nil);return;} + if (!self.coordinator.journalUsable) { [self.coordinator.lock unlock]; reject(@"journal_failed", @"Command journal is unreadable; execution is disabled to prevent replay", nil); return; } + if ([self.coordinator.closing containsObject:workspace]) { [self.coordinator.lock unlock]; reject(@"workspace_closing", @"Workspace cleanup is in progress", nil); return; } + if (self.coordinator.journal[identifier]) { NSDictionary *status = self.coordinator.journal[identifier]; [self.coordinator.lock unlock]; reject(@"duplicate_command", [NSString stringWithFormat:@"Command ID already recorded (%@); use commandStatus",status[@"status"]], nil); return; } + NSUInteger retainedInput = 0; + for (MooseISHJob *active in self.coordinator.jobs.allValues) retainedInput += active.input.length; + if (self.coordinator.jobs.count >= ActiveJobLimit || retainedInput + input.length > RetainedInputLimit || self.coordinator.journal.count >= JournalRecordLimit) { + [self.coordinator.lock unlock]; reject(@"spawn_error", @"Embedded execution capacity is exhausted; no command was started or recorded", nil); return; + } + NSError *error = nil; + if (![self prepareRoot:workspace error:&error]) { [self.coordinator.lock unlock]; reject(@"prepare_failed",error.localizedDescription,error); return; } + int inPipe[2] = {-1,-1}, outPipe[2] = {-1,-1}, errPipe[2] = {-1,-1}; + if (pipe(inPipe) || pipe(outPipe) || pipe(errPipe)) { + for (int i=0;i<2;i++) { if(inPipe[i]>=0)close(inPipe[i]); if(outPipe[i]>=0)close(outPipe[i]); if(errPipe[i]>=0)close(errPipe[i]); } + [self.coordinator.lock unlock]; reject(@"pipe_failed",@"Cannot allocate command pipes",nil); return; + } + const char **args = calloc(argv.count, sizeof(char *)); + const char **vars = calloc(MAX(environment.count,1), sizeof(char *)); + if (!args || !vars) { free(args);free(vars);for(int i=0;i<2;i++){close(inPipe[i]);close(outPipe[i]);close(errPipe[i]);}[self.coordinator.lock unlock];reject(@"allocation_failed",@"Cannot allocate command",nil);return; } + for (NSUInteger i=0;i0)offset+=n;else if(n<0 && errno!=EAGAIN && errno!=EINTR){close(job.inputFD);inputOpen=NO;} + } + for(int index=0;index<2;index++) { + BOOL open=index==0?outOpen:errOpen; + if(!open)continue; + char buffer[16384];ssize_t n=read(index==0?job.outputFD:job.errorFD,buffer,sizeof(buffer)); + if(n>0){ + NSUInteger retained=MIN((NSUInteger)n,CaptureLimit-out.length-err.length); + if(retained){NSData *chunk=[NSData dataWithBytes:buffer length:retained];[(index==0?out:err) appendData:chunk];NSDictionary *event=@{@"commandId":job.identifier,@"stream":index==0?@"stdout":@"stderr",@"data":[chunk base64EncodedStringWithOptions:0]};dispatch_async(dispatch_get_main_queue(),^{if(!self.invalidated)[self sendEventWithName:@"MooseISHOutput" body:event];});} + omitted+=(NSUInteger)n-retained; + if(omitted) { [self.coordinator.lock lock]; if(job.command)moose_ish_command_cancel(job.command);[self.coordinator.lock unlock]; } + } else if(n==0 || (n<0 && errno!=EINTR && errno!=EAGAIN)) {close(index==0?job.outputFD:job.errorFD);if(index==0)outOpen=NO;else errOpen=NO;} + } + if(finished && !outOpen && !errOpen)break; + } + pthread_join(core,NULL); + moose_ish_result result=job.result; + job.input = nil; // Release retained stdin before admitting more commands. + [self.coordinator.lock lock];moose_ish_command_destroy(job.command);job.command=NULL;self.coordinator.journal[job.identifier]=@{@"workspace":job.workspace,@"status":result.error?@"failed":@"completed",@"exitCode":@(result.exit_code)};BOOL saved=[self saveJournal];[self.coordinator.lock unlock]; + // Events and settlement share one queue, so JS sees all chunks before resolve. + dispatch_async(dispatch_get_main_queue(), ^{ + [self.coordinator.lock lock]; [self.coordinator.jobs removeObjectForKey:job.identifier]; [self.coordinator.lock unlock]; + if(!saved)job.reject(@"journal_failed",@"Command ended but final status could not be saved",nil); + else if(result.error)job.reject(@"spawn_error",[NSString stringWithFormat:@"Embedded command setup failed (%d)",result.error],nil); + else job.resolve(@{@"exitCode":@(result.exit_code),@"stdout":[out base64EncodedStringWithOptions:0],@"stderr":[err base64EncodedStringWithOptions:0],@"omittedBytes":@(omitted),@"cancelled":@(result.cancelled),@"timedOut":@(result.timed_out)}); + }); +} +RCT_REMAP_METHOD(cancel, cancel:(NSString *)identifier resolver:(RCTPromiseResolveBlock)resolve rejecter:(RCTPromiseRejectBlock)reject) { + if(!validID(identifier)){reject(@"invalid_argument",@"Invalid command ID",nil);return;} + [self.coordinator.lock lock];if(self.invalidated){[self.coordinator.lock unlock];reject(@"spawn_error",@"Native bridge generation is invalidated",nil);return;}MooseISHJob *job=self.coordinator.jobs[identifier];if(job.command)moose_ish_command_cancel(job.command);else if(!self.coordinator.journal[identifier]){if(self.coordinator.cancelledIDs.count>=JournalRecordLimit && ![self.coordinator.cancelledIDs containsObject:identifier]){[self.coordinator.lock unlock];reject(@"spawn_error",@"Pending cancellation capacity is exhausted",nil);return;}[self.coordinator.cancelledIDs addObject:identifier];}[self.coordinator.lock unlock];resolve(nil); +} +RCT_REMAP_METHOD(cleanup, cleanup:(NSString *)workspace resolver:(RCTPromiseResolveBlock)resolve rejecter:(RCTPromiseRejectBlock)reject) { + if(!validID(workspace)){reject(@"invalid_argument",@"Invalid workspace ID",nil);return;} + [self.coordinator.lock lock];if(self.invalidated){[self.coordinator.lock unlock];reject(@"spawn_error",@"Native bridge generation is invalidated",nil);return;}[self.coordinator.closing addObject:workspace];for(MooseISHJob *job in self.coordinator.jobs.allValues)if([job.workspace isEqual:workspace] && job.command)moose_ish_command_cancel(job.command);[self.coordinator.lock unlock]; + // Wait without occupying React Native's method queue, preserving prompt cancel. + dispatch_async(dispatch_get_global_queue(QOS_CLASS_UTILITY,0),^{for(;;){[self.coordinator.lock lock];BOOL active=NO;for(MooseISHJob *job in self.coordinator.jobs.allValues)if([job.workspace isEqual:workspace]){active=YES;break;}[self.coordinator.lock unlock];if(!active)break;usleep(10000);}[self.coordinator.lock lock];[self.coordinator.closing removeObject:workspace];[self.coordinator.lock unlock];resolve(nil);}); +} +@end diff --git a/modules/moose-ish/package.json b/modules/moose-ish/package.json new file mode 100644 index 0000000..8e7fb31 --- /dev/null +++ b/modules/moose-ish/package.json @@ -0,0 +1,14 @@ +{ + "name": "moose-ish", + "version": "0.1.0", + "private": true, + "main": "index.js", + "license": "UNLICENSED", + "files": [ + "ios", + "MooseISH.podspec", + "react-native.config.js", + "index.js", + "vendor" + ] +} diff --git a/modules/moose-ish/react-native.config.js b/modules/moose-ish/react-native.config.js new file mode 100644 index 0000000..b37e222 --- /dev/null +++ b/modules/moose-ish/react-native.config.js @@ -0,0 +1,16 @@ +const fs = require("node:fs"); +const path = require("node:path"); +const prepared = fs.existsSync( + path.join(__dirname, "vendor/MooseISHCore.xcframework"), +); +module.exports = { + dependency: { + platforms: { + ios: + prepared || process.env.MOOSE_ISH_NATIVE === "1" + ? { podspecPath: path.join(__dirname, "MooseISH.podspec") } + : null, + android: null, + }, + }, +}; diff --git a/modules/moose-ish/tests/bridge_harness.m b/modules/moose-ish/tests/bridge_harness.m new file mode 100644 index 0000000..c90ce2b --- /dev/null +++ b/modules/moose-ish/tests/bridge_harness.m @@ -0,0 +1,104 @@ +// Executes the production bridge with only React's registration layer substituted. +#import "../ios/MooseISH.m" +static NSUInteger eventBytes; +@implementation RCTEventEmitter +- (void)invalidate {} +- (void)sendEventWithName:(NSString *)name body:(id)body { eventBytes += [[NSData alloc] initWithBase64EncodedString:body[@"data"] options:0].length; } +@end +static void fail(NSString *message) { fprintf(stderr,"%s\n",message.UTF8String);exit(1); } +static id waitResult(void (^operation)(RCTPromiseResolveBlock,RCTPromiseRejectBlock)) { + dispatch_semaphore_t sem=dispatch_semaphore_create(0);__block id value=nil;__block NSString *failure=nil; + operation(^(id result){value=result;dispatch_semaphore_signal(sem);},^(NSString *code,NSString *message,NSError *error){failure=code;dispatch_semaphore_signal(sem);}); + if(dispatch_semaphore_wait(sem,dispatch_time(DISPATCH_TIME_NOW,30*NSEC_PER_SEC)))fail(@"bridge timed out"); + if(failure)fail(failure);return value; +} +int main(int argc,char **argv) { + @autoreleasepool { + if(argc!=2){fprintf(stderr,"usage: bridge-harness fakefs-root\n");return 2;} + NSString *seed=[NSString stringWithUTF8String:argv[1]]; + dispatch_async(dispatch_get_global_queue(QOS_CLASS_DEFAULT,0),^{@autoreleasepool { + NSString *base=[NSTemporaryDirectory() stringByAppendingPathComponent:NSUUID.UUID.UUIDString]; + MooseISH *bridge=[[MooseISH alloc] initWithBase:base]; + [[NSFileManager defaultManager] createDirectoryAtPath:[base stringByAppendingPathComponent:@"test"] withIntermediateDirectories:YES attributes:nil error:nil]; + NSError *error=nil;[[NSFileManager defaultManager] copyItemAtPath:seed toPath:[bridge rootFor:@"test"] error:&error];if(error)fail(error.description); + NSDictionary *result=waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge execute:@"test" commandId:@"one" argv:@[@"/bin/sh",@"-c",@"cat; printf err >&2; exit 7"] cwd:@"/" env:@{} inputBase64:[[@"bytes" dataUsingEncoding:NSUTF8StringEncoding] base64EncodedStringWithOptions:0] timeoutMs:@5000 resolver:resolve rejecter:reject];}); + if([result[@"exitCode"] intValue]!=7)fail(@"exit code"); + if(![[[NSData alloc] initWithBase64EncodedString:result[@"stdout"] options:0] isEqual:[@"bytes" dataUsingEncoding:NSUTF8StringEncoding]])fail(@"stdin/stdout"); + if(![[[NSData alloc] initWithBase64EncodedString:result[@"stderr"] options:0] isEqual:[@"err" dataUsingEncoding:NSUTF8StringEncoding]])fail(@"stderr"); + if(eventBytes!=8)fail(@"stream events must precede settlement"); + NSMutableData *large=[NSMutableData dataWithLength:8*1024*1024];memset(large.mutableBytes,0xa5,large.length); + fprintf(stderr,"bridge large transfer start\n"); + NSDictionary *transfer=waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge execute:@"test" commandId:@"large" argv:@[@"/bin/cat"] cwd:@"/" env:@{} inputBase64:[large base64EncodedStringWithOptions:0] timeoutMs:@10000 resolver:resolve rejecter:reject];}); + if(![[[NSData alloc] initWithBase64EncodedString:transfer[@"stdout"] options:0] isEqual:large])fail(@"8MiB binary transfer"); + fprintf(stderr,"bridge timeout start\n"); + NSDictionary *timeout=waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge execute:@"test" commandId:@"timeout" argv:@[@"/bin/sh",@"-c",@"sleep 20"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@100 resolver:resolve rejecter:reject];}); + if(![timeout[@"timedOut"] boolValue])fail(@"timeout status"); + double cancelStart=[NSDate timeIntervalSinceReferenceDate]; + NSDictionary *directSleep=waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge execute:@"test" commandId:@"direct-sleep" argv:@[@"/bin/sleep",@"2"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@5000 resolver:resolve rejecter:reject];usleep(50000);[bridge cancel:@"direct-sleep" resolver:^(id value){} rejecter:reject];}); + if(![directSleep[@"cancelled"] boolValue] || [NSDate timeIntervalSinceReferenceDate]-cancelStart>=1.0)fail(@"direct sleep cancellation must complete within one second"); + fprintf(stderr,"bridge flood start\n"); + NSDictionary *flood=waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge execute:@"test" commandId:@"flood" argv:@[@"/bin/sh",@"-c",@"yes x"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@10000 resolver:resolve rejecter:reject];}); + if(![flood[@"omittedBytes"] unsignedLongLongValue] || [[NSData alloc] initWithBase64EncodedString:flood[@"stdout"] options:0].length>16*1024*1024)fail(@"output flood bound"); + waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge cancel:@"precancel" resolver:resolve rejecter:reject];}); + NSDictionary *cancelled=waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge execute:@"test" commandId:@"precancel" argv:@[@"/bin/sleep",@"20"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@5000 resolver:resolve rejecter:reject];}); + if(![cancelled[@"cancelled"] boolValue])fail(@"pre-register cancel lost"); + bridge.coordinator.journalUsable=NO;__block BOOL failClosed=NO; + [bridge execute:@"test" commandId:@"corrupt" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@5000 resolver:^(id value){fail(@"journal failure replayed");} rejecter:^(NSString *code,NSString *message,NSError *error){failClosed=[code isEqual:@"journal_failed"];}]; + if(!failClosed)fail(@"corrupt journal must fail closed");bridge.coordinator.journalUsable=YES; + NSMutableArray *admitted=[NSMutableArray new]; + NSMutableData *sixteen=[NSMutableData dataWithLength:16*1024*1024];NSString *sixteen64=[sixteen base64EncodedStringWithOptions:0]; + for(int i=0;i<2;i++){ + dispatch_semaphore_t done=dispatch_semaphore_create(0);[admitted addObject:done];NSString *identifier=[NSString stringWithFormat:@"admit-bytes-%d",i]; + [bridge execute:@"test" commandId:identifier argv:@[@"/bin/sh",@"-c",@"while :; do :; done"] cwd:@"/" env:@{} inputBase64:sixteen64 timeoutMs:@25000 resolver:^(id value){dispatch_semaphore_signal(done);} rejecter:^(NSString *code,NSString *message,NSError *error){fail(code);}]; + } + __block BOOL byteBusy=NO; + [bridge execute:@"test" commandId:@"byte-overflow" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"eA==" timeoutMs:@1000 resolver:^(id value){fail(@"stdin admission exceeded");} rejecter:^(NSString *code,NSString *message,NSError *error){byteBusy=[code isEqual:@"spawn_error"];}]; + if(!byteBusy || bridge.coordinator.journal[@"byte-overflow"])fail(@"stdin budget must reject without journaling"); + for(int i=0;i<2;i++)[bridge cancel:[NSString stringWithFormat:@"admit-bytes-%d",i] resolver:^(id v){} rejecter:^(NSString *c,NSString *m,NSError *e){fail(c);}]; + for(dispatch_semaphore_t done in admitted)if(dispatch_semaphore_wait(done,dispatch_time(DISPATCH_TIME_NOW,30*NSEC_PER_SEC)))fail(@"admission cancellation"); + [admitted removeAllObjects]; + for(int i=0;i<4;i++){ + dispatch_semaphore_t done=dispatch_semaphore_create(0);[admitted addObject:done];NSString *identifier=[NSString stringWithFormat:@"admit-jobs-%d",i]; + [bridge execute:@"test" commandId:identifier argv:@[@"/bin/sh",@"-c",@"while :; do :; done"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@25000 resolver:^(id value){dispatch_semaphore_signal(done);} rejecter:^(NSString *code,NSString *message,NSError *error){fail(code);}]; + } + MooseISH *capacityGeneration=[[MooseISH alloc] initWithBase:base]; + __block BOOL jobBusy=NO; + [capacityGeneration execute:@"test" commandId:@"job-overflow" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@1000 resolver:^(id value){fail(@"job admission exceeded");} rejecter:^(NSString *code,NSString *message,NSError *error){jobBusy=[code isEqual:@"spawn_error"];}]; + if(!jobBusy || bridge.coordinator.journal[@"job-overflow"])fail(@"job budget must reject without journaling"); + for(int i=0;i<4;i++)[bridge cancel:[NSString stringWithFormat:@"admit-jobs-%d",i] resolver:^(id v){} rejecter:^(NSString *c,NSString *m,NSError *e){fail(c);}]; + for(dispatch_semaphore_t done in admitted)if(dispatch_semaphore_wait(done,dispatch_time(DISPATCH_TIME_NOW,30*NSEC_PER_SEC)))fail(@"job cancellation"); + NSMutableDictionary *previousJournal=bridge.coordinator.journal;bridge.coordinator.journal=[NSMutableDictionary new]; + for(int i=0;i<10000;i++)bridge.coordinator.journal[[NSString stringWithFormat:@"history-%d",i]]=@{@"workspace":@"test",@"status":@"completed",@"exitCode":@0}; + __block BOOL journalFull=NO; + [bridge execute:@"test" commandId:@"history-overflow" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@1000 resolver:^(id value){fail(@"journal capacity exceeded");} rejecter:^(NSString *code,NSString *message,NSError *error){journalFull=[code isEqual:@"spawn_error"];}]; + if(!journalFull || bridge.coordinator.journal.count!=10000 || bridge.coordinator.journal[@"history-overflow"])fail(@"journal limit must preserve prior IDs without adding rejected ID");bridge.coordinator.journal=previousJournal; + __block BOOL duplicate=NO; + [bridge execute:@"test" commandId:@"one" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@5000 resolver:^(id value){fail(@"duplicate replayed");} rejecter:^(NSString *code,NSString *message,NSError *error){duplicate=[code isEqual:@"duplicate_command"];}]; + if(!duplicate)fail(@"missing duplicate rejection"); + NSDictionary *journal=[NSJSONSerialization JSONObjectWithData:[NSData dataWithContentsOfFile:bridge.coordinator.journalPath] options:0 error:nil];for(NSDictionary *record in journal.allValues)for(id key in record)if(![@[@"workspace",@"status",@"exitCode"] containsObject:key])fail(@"journal leaked input"); + waitResult(^(RCTPromiseResolveBlock resolve,RCTPromiseRejectBlock reject){[bridge cleanup:@"test" resolver:resolve rejecter:reject];}); + dispatch_semaphore_t oldDone=dispatch_semaphore_create(0),freshDone=dispatch_semaphore_create(0); + RCTPromiseRejectBlock reloadReject=^(NSString *code,NSString *message,NSError *error){fail(code);}; + [bridge execute:@"test" commandId:@"reload-old" argv:@[@"/bin/sh",@"-c",@"while :; do :; done"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@5000 resolver:^(id value){dispatch_semaphore_signal(oldDone);} rejecter:reloadReject]; + MooseISH *fresh=[[MooseISH alloc] initWithBase:base]; + if(![fresh.coordinator.journal[@"reload-old"][@"status"] isEqual:@"running"])fail(@"reload must not mark live old-generation command interrupted"); + [fresh execute:@"test" commandId:@"reload-fresh" argv:@[@"/bin/sh",@"-c",@"printf effect > /workspace/reload-effect; while :; do :; done"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@5000 resolver:^(id value){dispatch_semaphore_signal(freshDone);} rejecter:reloadReject]; + [bridge invalidate]; + if(dispatch_semaphore_wait(oldDone,dispatch_time(DISPATCH_TIME_NOW,5*NSEC_PER_SEC)))fail(@"invalidated generation must cancel own command"); + if(dispatch_semaphore_wait(freshDone,dispatch_time(DISPATCH_TIME_NOW,100*NSEC_PER_MSEC))==0)fail(@"invalidating old generation cancelled fresh generation"); + NSDictionary *reloadDisk=[NSJSONSerialization JSONObjectWithData:[NSData dataWithContentsOfFile:fresh.coordinator.journalPath] options:0 error:nil]; + if(![reloadDisk[@"reload-fresh"][@"status"] isEqual:@"running"] || !fresh.coordinator.jobs[@"reload-fresh"])fail(@"old completion overwrote fresh-generation command journal"); + __block BOOL invalidated=NO; + [bridge execute:@"test" commandId:@"invalidated-admission" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@1000 resolver:^(id value){fail(@"invalidated generation admitted command");} rejecter:^(NSString *code,NSString *message,NSError *error){invalidated=[code isEqual:@"spawn_error"];}]; + if(!invalidated || fresh.coordinator.journal[@"invalidated-admission"])fail(@"invalidated admission must have no journal side effect"); + __block BOOL reloadDuplicate=NO; + [fresh execute:@"test" commandId:@"reload-old" argv:@[@"/bin/true"] cwd:@"/" env:@{} inputBase64:@"" timeoutMs:@1000 resolver:^(id value){fail(@"cross-generation duplicate replayed");} rejecter:^(NSString *code,NSString *message,NSError *error){reloadDuplicate=[code isEqual:@"duplicate_command"];}]; + if(!reloadDuplicate)fail(@"cross-generation duplicate protection"); + [fresh cancel:@"reload-fresh" resolver:^(id value){} rejecter:reloadReject]; + if(dispatch_semaphore_wait(freshDone,dispatch_time(DISPATCH_TIME_NOW,5*NSEC_PER_SEC)))fail(@"fresh generation cancellation"); + [fresh invalidate];[capacityGeneration invalidate]; + fprintf(stdout,"production native bridge: stdin, streams, exit, event ordering, 8MiB binary transfer, timeout, direct-sleep cancellation, flood, admission limits, cross-generation journal/invalidation, stable ID, journal, cleanup passed\n");exit(0); + }}); + dispatch_main(); + } +} diff --git a/modules/moose-ish/tests/react/React/RCTBridgeModule.h b/modules/moose-ish/tests/react/React/RCTBridgeModule.h new file mode 100644 index 0000000..d1cb777 --- /dev/null +++ b/modules/moose-ish/tests/react/React/RCTBridgeModule.h @@ -0,0 +1,6 @@ +#import +typedef void (^RCTPromiseResolveBlock)(id); +typedef void (^RCTPromiseRejectBlock)(NSString *, NSString *, NSError *); +@protocol RCTBridgeModule @end +#define RCT_EXPORT_MODULE(name) +#define RCT_REMAP_METHOD(name, signature) - (void)signature diff --git a/modules/moose-ish/tests/react/React/RCTEventEmitter.h b/modules/moose-ish/tests/react/React/RCTEventEmitter.h new file mode 100644 index 0000000..8518ba1 --- /dev/null +++ b/modules/moose-ish/tests/react/React/RCTEventEmitter.h @@ -0,0 +1,5 @@ +#import "RCTBridgeModule.h" +@interface RCTEventEmitter : NSObject +- (void)invalidate; +- (void)sendEventWithName:(NSString *)name body:(id)body; +@end diff --git a/modules/moose-ish/tests/run_bridge.py b/modules/moose-ish/tests/run_bridge.py new file mode 100644 index 0000000..7fbaa9f --- /dev/null +++ b/modules/moose-ish/tests/run_bridge.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 +"""Compile the actual Objective-C bridge against the real macOS embedded core.""" +import argparse +from pathlib import Path +import subprocess +p=argparse.ArgumentParser(description=__doc__) +p.add_argument('--core',type=Path,required=True,help='build_embedded.py --sdk macosx output') +a=p.parse_args(); here=Path(__file__).resolve().parent +binary=a.core/'bridge-harness' +subprocess.run(['clang','-fobjc-arc','-fblocks','-I'+str(here/'react'),'-I'+str(a.core/'include'),str(here/'bridge_harness.m'),str(a.core/'libMooseISHCore.a'),'-framework','Foundation','-lsqlite3','-lpthread','-o',str(binary)],check=True) +subprocess.run([str(binary),str(a.core/'root')],check=True,timeout=60) diff --git a/modules/moose-ish/tests/run_bridge_simulator.py b/modules/moose-ish/tests/run_bridge_simulator.py new file mode 100644 index 0000000..51c21b6 --- /dev/null +++ b/modules/moose-ish/tests/run_bridge_simulator.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Run production bridge/core tests in a disposable iOS simulator application.""" +import argparse +from pathlib import Path +import plistlib +import shutil +import subprocess +p=argparse.ArgumentParser(description=__doc__) +p.add_argument('--core',type=Path,required=True,help='build_embedded.py --sdk iphonesimulator output') +p.add_argument('--output',type=Path,required=True,help='Disposable external app build directory') +a=p.parse_args();here=Path(__file__).resolve().parent;out=a.output.resolve();repo=here.parents[2] +if out==repo or repo in out.parents:p.error('--output must be outside repository') +out.mkdir(parents=True,exist_ok=True) +app=out/'MooseISHBridgeTest.app';app.mkdir(exist_ok=True) +source=(here/'bridge_harness.m').read_text().replace('int main(int argc,char **argv)', 'static int bridgeMain(int argc,char **argv)').replace(' dispatch_main();',' return 0;') +source+=''' +#import +@interface Delegate : UIResponder +@property(nonatomic, strong) UIWindow *window; +@end +@implementation Delegate +- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)options { + self.window=[[UIWindow alloc] initWithFrame:UIScreen.mainScreen.bounds];self.window.rootViewController=[UIViewController new];[self.window makeKeyAndVisible]; + const char *seed=[NSBundle.mainBundle pathForResource:@"root" ofType:nil].fileSystemRepresentation; + char *args[]={"bridge-test",(char *)seed};bridgeMain(2,args);return YES; +} +@end +int main(int argc,char **argv){@autoreleasepool{return UIApplicationMain(argc,argv,nil,NSStringFromClass(Delegate.class));}} +''' +(out/'bridge_simulator.m').write_text(source) +sdk=subprocess.check_output(['xcrun','--sdk','iphonesimulator','--show-sdk-path'],text=True).strip() +subprocess.run(['xcrun','--sdk','iphonesimulator','clang','-target','arm64-apple-ios15.1-simulator','-isysroot',sdk,'-fobjc-arc','-fblocks','-I'+str(here),'-I'+str(here/'react'),'-I'+str(a.core/'include'),str(out/'bridge_simulator.m'),str(a.core/'libMooseISHCore.a'),'-framework','Foundation','-framework','UIKit','-lsqlite3','-lpthread','-o',str(app/'MooseISHBridgeTest')],check=True) +if(app/'root').exists():shutil.rmtree(app/'root') +shutil.copytree(a.core/'root',app/'root') +with(app/'Info.plist').open('wb') as f: + plistlib.dump(dict(CFBundleExecutable='MooseISHBridgeTest',CFBundleIdentifier='dev.moose.ishbridgetest',CFBundleName='Moose iSH Bridge Test',CFBundleVersion='1',CFBundleShortVersionString='0.1',CFBundlePackageType='APPL',LSRequiresIPhoneOS=True,MinimumOSVersion='15.1',UIDeviceFamily=[1,2],UILaunchScreen={}),f) +subprocess.run(['codesign','--force','--sign','-',str(app)],check=True) +subprocess.run(['xcrun','simctl','install','booted',str(app)],check=True) +result=subprocess.run(['xcrun','simctl','launch','--console','--terminate-running-process','booted','dev.moose.ishbridgetest'],stdout=subprocess.PIPE,stderr=subprocess.STDOUT,text=True,timeout=90) +print(result.stdout) +if 'production native bridge:' not in result.stdout or 'cleanup passed' not in result.stdout:raise SystemExit('Simulator bridge tests did not complete successfully') diff --git a/modules/moose-ish/tests/test_package.py b/modules/moose-ish/tests/test_package.py new file mode 100644 index 0000000..d5b7756 --- /dev/null +++ b/modules/moose-ish/tests/test_package.py @@ -0,0 +1,13 @@ +import unittest +from pathlib import Path +import json +ROOT=Path(__file__).resolve().parents[1] +class Package(unittest.TestCase): + def test_package_is_ios_only(self): + package=json.loads((ROOT/'package.json').read_text()) + self.assertEqual(package['name'],'moose-ish') + self.assertTrue((ROOT/'MooseISH.podspec').exists()) + self.assertIn('MooseISH', (ROOT/'react-native.config.js').read_text()) + def test_vendor_is_not_committed(self): + self.assertIn('vendor/', (ROOT/'.gitignore').read_text()) +if __name__=='__main__': unittest.main() diff --git a/native/ish/NOTICE.md b/native/ish/NOTICE.md new file mode 100644 index 0000000..c6297fe --- /dev/null +++ b/native/ish/NOTICE.md @@ -0,0 +1,40 @@ +# Embedded iSH source and license notice + +The optional native execution core links iSH, copyright its upstream contributors, +from https://github.com/ish-app/ish at commit +`c5ebcabec14f1d8709e91e1905ce0edd7cb9c13c`. + +Upstream `LICENSE.md` specifies GNU GPL version 3 plus the additional terms in +`LICENSE.IOS`. Contributions after the commit identified in `LICENSE.md`, and +identified authors' earlier contributions, are additionally licensed under GPL +version 2. That does not establish a GPLv2-only license for the complete source. +Preserve all upstream source notices and the full upstream license files. + +`embedded-policy.patch` is Moose's local modification to that pinned source: +it adds inherited execution domains, isolates mount selection by domain, +prevents guest host mounts, sockets, ptrace, privileged device/chroot operations, +cross-domain signals and cross-process resource-limit changes, and suppresses +setuid/setgid privilege changes at exec. `embedded.c` adds one surviving +supervisor, serialized command startup, explicit descriptors, domain cancellation +and reaping. The builder applies the patch only to a disposable copied source; +the pristine pinned checkout remains unchanged. + +Before distributing a linked binary, select a compatible whole-program license, +include complete applicable license texts and notices, and deliver complete +corresponding source with build instructions for the exact kernel, modifications, +frontend and bundled root filesystem packages. `LICENSE.IOS` retains source and +license-text obligations. Alpine packages have their own licenses and sources; +rootfs redistribution requires an additional package source/license manifest. +The current build outputs are local validation artifacts, not a distribution +license determination. See `docs/ios-ish-execution.md` for evidence and limits. + +The embedded policy also caps each active domain at 32 owned task slots (processes, threads and unreaped zombies +together), regular guest files at 16 MiB, and combined guest mapping +backing at 256 MiB. Shared COW backing counts once; private copies count when +allocated. Native syscall transfer buffers are separately bounded to 16 MiB, +with at most 1024 vector entries. These are explicit kernel-path checks, not +claims that upstream enforces RLIMIT_AS, RLIMIT_FSIZE, or RLIMIT_NPROC. The +mapping counter does not bound whole-process RSS, emulator/SQLite metadata or +all host allocations. There is no aggregate workspace disk quota. Mounts and +SQLite connections are retired after every command's tasks and descriptors +have been reaped; persistent fakefs files remain on disk. diff --git a/native/ish/embedded-policy.patch b/native/ish/embedded-policy.patch new file mode 100644 index 0000000..36ee0e9 --- /dev/null +++ b/native/ish/embedded-policy.patch @@ -0,0 +1,539 @@ +--- a/fs/fake.c ++++ b/fs/fake.c +@@ -383,7 +383,9 @@ + if (err != SQLITE_OK) { + printk("sqlite failed to close: %d\n", err); + } +- /* return realfs.umount(mount); */ ++ /* All guest fd/inode references are gone before embedded mount retirement. */ ++ if (close(mount->root_fd) < 0) ++ return errno_map(); + return 0; + } + +--- a/kernel/time.c ++++ b/kernel/time.c +@@ -189,8 +189,34 @@ + struct timespec req; + req.tv_sec = req_ts.sec; + req.tv_nsec = req_ts.nsec; +- struct timespec rem; +- if (nanosleep(&req, &rem) < 0) ++ struct timespec rem = {0}; ++ if (current->moose_domain) { ++ /* A host signal can arrive before nanosleep starts. Pending guest ++ signals coalesce, so do not depend on a second pthread_kill wake. */ ++ if (req.tv_sec < 0 || req.tv_nsec < 0 || req.tv_nsec >= 1000000000) ++ return _EINVAL; ++ struct timespec now; ++ clock_gettime(CLOCK_MONOTONIC, &now); ++ int64_t deadline = (int64_t)now.tv_sec * 1000000000 + now.tv_nsec ++ + (int64_t)req.tv_sec * 1000000000 + req.tv_nsec; ++ for (;;) { ++ lock(¤t->sighand->lock); ++ bool pending = !!(current->pending & ~current->blocked); ++ unlock(¤t->sighand->lock); ++ clock_gettime(CLOCK_MONOTONIC, &now); ++ int64_t left = deadline - ((int64_t)now.tv_sec * 1000000000 + now.tv_nsec); ++ if (pending) { ++ if (rem_addr && left > 0) { ++ struct timespec_ remaining = {left / 1000000000, left % 1000000000}; ++ if (user_put(rem_addr, remaining)) return _EFAULT; ++ } ++ return _EINTR; ++ } ++ if (left <= 0) break; ++ struct timespec slice = {0, left > 10000000 ? 10000000 : left}; ++ nanosleep(&slice, NULL); ++ } ++ } else if (nanosleep(&req, &rem) < 0) + return errno_map(); + if (rem_addr != 0) { + struct timespec_ rem_ts; +--- a/kernel/memory.h ++++ b/kernel/memory.h +@@ -35,6 +35,7 @@ + #define LEAK_DEBUG 0 + + struct data { ++ size_t moose_bytes; /* counted shared backing, not duplicate COW views */ + void *data; // immutable + size_t size; // also immutable + atomic_uint refcount; +--- a/kernel/memory.c ++++ b/kernel/memory.c +@@ -103,6 +103,8 @@ + return true; + } + ++static _Atomic size_t moose_backing_bytes; ++#define MOOSE_MEMORY_LIMIT (256u * 1024u * 1024u) + int pt_map(struct mem *mem, page_t start, pages_t pages, void *memory, size_t offset, unsigned flags) { + if (memory == MAP_FAILED) + return errno_map(); +@@ -110,10 +112,25 @@ + // If this fails, the munmap in pt_unmap would probably fail. + assert((uintptr_t) memory % real_page_size == 0 || memory == vdso_data); + ++ size_t moose_bytes = current && current->moose_domain && memory != vdso_data ++ ? pages * PAGE_SIZE + offset : 0; ++ size_t used = atomic_load(&moose_backing_bytes); ++ if (moose_bytes) { ++ do { ++ if (moose_bytes > MOOSE_MEMORY_LIMIT || used > MOOSE_MEMORY_LIMIT - moose_bytes) { ++ munmap(memory, pages * PAGE_SIZE + offset); ++ return _ENOMEM; ++ } ++ } while (!atomic_compare_exchange_weak(&moose_backing_bytes, &used, used + moose_bytes)); ++ } + struct data *data = malloc(sizeof(struct data)); +- if (data == NULL) ++ if (data == NULL) { ++ atomic_fetch_sub(&moose_backing_bytes, moose_bytes); ++ if (memory != vdso_data) munmap(memory, pages * PAGE_SIZE + offset); + return _ENOMEM; ++ } + *data = (struct data) { ++ .moose_bytes = moose_bytes, + .data = memory, + .size = pages * PAGE_SIZE + offset, + +@@ -160,6 +177,7 @@ + if (data->fd != NULL) { + fd_close(data->fd); + } ++ atomic_fetch_sub(&moose_backing_bytes, data->moose_bytes); + free(data); + } + } +--- a/kernel/fs.c ++++ b/kernel/fs.c +@@ -237,6 +237,7 @@ + } + + dword_t sys_read(fd_t fd_no, addr_t buf_addr, dword_t size) { ++ if (current->moose_domain && size > 16 * 1024 * 1024) return _EINVAL; + STRACE("read(%d, 0x%x, %d)", fd_no, buf_addr, size); + char *buf = (char *) malloc(size); + if (buf == NULL) +@@ -270,6 +271,7 @@ + } + + dword_t sys_write(fd_t fd_no, addr_t buf_addr, dword_t size) { ++ if (current->moose_domain && size > 16 * 1024 * 1024) return _EINVAL; + // FIXME this is a DOS vector, should ideally use vectorized I/O + char *buf = malloc(size); + if (buf == NULL) +@@ -296,6 +298,8 @@ + // by the inefficiency of the emulator. + + static struct iovec_ *read_iovec(addr_t iovec_addr, unsigned iovec_count) { ++ if (current->moose_domain && iovec_count > 1024) ++ return ERR_PTR(_EINVAL); + dword_t iovec_size = sizeof(struct iovec_) * iovec_count; + struct iovec_ *iovec = malloc(iovec_size); + if (iovec == NULL) +@@ -320,6 +324,10 @@ + if (IS_ERR(iovec)) + return PTR_ERR(iovec); + size_t io_size = iovec_size(iovec, iovec_count); ++ if (current->moose_domain && io_size > 16 * 1024 * 1024) { ++ free(iovec); ++ return _EINVAL; ++ } + char *buf = malloc(io_size); + if (buf == NULL) { + free(iovec); +@@ -354,6 +362,10 @@ + if (IS_ERR(iovec)) + return PTR_ERR(iovec); + size_t io_size = iovec_size(iovec, iovec_count); ++ if (current->moose_domain && io_size > 16 * 1024 * 1024) { ++ free(iovec); ++ return _EINVAL; ++ } + char *buf = malloc(io_size); + if (buf == NULL) { + free(iovec); +@@ -415,6 +427,7 @@ + } + + dword_t sys_pread(fd_t f, addr_t buf_addr, dword_t size, off_t_ off) { ++ if (current->moose_domain && size > 16 * 1024 * 1024) return _EINVAL; + STRACE("pread(%d, 0x%x, %d, %d)", f, buf_addr, size, off); + struct fd *fd = f_get(f); + if (fd == NULL) +@@ -452,6 +465,7 @@ + } + + dword_t sys_pwrite(fd_t f, addr_t buf_addr, dword_t size, off_t_ off) { ++ if (current->moose_domain && size > 16 * 1024 * 1024) return _EINVAL; + STRACE("pwrite(%d, 0x%x, %d, %d)", f, buf_addr, size, off); + struct fd *fd = f_get(f); + if (fd == NULL) +@@ -883,7 +897,7 @@ + char path[MAX_PATH]; + if (user_read_string(path_addr, path, sizeof(path))) + return _EFAULT; +- return generic_setattrat(NULL, path, make_attr(size, size), true); ++ return generic_setattrat(AT_PWD, path, make_attr(size, size), true); + } + + dword_t sys_ftruncate64(fd_t f, dword_t size_low, dword_t size_high) { +--- a/fs/real.c ++++ b/fs/real.c +@@ -130,7 +130,32 @@ + return res; + } + ++#define MOOSE_FILE_LIMIT (16 * 1024 * 1024) ++static pthread_mutex_t moose_file_lock = PTHREAD_MUTEX_INITIALIZER; ++static bool moose_bounded_file(struct fd *fd) { ++ return current && current->moose_domain && S_ISREG(fd->type); ++} ++static int moose_check_write(struct fd *fd, size_t size, off_t offset) { ++ if (fcntl(fd->real_fd, F_GETFL) & O_APPEND) { ++ struct stat stat; ++ if (fstat(fd->real_fd, &stat) < 0) ++ return errno_map(); ++ offset = stat.st_size; ++ } ++ if (offset < 0 || size > MOOSE_FILE_LIMIT || offset > MOOSE_FILE_LIMIT - size) ++ return _EFBIG; ++ return 0; ++} + ssize_t realfs_write(struct fd *fd, const void *buf, size_t bufsize) { ++ if (moose_bounded_file(fd)) { ++ pthread_mutex_lock(&moose_file_lock); ++ off_t offset = lseek(fd->real_fd, 0, SEEK_CUR); ++ int error = moose_check_write(fd, bufsize, offset); ++ ssize_t result = error < 0 ? error : write(fd->real_fd, buf, bufsize); ++ if (result == -1) result = errno_map(); ++ pthread_mutex_unlock(&moose_file_lock); ++ return result; ++ } + ssize_t res = write(fd->real_fd, buf, bufsize); + if (res < 0) + return errno_map(); +@@ -145,6 +170,20 @@ + } + + ssize_t realfs_pwrite(struct fd *fd, const void *buf, size_t bufsize, off_t off) { ++ if (moose_bounded_file(fd)) { ++ pthread_mutex_lock(&moose_file_lock); ++ /* Darwin honors pwrite's explicit offset even with O_APPEND. Check ++ both that offset and the EOF append position (for other hosts). */ ++ int error; ++ if (off < 0 || bufsize > MOOSE_FILE_LIMIT || off > MOOSE_FILE_LIMIT - bufsize) ++ error = _EFBIG; ++ else ++ error = moose_check_write(fd, bufsize, off); ++ ssize_t result = error < 0 ? error : pwrite(fd->real_fd, buf, bufsize, off); ++ if (result == -1) result = errno_map(); ++ pthread_mutex_unlock(&moose_file_lock); ++ return result; ++ } + ssize_t res = pwrite(fd->real_fd, buf, bufsize, off); + if (res < 0) + return errno_map(); +@@ -335,6 +374,8 @@ + } + + int realfs_truncate(struct mount *mount, const char *path, off_t_ size) { ++ if (current && current->moose_domain && size > MOOSE_FILE_LIMIT) ++ return _EFBIG; + int fd = openat(mount->root_fd, fix_path(path), O_RDWR); + if (fd < 0) + return errno_map(); +@@ -370,6 +411,8 @@ + } + + int realfs_fsetattr(struct fd *fd, struct attr attr) { ++ if (current && current->moose_domain && attr.type == attr_size && attr.size > MOOSE_FILE_LIMIT) ++ return _EFBIG; + int real_fd = fd->real_fd; + int err; + switch (attr.type) { +--- a/kernel/fork.c ++++ b/kernel/fork.c +@@ -153,7 +153,7 @@ + + struct task *task = task_create_(current); + if (task == NULL) +- return _ENOMEM; ++ return errno == EAGAIN ? _EAGAIN : _ENOMEM; + int err = copy_task(task, flags, stack, ptid, tls, ctid); + if (err < 0) { + // FIXME: there is a window between task_create_ and task_destroy where +@@ -183,7 +183,13 @@ + send_signal(current, SIGTRAP_, SIGINFO_NIL); + } + +- task_start(task); ++ err = task_start(task); ++ if (err < 0) { ++ task_discard_unstarted(task); ++ if (flags & CLONE_VFORK_) ++ cond_destroy(&vfork.cond); ++ return err; ++ } + + if (flags & CLONE_VFORK_) { + lock(&vfork.lock); +--- a/kernel/exec.c ++++ b/kernel/exec.c +@@ -585,11 +585,11 @@ + return err; + + // setuid/setgid +- if (stat.mode & S_ISUID) { ++ if (!current->moose_domain && (stat.mode & S_ISUID)) { + current->suid = current->euid; + current->euid = stat.uid; + } +- if (stat.mode & S_ISGID) { ++ if (!current->moose_domain && (stat.mode & S_ISGID)) { + current->sgid = current->egid; + current->egid = stat.gid; + } +--- a/kernel/task.h ++++ b/kernel/task.h +@@ -15,6 +15,7 @@ + // everything here is private to the thread executing this task and needs no + // locking, unless otherwise specified + struct task { ++ unsigned moose_domain; /* inherited by fork; host-assigned only */ + struct cpu_state cpu; + struct mm *mm; // locked by general_lock + struct mem *mem; // pointer to mm.mem, for convenience +@@ -187,7 +188,8 @@ + #define MAX_PID (1 << 15) // oughta be enough + + // TODO document +-void task_start(struct task *task); ++int task_start(struct task *task); ++void task_discard_unstarted(struct task *task); + void task_run_current(void); + + extern void (*exit_hook)(struct task *task, int code); +--- a/kernel/fs.h ++++ b/kernel/fs.h +@@ -71,6 +71,7 @@ + int access_check(struct statbuf *stat, int check); + + struct mount { ++ unsigned moose_domain; + const char *point; + const char *source; + const char *info; +--- a/fs/mount.c ++++ b/fs/mount.c +@@ -4,6 +4,7 @@ + #include "kernel/fs.h" + #include "fs/path.h" + #include "fs/real.h" ++#include "kernel/task.h" + + #define MAX_FILESYSTEMS 10 + static const struct fs_ops *filesystems[MAX_FILESYSTEMS] = { +@@ -29,6 +30,8 @@ + struct mount *mount = NULL; + assert(!list_empty(&mounts)); // this would mean there's no root FS mounted + list_for_each_entry(&mounts, mount, mounts) { ++ if (mount->moose_domain != (current ? current->moose_domain : 0)) ++ continue; + size_t n = strlen(mount->point); + if (strncmp(path, mount->point, n) == 0 && (path[n] == '/' || path[n] == '\0')) + break; +@@ -54,6 +57,7 @@ + struct mount *new_mount = malloc(sizeof(struct mount)); + if (new_mount == NULL) + return _ENOMEM; ++ new_mount->moose_domain = current ? current->moose_domain : 0; + new_mount->point = strdup(point); + new_mount->source = strdup(source); + new_mount->info = strdup(info); +--- a/kernel/init.c ++++ b/kernel/init.c +@@ -55,6 +55,8 @@ + // TODO error propagation + static struct task *construct_task(struct task *parent) { + struct task *task = task_create_(parent); ++ if (!task) ++ return ERR_PTR(errno == EAGAIN ? _EAGAIN : _ENOMEM); + + struct tgroup *group = malloc(sizeof(struct tgroup)); + *group = (struct tgroup) {}; +@@ -100,12 +102,16 @@ + return 0; + } + +-int become_new_init_child(void) { ++int become_new_init_child_domain(unsigned domain) { + // locking? who needs locking?! + struct task *init = pid_get_task(1); + assert(init != NULL); + ++ /* Only the serialized embedding owner calls this. Set inherited domain ++ before construct_task opens /, then restore the supervisor domain. */ ++ init->moose_domain = domain; + struct task *task = construct_task(init); ++ init->moose_domain = 0; + if (IS_ERR(task)) + return PTR_ERR(task); + +@@ -120,6 +126,10 @@ + return 0; + } + ++int become_new_init_child(void) { ++ return become_new_init_child_domain(0); ++} ++ + extern int console_major; + extern int console_minor; + void set_console_device(int major, int minor) { +--- a/kernel/calls.c ++++ b/kernel/calls.c +@@ -260,7 +260,20 @@ + struct cpu_state *cpu = ¤t->cpu; + if (interrupt == INT_SYSCALL) { + unsigned syscall_num = cpu->eax; +- if (syscall_num >= NUM_SYSCALLS || syscall_table[syscall_num] == NULL) { ++ bool moose_denied = syscall_num == 340 && current->moose_domain && cpu->ebx != 0 && cpu->ebx != current->pid; ++ if (current->moose_domain) { ++ switch (syscall_num) { ++ case 14: case 61: case 297: /* mknod/chroot/mknodat */ ++ case 21: case 26: case 52: /* mount, ptrace, umount */ ++ case 102: case 359: case 360: /* socketcall, socket, socketpair */ ++ case 169: case 88: /* reboot, reboot legacy */ ++ case 96: case 97: case 101: case 349: /* ioperm, kcmp */ ++ moose_denied = true; ++ } ++ } ++ if (moose_denied) { ++ cpu->eax = _EPERM; ++ } else if (syscall_num >= NUM_SYSCALLS || syscall_table[syscall_num] == NULL) { + printk("%d(%s) missing syscall %d\n", current->pid, current->comm, syscall_num); + cpu->eax = _ENOSYS; + } else { +--- a/kernel/signal.c ++++ b/kernel/signal.c +@@ -136,7 +136,6 @@ + lock(&pids_lock); + struct pid *pid = pid_get(pgid); + if (pid == NULL) { +- unlock(&pids_lock); + return _ESRCH; + } + struct tgroup *tgroup; +@@ -694,6 +693,8 @@ + } + + static int kill_task(struct task *task, dword_t sig) { ++ if (current->moose_domain && current->moose_domain != task->moose_domain) ++ return _EPERM; + if (!superuser() && + current->uid != task->uid && + current->uid != task->suid && +--- a/kernel/task.c ++++ b/kernel/task.c +@@ -6,6 +6,8 @@ + #include "kernel/task.h" + #include "kernel/memory.h" + #include "emu/tlb.h" ++#include "fs/fd.h" ++#include "kernel/mm.h" + + __thread struct task *current; + +@@ -17,7 +19,7 @@ + } + + struct pid *pid_get(dword_t id) { +- if (id > sizeof(pids)/sizeof(pids[0])) ++ if (id >= sizeof(pids)/sizeof(pids[0])) + return NULL; + struct pid *pid = &pids[id]; + if (pid_empty(pid)) +@@ -42,6 +44,21 @@ + + struct task *task_create_(struct task *parent) { + lock(&pids_lock); ++ /* Reserve task slots under the PID lock, before copying memory or creating ++ host threads. Count threads and processes together within the domain. */ ++ if (parent && parent->moose_domain) { ++ unsigned owned = 0; ++ for (unsigned i = 1; i <= MAX_PID; i++) { ++ struct task *other = pids[i].task; ++ if (other && other->moose_domain == parent->moose_domain) ++ owned++; ++ } ++ if (owned >= 32) { ++ unlock(&pids_lock); ++ errno = EAGAIN; ++ return NULL; ++ } ++ } + static int cur_pid = 0; + do { + cur_pid++; +@@ -53,8 +70,11 @@ + list_init(&pid->pgroup); + + struct task *task = malloc(sizeof(struct task)); +- if (task == NULL) ++ if (task == NULL) { ++ unlock(&pids_lock); ++ errno = ENOMEM; + return NULL; ++ } + *task = (struct task) {}; + if (parent != NULL) + *task = *parent; +@@ -120,11 +140,42 @@ + pthread_attr_setdetachstate(&task_thread_attr, PTHREAD_CREATE_DETACHED); + } + +-void task_start(struct task *task) { +- if (pthread_create(&task->thread, &task_thread_attr, task_thread, task) < 0) +- die("could not create thread"); ++int task_start(struct task *task) { ++ /* pthread_create returns a positive errno, not -1. Never kill the app. */ ++ int error = pthread_create(&task->thread, &task_thread_attr, task_thread, task); ++ return error ? _EAGAIN : 0; + } + ++/* Fully constructed but never scheduled: no guest instructions or child tasks. */ ++void task_discard_unstarted(struct task *task) { ++ mm_release(task->mm); ++ fdtable_release(task->files); ++ fs_info_release(task->fs); ++ lock(&pids_lock); ++ sighand_release(task->sighand); ++ struct sigqueue *sigqueue, *tmp; ++ list_for_each_entry_safe(&task->queue, sigqueue, tmp, queue) { ++ list_remove(&sigqueue->queue); ++ free(sigqueue); ++ } ++ struct tgroup *group = task->group; ++ lock(&group->lock); ++ list_remove(&task->group_links); ++ bool empty = list_empty(&group->threads); ++ if (empty) { ++ task_leave_session(task); ++ list_remove(&group->pgroup); ++ } ++ unlock(&group->lock); ++ if (empty) { ++ cond_destroy(&group->child_exit); ++ cond_destroy(&group->stopped_cond); ++ free(group); ++ } ++ task_destroy(task); ++ unlock(&pids_lock); ++} ++ + int_t sys_sched_yield(void) { + STRACE("sched_yield()"); + sched_yield(); diff --git a/native/ish/embedded.c b/native/ish/embedded.c new file mode 100644 index 0000000..19bf346 --- /dev/null +++ b/native/ish/embedded.c @@ -0,0 +1,265 @@ +/* iSH embedding owner. Upstream iSH license applies to linked kernel; see + * NOTICE.md. */ +// Preserve upstream header prerequisites (fs/path.h is not standalone). +// clang-format off +#include "embedded.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include "kernel/calls.h" +#include "kernel/init.h" +#include "kernel/task.h" +#include "fs/real.h" +#include "fs/path.h" +#include "fs/devices.h" +// clang-format on + +extern int become_new_init_child_domain(unsigned domain); +struct moose_ish_command { + char *root, *cwd, *executable; + char arguments[32768], environment[32768]; + int argc, fds[3]; + unsigned timeout_ms; + atomic_bool cancelled; +}; +static pthread_mutex_t owner = PTHREAD_MUTEX_INITIALIZER; +static struct task *supervisor; +static unsigned next_domain; +static int pack(int count, const char *const *values, char *out, size_t bound) { + size_t used = 0; + if (count < 0 || count > 1024 || (count && !values)) + return -1; + for (int i = 0; i < count; i++) { + if (!values[i]) + return -1; + size_t n = strnlen(values[i], bound) + 1; + if (n >= bound - used) + return -1; + memcpy(out + used, values[i], n); + used += n; + } + out[used] = 0; + return 0; +} +moose_ish_command *moose_ish_command_create(const moose_ish_options *o) { + if (!o || !o->root || o->root[0] != '/' || !o->cwd || o->cwd[0] != '/' || + o->argc < 1) + return NULL; + moose_ish_command *c = calloc(1, sizeof(*c)); + if (!c) + return NULL; + for (int i = 0; i < 3; i++) + c->fds[i] = -1; + if (pack(o->argc, o->argv, c->arguments, sizeof(c->arguments)) || + pack(o->envc, o->env, c->environment, sizeof(c->environment))) + goto fail; + c->root = strdup(o->root); + c->cwd = strdup(o->cwd); + c->executable = strdup(o->argv[0]); + if (!c->root || !c->cwd || !c->executable) + goto fail; + c->argc = o->argc; + c->timeout_ms = o->timeout_ms; + int fds[] = {o->stdin_fd, o->stdout_fd, o->stderr_fd}; + for (int i = 0; i < 3; i++) + if ((c->fds[i] = dup(fds[i])) < 0) + goto fail; + return c; +fail: + moose_ish_command_destroy(c); + return NULL; +} +void moose_ish_command_cancel(moose_ish_command *c) { + if (c) + atomic_store(&c->cancelled, true); +} +void moose_ish_command_destroy(moose_ish_command *c) { + if (!c) + return; + for (int i = 0; i < 3; i++) + if (c->fds[i] >= 0) + close(c->fds[i]); + free(c->root); + free(c->cwd); + free(c->executable); + free(c); +} +static double milliseconds(void) { + struct timespec now; + clock_gettime(CLOCK_MONOTONIC, &now); + return now.tv_sec * 1000.0 + now.tv_nsec / 1000000.0; +} +static int attach_stdio(moose_ish_command *c) { + for (int i = 0; i < 3; i++) { + struct fd *fd = adhoc_fd_create(&realfs_fdops); + if (!fd) + return -12; + fd->real_fd = dup(c->fds[i]); + fd->dir = NULL; + if (fd->real_fd < 0) { + fd_close(fd); + return -9; + } + current->files->files[i] = fd; + } + return 0; +} +int moose_ish_command_run(moose_ish_command *c, moose_ish_result *result) { + if (!c || !result) + return -1; + *result = (moose_ish_result){0}; + int acquired = 0; + unsigned domain = 0; + struct mount *command_mount = NULL; + double admitted_at = milliseconds(); + while (pthread_mutex_trylock(&owner)) { + if (atomic_load(&c->cancelled)) { + result->cancelled = 1; + result->exit_code = 137; + goto finish; + } + if (c->timeout_ms && milliseconds() - admitted_at >= c->timeout_ms) { + result->timed_out = 1; + result->exit_code = 137; + goto finish; + } + struct timespec pause = {0, 10000000}; + nanosleep(&pause, NULL); + } + acquired = 1; + if (c->timeout_ms && milliseconds() - admitted_at >= c->timeout_ms) { + result->timed_out = 1; + result->exit_code = 137; + goto finish; + } + if (atomic_load(&c->cancelled)) { + result->cancelled = 1; + result->exit_code = 137; + goto finish; + } + int err; + if (!supervisor) { + /* tmpfs records the creating uid; bootstrap before PID1 exists. */ + struct task bootstrap = {0}; + current = &bootstrap; + if ((err = do_mount(&tmpfs, "", "", "", 0)) < 0) + goto error; + if ((err = become_first_process()) < 0) + goto error; + supervisor = current; + supervisor->thread = pthread_self(); + } + current = supervisor; + supervisor->thread = pthread_self(); + domain = ++next_domain; + supervisor->moose_domain = domain; + err = mount_root(&fakefs, c->root); + if (err >= 0) { + char root_path[] = ""; + command_mount = mount_find(root_path); + mount_release(command_mount); + } + supervisor->moose_domain = 0; + if (err < 0) + goto error; + if ((err = become_new_init_child_domain(domain)) < 0) + goto error; + struct task *command_task = current; + int main_pid = current->pid; + /* Create only in-guest devices. No procfs, host mount, socket, or inherited + * fds. */ + generic_mknodat(AT_PWD, "/dev/null", S_IFCHR | 0666, + dev_make(MEM_MAJOR, DEV_NULL_MINOR)); + generic_mknodat(AT_PWD, "/dev/zero", S_IFCHR | 0666, + dev_make(MEM_MAJOR, DEV_ZERO_MINOR)); + generic_mknodat(AT_PWD, "/dev/urandom", S_IFCHR | 0666, + dev_make(MEM_MAJOR, DEV_URANDOM_MINOR)); + generic_mkdirat(AT_PWD, "/workspace", 0777); + struct attr mode = make_attr(mode, 0777); + generic_setattrat(AT_PWD, "/workspace", mode, true); + struct fd *pwd = generic_open(c->cwd, O_RDONLY_, 0); + if (IS_ERR(pwd)) { + err = PTR_ERR(pwd); + goto child_error; + } + fs_chdir(current->fs, pwd); + if ((err = attach_stdio(c)) < 0) + goto child_error; + current->uid = current->euid = current->suid = 501; + current->gid = current->egid = current->sgid = 501; + if ((err = do_execve(c->executable, c->argc, c->arguments, c->environment)) < + 0) + goto child_error; + if ((err = task_start(command_task)) < 0) + goto child_error; + current = supervisor; + double start = admitted_at; + int main_done = 0; + for (;;) { + int alive = 0; + if (atomic_load(&c->cancelled)) + result->cancelled = 1; + if (c->timeout_ms && milliseconds() - start >= c->timeout_ms) + result->timed_out = 1; + lock(&pids_lock); + struct task *main = pid_get_task_zombie(main_pid); + if (main && main->zombie && !main_done) { + int status = main->group->doing_group_exit ? main->group->group_exit_code + : main->exit_code; + result->exit_code = + (status & 127) ? 128 + (status & 127) : (status >> 8) & 255; + main_done = 1; + } + for (int pid = 2; pid <= MAX_PID; pid++) { + struct task *task = pid_get_task_zombie(pid); + if (!task || task->moose_domain != domain || task->zombie) + continue; + alive++; + if (main_done || result->cancelled || result->timed_out) + send_signal(task, SIGKILL_, SIGINFO_NIL); + } + unlock(&pids_lock); + /* Reap only after all tasks have stopped, preserving the primary status. */ + if (!alive) { + while ((int)sys_wait4(-1, 0, 1, 0) > 0) { + } + break; + } + struct timespec pause = {0, 10000000}; + nanosleep(&pause, NULL); + } + if (result->cancelled || result->timed_out) + result->exit_code = 137; + goto finish; +child_error: + result->error = err; + current = supervisor; + task_discard_unstarted(command_task); + goto finish; +error: + result->error = err; +finish: + if (command_mount) { + lock(&mounts_lock); + int removed = mount_remove(command_mount); + unlock(&mounts_lock); + if (removed < 0 && !result->error) + result->error = removed; + } + current = NULL; + /* Drop command-owned pipe duplicates before returning so pumps observe EOF. + */ + for (int i = 0; i < 3; i++) + if (c->fds[i] >= 0) { + close(c->fds[i]); + c->fds[i] = -1; + } + if (acquired) + pthread_mutex_unlock(&owner); + return 0; +} diff --git a/native/ish/embedded.h b/native/ish/embedded.h new file mode 100644 index 0000000..3ba9297 --- /dev/null +++ b/native/ish/embedded.h @@ -0,0 +1,40 @@ +/* Moose's embedding API for pinned iSH; see native/ish/NOTICE.md for licensing. + */ +#ifndef MOOSE_ISH_EMBEDDED_H +#define MOOSE_ISH_EMBEDDED_H +#include +#ifdef __cplusplus +extern "C" { +#endif +typedef struct moose_ish_command moose_ish_command; +typedef struct { + const char *root; /* absolute fakefs data directory; bridge clones full root + incl metadata */ + const char *cwd; /* absolute guest directory */ + int argc; + const char *const *argv; + int envc; + const char *const *env; + int stdin_fd, stdout_fd, + stderr_fd; /* borrowed; core duplicates, never changes host stdio */ + unsigned timeout_ms; /* zero means no timeout; includes queue admission */ +} moose_ish_options; +typedef struct { + int exit_code; /* shell convention: 128 + signal */ + int cancelled; + int timed_out; + int error; /* negative guest errno for setup failures */ +} moose_ish_result; +/* Copies all options; NULL on invalid arguments/allocation failure. */ +moose_ish_command *moose_ish_command_create(const moose_ish_options *options); +/* Blocking, safe on a dedicated pthread. All commands serialize process-wide. + */ +int moose_ish_command_run(moose_ish_command *command, moose_ish_result *result); +/* Thread safe, including before run; destroys no host threads/processes. */ +void moose_ish_command_cancel(moose_ish_command *command); +/* Only after run returns (or before run starts). */ +void moose_ish_command_destroy(moose_ish_command *command); +#ifdef __cplusplus +} +#endif +#endif diff --git a/native/ish/embedded_guest_test.c b/native/ish/embedded_guest_test.c new file mode 100644 index 0000000..5aeb6fa --- /dev/null +++ b/native/ish/embedded_guest_test.c @@ -0,0 +1,86 @@ +/* Freestanding i386 Linux fixture: real guest syscalls, no host/library mocks. + */ +typedef unsigned long u32; +static __attribute__((always_inline)) inline int +call6(int n, int a, int b, int c, int d, int e, int f) { + register int sixth __asm__("ebp") = f; + __asm__ volatile("int $0x80" + : "+a"(n) + : "b"(a), "c"(b), "d"(c), "S"(d), "D"(e), "r"(sixth) + : "memory", "cc"); + return n; +} +static void stop(int code) { + call6(1, code, 0, 0, 0, 0, 0); + for (;;) { + } +} +#define CHECK(value, code) \ + do { \ + if (!(value)) \ + stop(code); \ + } while (0) +void _start(void) { + const int limit = 16 * 1024 * 1024; + const char path[] = "resource-syscalls"; + int fd = call6(5, (int)path, 0100 | 02 | 01000, 0600, 0, 0, 0); + CHECK(fd >= 0, 10); + CHECK(call6(19, fd, limit - 1, 0, 0, 0, 0) == limit - 1, 11); + CHECK(call6(4, fd, (int)"x", 1, 0, 0, 0) == 1, 12); + CHECK(call6(4, fd, (int)"x", 1, 0, 0, 0) == -27, 13); + CHECK(call6(181, fd, (int)"x", 1, limit, 0, 0) == -27, 14); + struct { + const char *base; + u32 size; + } vector = {(const char *)"x", 1}; + CHECK(call6(146, fd, (int)&vector, 1, 0, 0, 0) == -27, 15); + CHECK(call6(194, fd, limit + 1, 0, 0, 0, 0) == -27, 16); + CHECK(call6(193, (int)path, limit + 1, 0, 0, 0, 0) == -27, 17); + call6(6, fd, 0, 0, 0, 0, 0); + const char append_path[] = "resource-append"; + int append_fd = + call6(5, (int)append_path, 0100 | 02 | 01000 | 02000, 0600, 0, 0, 0); + CHECK(append_fd >= 0, 27); + CHECK(call6(181, append_fd, (int)"x", 1, 32 * 1024 * 1024, 0, 0) == -27, 28); + call6(6, append_fd, 0, 0, 0, 0, 0); + CHECK(call6(192, 0, 300 * 1024 * 1024, 3, 0x22, -1, 0) == -12, 18); + int old = call6(45, 0, 0, 0, 0, 0, 0); + CHECK(call6(45, old + 300 * 1024 * 1024, 0, 0, 0, 0, 0) == old, 19); + int small = call6(192, 0, 1024 * 1024, 3, 0x22, -1, 0); + CHECK((u32)small < (u32)-4095, 20); + CHECK(call6(91, small, 1024 * 1024, 0, 0, 0, 0) == 0, 21); + int count = 0; + for (int i = 0; i < 40; i++) { + int child = call6(2, 0, 0, 0, 0, 0, 0); + if (child == 0) + for (;;) + call6(29, 0, 0, 0, 0, 0, 0); + if (child < 0) { + CHECK(child == -11 && count == 31, 22); + break; + } + count++; + } + CHECK(count == 31, 23); + CHECK(call6(37, -1, 9, 0, 0, 0, 0) == 0, 24); + while (call6(114, -1, 0, 0, 0, 0, 0) > 0) { + } + count = 0; + for (int i = 0; i < 40; i++) { + int child = call6(2, 0, 0, 0, 0, 0, 0); + if (child == 0) + stop(0); + if (child < 0) { + CHECK(child == -11 && count == 31, 25); + break; + } + count++; + struct { + int sec, nsec; + } delay = {0, 10000000}; + call6(162, (int)&delay, 0, 0, 0, 0, 0); + } + CHECK(count == 31, 26); + call6(4, 1, (int)"native-resource-syscalls-ok\n", 28, 0, 0, 0); + stop(0); +} diff --git a/native/ish/embedded_host.c b/native/ish/embedded_host.c new file mode 100644 index 0000000..ead1595 --- /dev/null +++ b/native/ish/embedded_host.c @@ -0,0 +1,32 @@ +/* Standalone adapter for testing the same embedded core used by the native + * module. */ +#include "embedded.h" +#include +#include +int main(int argc, char **argv) { + if (argc < 5) + return 125; + int split = 3; + while (split < argc && + !(argv[split][0] == '-' && argv[split][1] == '-' && !argv[split][2])) + split++; + if (split == argc || split + 1 == argc) + return 125; + moose_ish_options o = {argv[1], + argv[2], + argc - split - 1, + (const char *const *)argv + split + 1, + split - 3, + (const char *const *)argv + 3, + 0, + 1, + 2, + 30000}; + moose_ish_command *c = moose_ish_command_create(&o); + if (!c) + return 125; + moose_ish_result r; + int error = moose_ish_command_run(c, &r); + moose_ish_command_destroy(c); + return error || r.error ? 125 : r.exit_code; +} diff --git a/native/ish/embedded_simulator.m b/native/ish/embedded_simulator.m new file mode 100644 index 0000000..8c46805 --- /dev/null +++ b/native/ish/embedded_simulator.m @@ -0,0 +1,67 @@ +/* Dedicated disposable UIKit test application for the real embedded core. */ +#import +#include +extern int moose_ish_embedded_test_main(int, char **); +static NSString *directory; +static void *run_tests(void *unused) { + @autoreleasepool { + NSFileManager *fm = NSFileManager.defaultManager; + NSString *seed = [NSBundle.mainBundle.resourcePath + stringByAppendingPathComponent:@"root"]; + NSMutableArray *paths = [NSMutableArray array]; + for (NSString *name in @[ @"a", @"b" ]) { + NSString *path = [directory stringByAppendingPathComponent:name]; + [fm removeItemAtPath:path error:nil]; + NSError *error = nil; + if (![fm copyItemAtPath:seed toPath:path error:&error]) { + [error.description + writeToFile:[directory stringByAppendingPathComponent:@"error"] + atomically:YES + encoding:NSUTF8StringEncoding + error:nil]; + return NULL; + } + [paths addObject:[path stringByAppendingPathComponent:@"data"]]; + } + char *args[] = {"embedded-test", + (char *)[paths[0] fileSystemRepresentation], + (char *)[paths[1] fileSystemRepresentation]}; + int result = moose_ish_embedded_test_main(3, args); + [@{ + @"exitCode" : @(result), + @"hostAlive" : @YES + } writeToFile:[directory stringByAppendingPathComponent:@"result.plist"] + atomically:YES]; + } + return NULL; +} +@interface EmbeddedDelegate : UIResponder +@property(nonatomic, strong) UIWindow *window; +@end +@implementation EmbeddedDelegate +- (BOOL)application:(UIApplication *)app + didFinishLaunchingWithOptions:(NSDictionary *)options { + self.window = [[UIWindow alloc] initWithFrame:UIScreen.mainScreen.bounds]; + UIViewController *view = [UIViewController new]; + view.view.backgroundColor = UIColor.systemBackgroundColor; + self.window.rootViewController = view; + [self.window makeKeyAndVisible]; + directory = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, + NSUserDomainMask, YES) + .firstObject stringByAppendingPathComponent:@"embedded"]; + [NSFileManager.defaultManager createDirectoryAtPath:directory + withIntermediateDirectories:YES + attributes:nil + error:nil]; + pthread_t thread; + pthread_create(&thread, NULL, run_tests, NULL); + pthread_detach(thread); + return YES; +} +@end +int main(int argc, char **argv) { + @autoreleasepool { + return UIApplicationMain(argc, argv, nil, + NSStringFromClass(EmbeddedDelegate.class)); + } +} diff --git a/native/ish/embedded_test.c b/native/ish/embedded_test.c new file mode 100644 index 0000000..30a0c34 --- /dev/null +++ b/native/ish/embedded_test.c @@ -0,0 +1,303 @@ +/* Real pinned iSH integration harness. No host shell executes guest commands. + */ +#include "embedded.h" +#include "embedded_guest_bytes.h" +#include +#include +#include +#include +#include +#include +#include +static moose_ish_result execute(const char *root, const char *script, + unsigned timeout, char *out) { + char name[] = "/tmp/moose-ish-output-XXXXXX"; + int fd = mkstemp(name); + unlink(name); + int input = open("/dev/null", O_RDONLY); + const char *argv[] = {"/bin/sh", "-c", script}; + const char *env[] = {"PATH=/bin:/usr/bin", "TEST_VALUE=literal value"}; + moose_ish_options options = {root, "/workspace", 3, argv, 2, + env, input, fd, fd, timeout}; + moose_ish_command *command = moose_ish_command_create(&options); + assert(command); + moose_ish_result result = {0}; + assert(moose_ish_command_run(command, &result) == 0); + moose_ish_command_destroy(command); + lseek(fd, 0, SEEK_SET); + ssize_t n = read(fd, out, 4095); + assert(n >= 0); + out[n] = 0; + close(fd); + close(input); + return result; +} +struct pending { + moose_ish_command *command; + moose_ish_result result; +}; +static void *run_pending(void *value) { + struct pending *p = value; + assert(moose_ish_command_run(p->command, &p->result) == 0); + return NULL; +} +static void cancel_descendants(const char *root) { + int fd = open("/dev/null", O_RDWR); + const char *args[] = { + "/bin/sh", "-c", + "setsid sh -c 'sleep 1; echo unsafe > cancelled-child' & sleep 30"}; + moose_ish_options o = {root, "/workspace", 3, args, 0, NULL, fd, fd, fd, 0}; + struct pending p = {moose_ish_command_create(&o), {0}}; + assert(p.command); + pthread_t thread; + assert(!pthread_create(&thread, NULL, run_pending, &p)); + usleep(100000); + moose_ish_command_cancel(p.command); + pthread_join(thread, NULL); + assert(p.result.cancelled && !p.result.timed_out && + p.result.exit_code == 137); + moose_ish_command_destroy(p.command); + close(fd); + sleep(2); + char out[4096]; + moose_ish_result r = + execute(root, "test ! -e cancelled-child && echo alive", 2000, out); + assert(r.exit_code == 0 && !strcmp(out, "alive\n")); +} +static void streams_argv(const char *root) { + char n1[] = "/tmp/moose-stream-out-XXXXXX", + n2[] = "/tmp/moose-stream-err-XXXXXX"; + int out = mkstemp(n1), err = mkstemp(n2); + unlink(n1); + unlink(n2); + int pipes[2]; + assert(!pipe(pipes)); + assert(write(pipes[1], "input", 5) == 5); + close(pipes[1]); + const char *args[] = {"/bin/sh", "-c", + "cat; printf '%s' \"$1\"; echo error >&2", "test", + "literal; $(false)"}; + moose_ish_options o = {root, "/workspace", 5, args, 0, + NULL, pipes[0], out, err, 2000}; + moose_ish_command *c = moose_ish_command_create(&o); + assert(c); + moose_ish_result r; + assert(!moose_ish_command_run(c, &r)); + assert(!r.error && !r.exit_code); + moose_ish_command_destroy(c); + char buffer[100] = {0}; + lseek(out, 0, SEEK_SET); + read(out, buffer, 99); + assert(!strcmp(buffer, "inputliteral; $(false)")); + memset(buffer, 0, 100); + lseek(err, 0, SEEK_SET); + read(err, buffer, 99); + assert(!strcmp(buffer, "error\n")); + close(out); + close(err); + close(pipes[0]); +} +static void initialization_race(const char *first, const char *second) { + int fd = open("/dev/null", O_RDWR); + const char *args[] = {"/bin/true"}; + moose_ish_options a = {first, "/workspace", 1, args, 0, + NULL, fd, fd, fd, 2000}; + moose_ish_options b = a; + b.root = second; + struct pending p = {moose_ish_command_create(&a), {0}}, + q = {moose_ish_command_create(&b), {0}}; + assert(p.command && q.command); + pthread_t t1, t2; + assert(!pthread_create(&t1, NULL, run_pending, &p)); + assert(!pthread_create(&t2, NULL, run_pending, &q)); + pthread_join(t1, NULL); + pthread_join(t2, NULL); + assert(!p.result.error && !p.result.exit_code && !q.result.error && + !q.result.exit_code); + moose_ish_command_destroy(p.command); + moose_ish_command_destroy(q.command); + close(fd); +} +static void cancel_queued(const char *root) { + int output[2]; + assert(!pipe(output)); + int fd = open("/dev/null", O_RDWR); + const char *long_args[] = {"/bin/sh", "-c", "echo ready; sleep 2"}; + const char *short_args[] = {"/bin/true"}; + moose_ish_options a = {root, "/workspace", 3, long_args, 0, + NULL, fd, output[1], fd, 0}; + moose_ish_options b = {root, "/workspace", 1, short_args, 0, + NULL, fd, fd, fd, 0}; + struct pending p = {moose_ish_command_create(&a), {0}}, + q = {moose_ish_command_create(&b), {0}}; + assert(p.command && q.command); + pthread_t t1, t2; + assert(!pthread_create(&t1, NULL, run_pending, &p)); + char ready[6]; + assert(read(output[0], ready, 6) > 0); + assert(!pthread_create(&t2, NULL, run_pending, &q)); + usleep(30000); + struct timespec begin, end; + clock_gettime(CLOCK_MONOTONIC, &begin); + moose_ish_command_cancel(q.command); + pthread_join(t2, NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + assert((end.tv_sec - begin.tv_sec) * 1000 + + (end.tv_nsec - begin.tv_nsec) / 1000000 < + 500); + assert(q.result.cancelled); + const char *queued_args[] = {"/bin/sh", "-c", "echo unsafe > queued-started"}; + moose_ish_options queued = {root, "/workspace", 3, queued_args, 0, + NULL, fd, fd, fd, 30}; + moose_ish_command *expired = moose_ish_command_create(&queued); + assert(expired); + moose_ish_result timeout_result; + assert(!moose_ish_command_run(expired, &timeout_result)); + assert(timeout_result.timed_out && timeout_result.exit_code == 137); + moose_ish_command_destroy(expired); + moose_ish_command_cancel(p.command); + pthread_join(t1, NULL); + moose_ish_command_destroy(p.command); + moose_ish_command_destroy(q.command); + close(fd); + close(output[0]); + close(output[1]); + char out[4096]; + moose_ish_result check = + execute(root, "test ! -e queued-started && echo queue-safe", 1000, out); + assert(!check.error && !check.exit_code && strstr(out, "queue-safe")); +} +static void direct_sleep_timeout(const char *root) { + int fd = open("/dev/null", O_RDWR); + const char *args[] = {"/bin/sleep", "2"}; + moose_ish_options options = {root, "/workspace", 2, args, 0, + NULL, fd, fd, fd, 20}; + for (int i = 0; i < 5; i++) { + moose_ish_command *command = moose_ish_command_create(&options); + assert(command); + struct timespec begin, end; + clock_gettime(CLOCK_MONOTONIC, &begin); + moose_ish_result result; + assert(!moose_ish_command_run(command, &result)); + clock_gettime(CLOCK_MONOTONIC, &end); + assert(result.timed_out && result.exit_code == 137); + assert((end.tv_sec - begin.tv_sec) * 1000 + + (end.tv_nsec - begin.tv_nsec) / 1000000 < + 500); + moose_ish_command_destroy(command); + } + close(fd); +} +static void guest_resource_syscalls(const char *root) { + char name[] = "/tmp/moose-fixture-input-XXXXXX"; + int input = mkstemp(name); + unlink(name); + assert(input >= 0); + assert(write(input, guest_resource_test, sizeof(guest_resource_test)) == + sizeof(guest_resource_test)); + lseek(input, 0, SEEK_SET); + int fd = open("/dev/null", O_RDWR); + const char *args[] = {"/bin/sh", "-c", + "cat > native-resource-test; chmod 700 " + "native-resource-test; exec ./native-resource-test"}; + moose_ish_options options = {root, "/workspace", 3, args, 0, + NULL, input, fd, fd, 3000}; + moose_ish_command *command = moose_ish_command_create(&options); + assert(command); + moose_ish_result result; + assert(!moose_ish_command_run(command, &result)); + if (result.error || result.exit_code) + fprintf(stderr, "resource fixture error=%d status=%d timeout=%d\n", + result.error, result.exit_code, result.timed_out); + assert(!result.error && !result.exit_code && !result.timed_out); + moose_ish_command_destroy(command); + close(fd); + close(input); +} +static int count_host_fds(void) { + int count = 0; + for (int fd = 0; fd < 1024; fd++) + if (fcntl(fd, F_GETFD) >= 0) + count++; + return count; +} +static void steady_descriptors(const char *root) { + int before = count_host_fds(); + char out[4096]; + for (int i = 0; i < 12; i++) { + moose_ish_result r = execute(root, "true", 1000, out); + assert(!r.error && !r.exit_code); + } + assert(count_host_fds() <= before); +} +int main(int argc, char **argv) { + assert(argc == 3); + char out[4096]; + initialization_race(argv[1], argv[2]); + moose_ish_result r = + execute(argv[1], "printf '%s' \"$TEST_VALUE\"; exit 7", 2000, out); + assert(r.error == 0 && r.exit_code == 7 && !strcmp(out, "literal value")); + r = execute(argv[1], "echo first > state; cat state | tr a-z A-Z", 2000, out); + assert(r.exit_code == 0 && !strcmp(out, "FIRST\n")); + r = execute(argv[2], "test ! -e state; echo second > state; cat state", 2000, + out); + assert(r.exit_code == 0 && !strcmp(out, "second\n")); + r = execute(argv[1], "cat state; kill -9 1; echo survived", 2000, out); + assert(r.exit_code == 0 && strstr(out, "first\n") && strstr(out, "survived")); + r = execute(argv[1], + "setsid sh -c 'sleep 1; echo escaped > escaped' & sleep 30", 80, + out); + assert(r.timed_out && r.exit_code == 137); + sleep(2); + r = execute(argv[1], "test ! -e escaped && echo alive", 2000, out); + assert(r.exit_code == 0 && !strcmp(out, "alive\n")); + r = execute(argv[1], + "mkdir -p /workspace/mnt; if mount -t real /tmp /workspace/mnt; " + "then echo UNSAFE; exit 1; fi; echo denied", + 2000, out); + assert(r.exit_code == 0 && strstr(out, "denied") && !strstr(out, "UNSAFE")); + r = execute(argv[1], "kill -9 32769; kill -9 -- -32769; echo survived", 2000, + out); + assert(r.exit_code == 0 && strstr(out, "survived")); + r = execute(argv[1], + "i=0; while test $i -lt 40; do sleep 30 & i=$((i+1)); done; wait", + 3000, out); + assert(!r.timed_out && r.exit_code != 0 && + strstr(out, "Resource temporarily unavailable")); + r = execute(argv[1], + "dd if=/dev/zero of=resource-file bs=1048576 count=17; code=$?; " + "test $code -ne 0 && test $(stat -c %s resource-file) -eq " + "16777216 && echo file-bounded", + 3000, out); + assert(r.exit_code == 0 && strstr(out, "file-bounded")); + r = execute(argv[1], + "truncate -s 16777217 truncate-file; test $? -ne 0 && echo " + "truncate-bounded", + 2000, out); + assert(r.exit_code == 0 && strstr(out, "truncate-bounded")); + r = execute(argv[1], + "dd if=/dev/zero of=/dev/null bs=300M count=0; test $? -ne 0 && " + "echo memory-bounded", + 2000, out); + assert(r.exit_code == 0 && strstr(out, "memory-bounded")); + guest_resource_syscalls(argv[1]); + direct_sleep_timeout(argv[1]); + cancel_queued(argv[1]); + cancel_descendants(argv[1]); + streams_argv(argv[1]); + r = execute( + argv[1], + "ln -sf /tmp/moose-core-b/data/workspace/state leak; test ! -r leak && " + "test ! -r /proc/1/root && test $(id -u) = 501 && echo isolated", + 2000, out); + assert(r.exit_code == 0 && !strcmp(out, "isolated\n")); + r = execute(argv[1], + "if nc -w 1 127.0.0.1 9; then exit 1; fi; echo network-denied", + 2000, out); + assert(r.exit_code == 0 && strstr(out, "network-denied") && + strstr(out, "Operation not permitted")); + steady_descriptors(argv[1]); + puts("PASS embedded real core: repeat, streams/status/env, pipeline, roots, " + "PID1, descendants, timeout"); + return 0; +} diff --git a/package-lock.json b/package-lock.json index fd310a7..879e900 100644 --- a/package-lock.json +++ b/package-lock.json @@ -22,6 +22,7 @@ "expo-sqlite": "~58.0.13", "expo-status-bar": "~58.0.3", "expo-web-browser": "~58.0.6", + "moose-ish": "file:modules/moose-ish", "pi-durable-expo-env": "0.1.0", "react": "19.3.0", "react-dom": "19.3.0", @@ -38,6 +39,10 @@ "vitest": "^4.1.0" } }, + "modules/moose-ish": { + "version": "0.1.0", + "license": "UNLICENSED" + }, "node_modules/@anthropic-ai/sdk": { "version": "0.129.0", "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.129.0.tgz", @@ -6883,6 +6888,10 @@ "node": ">=16 || 14 >=14.17" } }, + "node_modules/moose-ish": { + "resolved": "modules/moose-ish", + "link": true + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", diff --git a/package.json b/package.json index bc711b3..7c04708 100644 --- a/package.json +++ b/package.json @@ -12,7 +12,9 @@ "bundle:runtime": "node scripts/bundle-runtime.mjs", "export": "npm run bundle:runtime && expo export --platform ios --platform android", "smoke:portable": "npm run bundle:runtime && node scripts/smoke-portable.mjs", - "verify": "npm run bundle:runtime && npm test && npm run typecheck && npm run smoke:portable && npm run export" + "verify": "npm run bundle:runtime && npm test && npm run typecheck && npm run smoke:portable && npm run export", + "prepare:ish": "python3 scripts/ish/prepare_native.py --cache /tmp/moose-ish-embedded", + "test:ish": "node scripts/ish/verify.mjs" }, "dependencies": { "@earendil-works/chord": "1.3.0", @@ -29,6 +31,7 @@ "expo-sqlite": "~58.0.13", "expo-status-bar": "~58.0.3", "expo-web-browser": "~58.0.6", + "moose-ish": "file:modules/moose-ish", "pi-durable-expo-env": "0.1.0", "react": "19.3.0", "react-dom": "19.3.0", diff --git a/scripts/ish/audit_guest_boundary.py b/scripts/ish/audit_guest_boundary.py index 8e957c7..dc1cc2f 100644 --- a/scripts/ish/audit_guest_boundary.py +++ b/scripts/ish/audit_guest_boundary.py @@ -17,8 +17,8 @@ sentinel = b'controlled-host-sentinel' Path(directory, 'marker').write_bytes(sentinel) result = subprocess.run([str(a.runner.resolve()), str(a.root.resolve()), '/', '--', - '/bin/sh', '-c', 'printf "MOOSE_AUDIT_READY\\n"; mkdir -p /mnt/boundary || exit 92; ' - 'if mount -t real "$1" /mnt/boundary; then cat /mnt/boundary/marker; ' + '/bin/sh', '-c', 'printf "MOOSE_AUDIT_READY\\n"; mkdir -p /workspace/.mount-boundary || exit 92; ' + 'if mount -t real "$1" /workspace/.mount-boundary; then cat /workspace/.mount-boundary/marker; ' 'else printf "MOOSE_AUDIT_DENIED\\n"; exit 17; fi', 'audit', directory], env={}, capture_output=True, timeout=10) if not result.stdout.startswith(b'MOOSE_AUDIT_READY\n'): diff --git a/scripts/ish/build_embedded.py b/scripts/ish/build_embedded.py new file mode 100644 index 0000000..f9f3fd6 --- /dev/null +++ b/scripts/ish/build_embedded.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""Build reproducibly patched, pinned embedded iSH without modifying pristine source.""" +import argparse +import hashlib +import plistlib +from pathlib import Path +import shutil +import subprocess +import sys +from build_probe import REVISION, ROOTFS_SHA256, REPO, NATIVE, run, capture + + +def main(): + p=argparse.ArgumentParser(description=__doc__) + p.add_argument('--cache',required=True,type=Path) + p.add_argument('--sdk',choices=['macosx','iphonesimulator','iphoneos'],default='macosx') + p.add_argument('--output',required=True,type=Path) + a=p.parse_args(); cache=a.cache.resolve(); output=a.output.resolve() + if cache==REPO or REPO in cache.parents: p.error('--cache must be outside repository') + source=cache/'ish' + if not (cache/'root/data').exists(): + run(sys.executable,REPO/'scripts/ish/build_probe.py','--cache',cache) + if capture('git','-C',source,'rev-parse','HEAD')!=REVISION: raise RuntimeError('Wrong upstream revision') + if capture('git','-C',source,'status','--porcelain','--untracked-files=no'): raise RuntimeError('Modified pristine tree') + patch=NATIVE/'embedded-policy.patch' + identity=hashlib.sha256(patch.read_bytes()).hexdigest() + embedded=cache/'embedded-source' + stamp=embedded/'.moose-policy' + if not stamp.exists() or stamp.read_text()!=identity: + if embedded.exists(): shutil.rmtree(embedded) + embedded.mkdir() + archive=subprocess.check_output(['git','-C',str(source),'archive',REVISION]) + subprocess.run(['tar','-x','-C',str(embedded)],input=archive,check=True) + with patch.open('rb') as stream: + subprocess.run(['patch','-p1','-i',str(patch)],cwd=embedded,check=True) + stamp.write_text(identity) + for old in cache.glob('embedded-build-*'): shutil.rmtree(old) + build=cache/f'embedded-build-{a.sdk}' + flags=['-O2','-DENGINE_ASBESTOS=1',f'-I{embedded}',f'-I{build}',f'-I{NATIVE}'] + common=['-Dbuildtype=release','-Dno_crlf=true'] + if a.sdk!='macosx': + sdk=capture('xcrun','--sdk',a.sdk,'--show-sdk-path') + target='arm64-apple-ios15.0'+('-simulator' if a.sdk=='iphonesimulator' else '') + flags+=['-target',target,'-isysroot',sdk] + cross=cache/f'embedded-{a.sdk}.cross' + cross.write_text("[binaries]\nc = "+repr(['/usr/bin/clang','-target',target,'-isysroot',sdk])+"\nar = '/usr/bin/ar'\nstrip = '/usr/bin/strip'\n[host_machine]\nsystem = 'darwin'\ncpu_family = 'aarch64'\ncpu = 'arm64'\nendian = 'little'\n") + common+=['--cross-file',cross] + if not (build/'build.ninja').exists(): run('meson','setup',build,embedded,*common) + run('ninja','-C',build,'libish.a','libish_emu.a','libfakefs.a') + output.mkdir(parents=True,exist_ok=True) + obj=build/'moose-embedded.o' + run('/usr/bin/clang',*flags,'-c',NATIVE/'embedded.c','-o',obj) + run('libtool','-static','-o',output/'libMooseISHCore.a',obj,*[build/n for n in ['libish.a','libish_emu.a','libfakefs.a']]) + (output/'include').mkdir(exist_ok=True) + shutil.copy2(NATIVE/'embedded.h',output/'include/embedded.h') + seed=cache/'embedded-seed-v2' + if not seed.exists(): + tarball=cache/'alpine.tar.gz' + if hashlib.sha256(tarball.read_bytes()).hexdigest()!=ROOTFS_SHA256: + raise RuntimeError('Rootfs SHA-256 mismatch') + run(cache/'host/tools/fakefsify',tarball,seed) + seed_stamp=output/'.moose-seed-version' + if not seed_stamp.exists() or seed_stamp.read_text()!='2': + if (output/'root').exists(): shutil.rmtree(output/'root') + shutil.copytree(seed,output/'root') + seed_stamp.write_text('2') + if a.sdk in ('macosx','iphonesimulator'): + guest=build/'moose-resource-test' + run('/usr/bin/clang','--target=i386-linux-gnu','-O2','-fomit-frame-pointer', + '-fno-stack-protector','-fno-pic','-ffreestanding','-nostdlib','-static', + '-fuse-ld=lld','-Wl,-e,_start',NATIVE/'embedded_guest_test.c','-o',guest) + data=guest.read_bytes() + (build/'embedded_guest_bytes.h').write_text( + 'static const unsigned char guest_resource_test[] = {'+ + ','.join(map(str,data))+'};\n') + if a.sdk=='macosx': + for name,sourcefile in [('moose-ish-exec','embedded_host.c'),('moose-ish-core-test','embedded_test.c')]: + run('/usr/bin/clang',*flags,NATIVE/sourcefile,output/'libMooseISHCore.a','-lsqlite3','-lpthread','-o',output/name) + if a.sdk=='iphonesimulator': + app=output/'MooseISHEmbeddedTest.app';app.mkdir(exist_ok=True) + testobj=build/'moose-test.o' + run('/usr/bin/clang',*flags,'-Dmain=moose_ish_embedded_test_main','-c',NATIVE/'embedded_test.c','-o',testobj) + run('/usr/bin/clang',*flags,'-fobjc-arc',NATIVE/'embedded_simulator.m',testobj,output/'libMooseISHCore.a','-framework','UIKit','-framework','Foundation','-lsqlite3','-lpthread','-o',app/'MooseISHEmbeddedTest') + with (app/'Info.plist').open('wb') as stream: + plistlib.dump(dict(CFBundleExecutable='MooseISHEmbeddedTest',CFBundleIdentifier='dev.moose.ishembeddedtest',CFBundleName='Moose iSH Embedded Test',CFBundleVersion='1',CFBundleShortVersionString='0.1',CFBundlePackageType='APPL',LSRequiresIPhoneOS=True,MinimumOSVersion='15.0',UIDeviceFamily=[1,2],UILaunchScreen={}),stream) + if (app/'root').exists(): shutil.rmtree(app/'root') + shutil.copytree(output/'root',app/'root') + run('codesign','--force','--sign','-',app) + print(output) + +if __name__=='__main__': main() diff --git a/scripts/ish/prepare_native.py b/scripts/ish/prepare_native.py new file mode 100644 index 0000000..e465dff --- /dev/null +++ b/scripts/ish/prepare_native.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Prepare the optional iOS module's ignored, locally linked iSH artifacts. + +Requires Xcode, Meson, Ninja, LLVM + LLD, libarchive and pkg-config. Does not +publish binaries or decide the whole-program redistribution license. +""" +import argparse +import hashlib +import json +from pathlib import Path +import shutil +import subprocess +import sys +from build_probe import REVISION, ROOTFS_URL, ROOTFS_SHA256, REPO + + +def run(*args): + subprocess.run([str(x) for x in args], check=True) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cache', required=True, type=Path) + args = parser.parse_args() + cache = args.cache.resolve() + if cache == REPO or REPO in cache.parents: + parser.error('--cache must be outside the repository') + vendor = REPO / 'modules/moose-ish/vendor' + stage = cache / 'module-stage' + if stage.exists(): + shutil.rmtree(stage) + stage.mkdir(parents=True) + outputs = [] + for sdk in ('iphoneos', 'iphonesimulator'): + output = cache / ('module-' + sdk) + run(sys.executable, REPO / 'scripts/ish/build_embedded.py', '--cache', cache, + '--sdk', sdk, '--output', output) + outputs.append(output) + run('xcodebuild', '-create-xcframework', + '-library', outputs[0] / 'libMooseISHCore.a', '-headers', outputs[0] / 'include', + '-library', outputs[1] / 'libMooseISHCore.a', '-headers', outputs[1] / 'include', + '-output', stage / 'MooseISHCore.xcframework') + shutil.copytree(outputs[0] / 'root', stage / 'root') + licenses = stage / 'licenses' + licenses.mkdir() + # Exact upstream license texts, not summaries. + source = cache / 'ish' + for name in ('LICENSE.md', 'LICENSE.IOS'): + shutil.copy2(source / name, licenses / name) + shutil.copy2(REPO / 'native/ish/NOTICE.md', licenses / 'MOOSE-ISH-NOTICE.md') + tracked = subprocess.check_output(['git', '-C', str(REPO), 'ls-files', + 'native/ish', 'scripts/ish', 'modules/moose-ish'], text=True).splitlines() + # Include uncommitted embedding files too: manifest describes this actual build. + paths = set(tracked) + for directory in ('native/ish', 'scripts/ish', 'modules/moose-ish/ios'): + paths.update(str(path.relative_to(REPO)) for path in (REPO / directory).rglob('*') + if path.is_file() and '__pycache__' not in path.parts) + manifest = { + 'ishRevision': REVISION, + 'ishSource': 'https://github.com/ish-app/ish', + 'rootfs': {'url': ROOTFS_URL, 'sha256': ROOTFS_SHA256, + 'packages': 'Alpine 3.23.0 x86 minirootfs; inspect root/data/lib/apk/db/installed', + 'sourceRepository': 'https://gitlab.alpinelinux.org/alpine/aports/-/tree/3.23-stable'}, + 'localSourceSHA256': {path: hashlib.sha256((REPO / path).read_bytes()).hexdigest() + for path in sorted(paths) if (REPO / path).is_file()}, + 'rebuild': 'python3 scripts/ish/prepare_native.py --cache /absolute/external/cache', + 'redistribution': 'Local build only. Before redistribution supply exact corresponding whole-app source, build tools/instructions, iSH modifications, all package sources/licenses, and select a compatible whole-program license. This manifest is not corresponding source.'} + (stage / 'source-manifest.json').write_text(json.dumps(manifest, indent=2) + '\n') + if vendor.exists(): + shutil.rmtree(vendor) + shutil.copytree(stage, vendor) + print(f'Prepared local module artifacts: {vendor}. Rebuild the iOS development client.') + + +if __name__ == '__main__': + main() diff --git a/scripts/ish/test_embedded.py b/scripts/ish/test_embedded.py new file mode 100644 index 0000000..e190d71 --- /dev/null +++ b/scripts/ish/test_embedded.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Run real same-kernel host suite and optionally the UIKit simulator suite.""" +import argparse +from pathlib import Path +import plistlib +import shutil +import subprocess +import tempfile +import time + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('--core',required=True,type=Path,help='macosx build output') + parser.add_argument('--simulator',type=Path,help='iphonesimulator build output') + args=parser.parse_args() + with tempfile.TemporaryDirectory(prefix='moose-ish-core-test-') as temp: + roots=[] + for name in ['a','b']: + root=Path(temp)/name + shutil.copytree(args.core/'root',root) + roots.append(root/'data') + subprocess.run([str(args.core/'moose-ish-core-test'),*map(str,roots)],check=True,timeout=60) + # Separate process opens the persisted same fakefs, demonstrating reopen. + subprocess.run([str(args.core/'moose-ish-exec'),str(roots[0]),'/workspace','--','/bin/sh','-c',"test \"$(cat state)\" = first"],check=True,timeout=20) + if args.simulator: + app=args.simulator/'MooseISHEmbeddedTest.app' + bundle='dev.moose.ishembeddedtest' + subprocess.run(['xcrun','simctl','install','booted',str(app)],check=True) + subprocess.run(['xcrun','simctl','terminate','booted',bundle],capture_output=True) + container=Path(subprocess.check_output(['xcrun','simctl','get_app_container','booted',bundle,'data'],text=True).strip()) + result=container/'Documents/embedded/result.plist' + result.unlink(missing_ok=True) + subprocess.run(['xcrun','simctl','launch','booted',bundle],check=True) + deadline=time.monotonic()+60 + while not result.exists() and time.monotonic() + execFileSync(command, args, { stdio: "inherit", env }); +run("python3", [ + "scripts/ish/build_embedded.py", + "--cache", + cache, + "--sdk", + "macosx", + "--output", + core, +]); +run("python3", ["scripts/ish/test_embedded.py", "--core", core]); +run("python3", ["modules/moose-ish/tests/run_bridge.py", "--core", core]); +run( + process.execPath, + [ + "node_modules/vitest/vitest.mjs", + "run", + "tests/ish.test.ts", + "tests/ish-native.test.ts", + ], + { ...process.env, MOOSE_ISH_CORE: core }, +); diff --git a/src/dev/FilesSmoke.tsx b/src/dev/FilesSmoke.tsx index d2c5b6f..055ca07 100644 --- a/src/dev/FilesSmoke.tsx +++ b/src/dev/FilesSmoke.tsx @@ -4,6 +4,7 @@ import { SafeAreaProvider, SafeAreaView } from "react-native-safe-area-context"; import { openDatabaseAsync, deleteDatabaseAsync } from "expo-sqlite"; import { createExpoWorkspace } from "pi-durable-expo-env"; import { SerialSqliteDatabase } from "../generated/runtime"; +import { createIshWorkspaceFactory } from "../platform/ish"; import { runFileSmoke } from "./files-smoke"; /** Explicit developer entry. Never opens the user's database or credentials. */ @@ -22,8 +23,10 @@ export function FilesSmokeApp() { new SerialSqliteDatabase( await openDatabaseAsync("moose-files-smoke.sqlite"), ), - (workspaceId) => - createExpoWorkspace({ workspaceId: `smoke-${workspaceId}` }), + process.env.EXPO_PUBLIC_MOOSE_ISH === "1" + ? createIshWorkspaceFactory("smoke-") + : (workspaceId) => + createExpoWorkspace({ workspaceId: `smoke-${workspaceId}` }), ); const result = `Moose filesystem smoke passed\n\n${checks.map((check) => `✓ ${check}`).join("\n")}`; console.log(result); diff --git a/src/dev/files-smoke.ts b/src/dev/files-smoke.ts index 722c17d..3dd52e2 100644 --- a/src/dev/files-smoke.ts +++ b/src/dev/files-smoke.ts @@ -19,7 +19,7 @@ function sseResponse(body: string): Response { }); return response; } -function responseEvents(step: number) { +function responseEvents(step: number, shell = false) { const calls = [ { name: "write", @@ -32,6 +32,17 @@ function responseEvents(step: number) { edits: [{ oldText: "Hello", newText: "你好" }], }, }, + ...(shell + ? [ + { + name: "bash", + arguments: { + command: + 'printf " guest" >> notes/smoke.txt; cat notes/smoke.txt', + }, + }, + ] + : []), { name: "read", arguments: { path: "notes/smoke.txt" } }, ]; const call = calls[step]; @@ -136,7 +147,7 @@ export async function runFileSmoke( if (request.url !== "https://api.openai.com/v1/responses") throw new Error("Unexpected fixture transport URL"); requests.push(await request.text()); - return respond(responseEvents(step++)); + return respond(responseEvents(step++, workspaceFactory.shell)); }, ); const settings = { provider: "openai", modelId: "gpt-4.1-mini" }; @@ -159,18 +170,22 @@ export async function runFileSmoke( ) ).wait(engine.context); assert( - receipt.status === "done" && step === 4, - "Moose → OpenAI SSE → Pi write/edit/read", + receipt.status === "done" && step === (workspaceFactory.shell ? 5 : 4), + workspaceFactory.shell + ? "Moose → OpenAI SSE → Pi write/edit/bash/read" + : "Moose → OpenAI SSE → Pi write/edit/read", ); assert( requests[0].includes('"name":"write"') && requests[0].includes('"name":"edit"') && requests[0].includes('"name":"read"') && - !requests[0].includes('"name":"bash"'), + requests[0].includes('"name":"bash"') === !!workspaceFactory.shell, "Only supported tools advertised", ); assert( - requests[3].includes("你好 Moose"), + requests[workspaceFactory.shell ? 4 : 3].includes( + workspaceFactory.shell ? "你好 Moose guest" : "你好 Moose", + ), "Read result reaches the next model request", ); const messages = await engine.messages(chat.id); @@ -206,7 +221,9 @@ export async function runFileSmoke( engine.context, ); assert( - content.ok && content.value === "你好 Moose", + content.ok && + content.value === + (workspaceFactory.shell ? "你好 Moose guest" : "你好 Moose"), "File persists across engine / SQLite reopen", ); const rejected = await reopened.writeFile( diff --git a/src/platform/ish.ts b/src/platform/ish.ts new file mode 100644 index 0000000..37dfd8b --- /dev/null +++ b/src/platform/ish.ts @@ -0,0 +1,41 @@ +import { NativeEventEmitter, NativeModules, Platform } from "react-native"; +import { createIshWorkspace } from "../generated/runtime"; +import type { IshBridge, IshOutput } from "../runtime/ish"; +import type { MobileWorkspaceFactory } from "../runtime/files"; + +/** Explicit opt-in: a missing native build is an error, never a shell substitute. */ +export function createIshWorkspaceFactory(prefix = ""): MobileWorkspaceFactory { + if (Platform.OS !== "ios") + throw new Error("The iSH environment requires iOS"); + const native = NativeModules.MooseISH; + if (!native) + throw new Error( + "iSH native module is missing. Run prepare:ish and rebuild the iOS development client.", + ); + const emitter = new NativeEventEmitter(native); + const bridge: IshBridge = { + prepare: (id) => native.prepare(id), + execute: (...args) => native.execute(...args), + cancel: (id) => native.cancel(id), + cleanup: (id) => native.cleanup(id), + subscribe(listener) { + const subscription = emitter.addListener( + "MooseISHOutput", + (event: Object) => { + const value = event as Partial; + if ( + typeof value.commandId === "string" && + typeof value.data === "string" && + (value.stream === "stdout" || value.stream === "stderr") + ) + listener(value as IshOutput); + }, + ); + return () => subscription.remove(); + }, + }; + return Object.assign( + (id: string) => createIshWorkspace(prefix + id, bridge), + { shell: true }, + ); +} diff --git a/src/platform/open.ts b/src/platform/open.ts index 1557b91..2137a3e 100644 --- a/src/platform/open.ts +++ b/src/platform/open.ts @@ -9,6 +9,7 @@ import { } from "../generated/runtime"; import type { ModelSettings } from "../runtime/engine"; import { codexAuth } from "./codex"; +import { createIshWorkspaceFactory } from "./ish"; export const defaultSettings: ModelSettings = { provider: "openai", @@ -44,7 +45,9 @@ export async function openMobile() { models, defaultSettings, Date.now, - (workspaceId) => createExpoWorkspace({ workspaceId }), + process.env.EXPO_PUBLIC_MOOSE_ISH === "1" + ? createIshWorkspaceFactory() + : (workspaceId) => createExpoWorkspace({ workspaceId }), ); const settings = await engine.read( "settings", diff --git a/src/runtime/engine.ts b/src/runtime/engine.ts index 0bb87b6..d4baed9 100644 --- a/src/runtime/engine.ts +++ b/src/runtime/engine.ts @@ -27,6 +27,7 @@ import { createWriteTool, createEditTool, } from "@earendil-works/pi-durable/tools"; +import { createIshBashTool } from "./ish"; import { MobileFileEnvironments, type MobileWorkspaceFactory } from "./files"; export interface ModelSettings { @@ -169,6 +170,7 @@ export class MobileEngine { private database: SqliteDatabase, private settings: ModelSettings, private files?: MobileFileEnvironments, + private shell = false, ) {} static async open( @@ -188,16 +190,22 @@ export class MobileEngine { const files = workspaceFactory ? new MobileFileEnvironments(workspaceFactory) : undefined; + const shell = workspaceFactory?.shell === true; if (files) registry.install( defineExtension({ name: "moose.files", - tools: [createReadTool(), createWriteTool(), createEditTool()], + tools: [ + createReadTool(), + createWriteTool(), + createEditTool(), + ...(shell ? [createIshBashTool()] : []), + ], sections: [ - section( - "mobile-workspace", - () => - "Use read, write and edit for files inside /workspace in this conversation’s app-private device workspace. Relative paths start at /workspace. Other conversations and new checkpoint branches have separate workspaces. File contents are not part of a conversation checkpoint. Shell, script execution, web search and external document access are unavailable.", + section("mobile-workspace", () => + shell + ? "Use read, write and edit for files inside /workspace and bash for the local iSH POSIX shell. They share this conversation’s persistent guest files. Other conversations and new checkpoint branches have separate roots. Files are not checkpointed. Network access is unavailable. iOS suspension may pause commands." + : "Use read, write and edit for files inside /workspace in this conversation’s app-private device workspace. Relative paths start at /workspace. Other conversations and new checkpoint branches have separate workspaces. File contents are not part of a conversation checkpoint. Shell, script execution, web search and external document access are unavailable.", ), ], }), @@ -213,7 +221,7 @@ export class MobileEngine { }, BACKGROUND_CONTEXT, ); - const engine = new MobileEngine(harness, database, settings, files); + const engine = new MobileEngine(harness, database, settings, files, shell); try { if (files) for (const item of await engine.listConversations()) @@ -228,7 +236,7 @@ export class MobileEngine { const conversation = await this.conversation(id); const instructions = (await conversation.agent(this.context)).instructions; if (instructions) { - const next = upgradeSkillPrompt(instructions); + const next = upgradeSkillPrompt(instructions, this.shell); if (next !== instructions) await conversation.configure({ instructions: next }, this.context); } @@ -265,7 +273,7 @@ export class MobileEngine { ownership: { kind: "ownerless" }, agent: { model: this.settings, - instructions: selectedSkillPrompt([], []), + instructions: selectedSkillPrompt([], [], this.shell), }, }, this.context, @@ -287,7 +295,7 @@ export class MobileEngine { skills: Skill[], selected: string[], ) { - const prompt = selectedSkillPrompt(skills, selected); + const prompt = selectedSkillPrompt(skills, selected, this.shell); await ( await this.conversation(id) ).configure({ model: settings, instructions: prompt }, this.context); @@ -341,7 +349,7 @@ export class MobileEngine { await ( await this.conversation(conversation.id) ).configure( - { instructions: selectedSkillPrompt(installed, next) }, + { instructions: selectedSkillPrompt(installed, next, this.shell) }, this.context, ); await this.write(`skills:${conversation.id}`, next); diff --git a/src/runtime/files.ts b/src/runtime/files.ts index 7a82e1a..236497b 100644 --- a/src/runtime/files.ts +++ b/src/runtime/files.ts @@ -1,7 +1,10 @@ import type { Context } from "@earendil-works/chord"; import type { EnvTarget } from "@earendil-works/pi-durable"; import type { ExecutionEnv } from "@earendil-works/pi-durable/env"; -export type MobileWorkspaceFactory = (workspaceId: string) => ExecutionEnv; +export type MobileWorkspaceFactory = ((workspaceId: string) => ExecutionEnv) & { + /** Explicit capability; never infer shell availability by trying a command. */ + shell?: boolean; +}; /** Owned by one engine; Harness must finish its tools before this pool closes. */ export class MobileFileEnvironments { diff --git a/src/runtime/ish.ts b/src/runtime/ish.ts new file mode 100644 index 0000000..9603dda --- /dev/null +++ b/src/runtime/ish.ts @@ -0,0 +1,707 @@ +import { createBashTool } from "@earendil-works/pi-durable/tools"; +import { + BACKGROUND_CONTEXT, + withAbortSignal, + type Context, +} from "@earendil-works/chord/context"; +import { + createWorkspaceEnv, + type WorkspaceBackend, +} from "pi-durable-expo-env/core"; +import { + ExecutionError, + FileError, + StreamDecoder, + err, + ok, + type ExecutionEnv, + type ShellExecOptions, + type Result, + type ShellExecResult, + type WatchTarget, + type WatchChange, + type FileWatcher, +} from "@earendil-works/pi-durable/env"; + +export interface IshOutput { + commandId: string; + stream: "stdout" | "stderr"; + data: string; +} +export interface IshResult { + exitCode: number; + stdout: string; + stderr: string; + cancelled?: boolean; + timedOut?: boolean; + omittedBytes?: number; +} +export interface IshBridge { + prepare(workspaceId: string): Promise; + execute( + workspaceId: string, + commandId: string, + argv: string[], + cwd: string, + env: Record, + inputBase64: string, + timeoutMs: number, + ): Promise; + subscribe(listener: (event: IshOutput) => void): () => void; + cancel(commandId: string): Promise; + cleanup(workspaceId: string): Promise; +} +const alphabet = + "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; +export function encodeBase64(bytes: Uint8Array) { + const pieces: string[] = []; + for (let start = 0; start < bytes.length; start += 12288) { + let part = ""; + for (let i = start; i < Math.min(start + 12288, bytes.length); i += 3) { + const n = + (bytes[i] << 16) | ((bytes[i + 1] ?? 0) << 8) | (bytes[i + 2] ?? 0); + part += + alphabet[(n >>> 18) & 63] + + alphabet[(n >>> 12) & 63] + + (i + 1 < bytes.length ? alphabet[(n >>> 6) & 63] : "=") + + (i + 2 < bytes.length ? alphabet[n & 63] : "="); + } + pieces.push(part); + } + return pieces.join(""); +} +export function decodeBase64(value: string) { + if ( + value.length % 4 || + /[^A-Za-z0-9+/=]/.test(value) || + !/^[A-Za-z0-9+/]*={0,2}$/.test(value) + ) + throw new Error("Invalid native base64"); + const size = + (value.length / 4) * 3 - + (value.endsWith("==") ? 2 : value.endsWith("=") ? 1 : 0); + const out = new Uint8Array(size); + let offset = 0; + for (let i = 0; i < value.length; i += 4) { + const n = + (alphabet.indexOf(value[i]) << 18) | + (alphabet.indexOf(value[i + 1]) << 12) | + ((alphabet.indexOf(value[i + 2]) & 63) << 6) | + (alphabet.indexOf(value[i + 3]) & 63); + if (offset < size) out[offset++] = (n >>> 16) & 255; + if (offset < size) out[offset++] = (n >>> 8) & 255; + if (offset < size) out[offset++] = n & 255; + } + return out; +} +const encoder = new TextEncoder(); +const text = (b: Uint8Array) => new TextDecoder().decode(b); +const guestEnv = { + PATH: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + LANG: "C", + LC_ALL: "C", +}; +let commandSequence = 0; +const commandId = () => + `ish-${Date.now().toString(36)}-${(++commandSequence).toString(36)}-${Math.random().toString(36).slice(2)}`; + +const durableCommandId = Symbol.for("moose.ish.commandId"); +/** Persist native intent before shell effects; unsafe tools are never replayed by Pi. */ +export function createIshBashTool() { + const tool = createBashTool(); + return { + ...tool, + replay: "unsafe" as const, + async execute( + args: Parameters[0], + api: Parameters[1], + context: Context, + ) { + const stable = await api.memo( + "moose.ish.commandId", + commandId(), + context, + ); + const env = api.env; + if (!env) return tool.execute(args, api, context); + const bound = new Proxy(env, { + get(target, key) { + if (key === "exec") + return ( + command: string | readonly string[], + options: ShellExecOptions | undefined, + ctx: Context, + ) => + target.exec( + command, + { ...options, [durableCommandId]: stable } as ShellExecOptions, + ctx, + ); + return Reflect.get(target, key); + }, + }); + return tool.execute(args, { ...api, env: bound }, context); + }, + }; +} + +/** Files and shell use one jailed guest root. No direct host fakefs access. */ +export function createIshWorkspace( + workspaceId: string, + bridge: IshBridge, +): ExecutionEnv { + if (!/^[A-Za-z0-9_-]{1,64}$/.test(workspaceId)) + throw new Error("Invalid iSH workspace ID"); + const rootUri = `file:///moose-ish/${workspaceId}`; + let closing: Promise | undefined; + let ready: Promise | undefined, + closed = false; + let tail: Promise = Promise.resolve(); + let operationContext: Context = BACKGROUND_CONTEXT; + const prepare = () => + (ready ??= bridge.prepare(workspaceId).catch((e) => { + ready = undefined; + throw e; + })); + const path = (uri: string) => { + if (uri !== rootUri && !uri.startsWith(rootUri + "/")) + throw new FileError("invalid", "Invalid guest URI"); + return "/workspace" + decodeURIComponent(uri.slice(rootUri.length)); + }; + async function invoke( + argv: string[], + cwd: string, + env: Record, + input: Uint8Array, + timeoutMs: number, + context: Context, + onChunk?: (stream: "stdout" | "stderr", bytes: Uint8Array) => void, + stableId?: string, + ) { + context.abortSignal?.throwIfAborted(); + await prepare(); + context.abortSignal?.throwIfAborted(); + const id = stableId ?? commandId(); + let callbackError: unknown; + const cancel = () => { + void bridge.cancel(id).catch(() => {}); + }; + const unsubscribe = bridge.subscribe((event) => { + if (event.commandId !== id || callbackError) return; + try { + onChunk?.(event.stream, decodeBase64(event.data)); + } catch (error) { + callbackError = error; + cancel(); + } + }); + context.abortSignal?.addEventListener("abort", cancel, { once: true }); + try { + const pending = bridge.execute( + workspaceId, + id, + argv, + cwd, + env, + encodeBase64(input), + timeoutMs, + ); + if (context.abortSignal?.aborted) cancel(); + const result = await pending; + if (callbackError) + throw new ExecutionError( + "callback_error", + "Shell output callback failed", + callbackError instanceof Error ? callbackError : undefined, + ); + if (context.abortSignal?.aborted) + throw new ExecutionError("aborted", "Command cancelled"); + if (result.omittedBytes) + throw new ExecutionError("unknown", "Native output limit exceeded"); + if (result.timedOut) + throw new ExecutionError("timeout", "Command timed out"); + if (result.cancelled) + throw new ExecutionError("aborted", "Command cancelled"); + return result; + } finally { + context.abortSignal?.removeEventListener("abort", cancel); + unsubscribe(); + } + } + async function guest( + argv: string[], + input: Uint8Array = new Uint8Array(), + context = operationContext, + ) { + const result = await invoke( + argv, + "/workspace", + guestEnv, + input, + 30000, + context, + ).catch((error: unknown) => { + throw new FileError( + context.abortSignal?.aborted || + (error instanceof ExecutionError && error.code === "aborted") + ? "aborted" + : "unknown", + error instanceof Error + ? error.message + : "Native filesystem operation failed", + ); + }); + if (result.exitCode !== 0) { + const message = text(decodeBase64(result.stderr)); + const code = + result.exitCode === 44 || /No such file/.test(message) + ? "not_found" + : /Permission denied/.test(message) + ? "permission_denied" + : /Not a directory/.test(message) + ? "not_directory" + : /Is a directory/.test(message) + ? "is_directory" + : "unknown"; + throw new FileError(code, message || "Guest filesystem operation failed"); + } + return decodeBase64(result.stdout); + } + const backend: WorkspaceBackend = { + async stat(uri) { + try { + const output = text( + await guest([ + "/bin/sh", + "-c", + 'if [ -e "$1" ] || [ -L "$1" ]; then /bin/busybox stat -L -c "%F\n%s\n%Y" -- "$1"; else exit 44; fi', + "stat", + path(uri), + ]), + ); + const [kind, size, mtime] = output.trimEnd().split("\n"); + if (kind !== "directory" && !kind.startsWith("regular")) + throw new FileError( + "invalid", + "Only regular files and directories are supported", + ); + return { + kind: kind === "directory" ? "directory" : "file", + size: Number(size), + mtimeMs: Number(mtime) * 1000, + }; + } catch (e) { + if (e instanceof FileError && e.code === "not_found") return undefined; + throw e; + } + }, + async read(uri, limit, signal) { + const data = await guest( + ["/bin/busybox", "head", "-c", String(limit + 1), "--", path(uri)], + undefined, + withAbortSignal(signal, BACKGROUND_CONTEXT), + ); + if (data.length > limit) + throw new FileError("invalid", "File exceeds byte limit"); + return data; + }, + async write(uri, bytes) { + await guest(["/bin/sh", "-c", 'cat > "$1"', "write", path(uri)], bytes); + }, + async mkdir(uri, recursive) { + await guest([ + "/bin/busybox", + "mkdir", + ...(recursive ? ["-p"] : []), + "--", + path(uri), + ]); + }, + async names(uri) { + const parent = path(uri); + const data = text( + await guest([ + "/bin/busybox", + "find", + parent, + "-mindepth", + "1", + "-maxdepth", + "1", + "-print0", + ]), + ); + return data + .split("\0") + .filter(Boolean) + .map((value) => value.slice(parent.length + 1)); + }, + async remove(uri, recursive) { + if (recursive) await guest(["/bin/busybox", "rm", "-r", "--", path(uri)]); + else + await guest([ + "/bin/sh", + "-c", + 'if [ -d "$1" ] && ! [ -L "$1" ]; then rmdir -- "$1"; else rm -- "$1"; fi', + "remove", + path(uri), + ]); + }, + async move(source, destination) { + await guest([ + "/bin/busybox", + "mv", + "--", + path(source), + path(destination), + ]); + }, + }; + const files = createWorkspaceEnv({ backend, rootUri }); + Object.defineProperty(files, "id", { value: `ish:${workspaceId}` }); + async function exec( + command: string | readonly string[], + options: ShellExecOptions | undefined, + context: Context, + ) { + const requestedCwd = options?.cwd ?? files.cwd; + if (typeof requestedCwd !== "string" || requestedCwd.includes("\0")) + return err( + new ExecutionError("spawn_error", "Invalid working directory"), + ); + const components: string[] = []; + for (const part of (requestedCwd.startsWith("/") + ? requestedCwd + : files.cwd + "/" + requestedCwd + ).split("/")) { + if (part === "..") components.pop(); + else if (part && part !== ".") components.push(part); + } + const cwd = "/" + components.join("/"); + if (closed) + return err(new ExecutionError("spawn_error", "Environment is closed")); + if (context.abortSignal?.aborted) + return err(new ExecutionError("aborted", "Command cancelled")); + if ( + (typeof command !== "string" && + (!command.length || + command.some((s) => typeof s !== "string" || s.includes("\0")))) || + (typeof command === "string" && command.includes("\0")) + ) + return err(new ExecutionError("spawn_error", "Invalid command")); + if ( + options?.timeout !== undefined && + (!Number.isFinite(options.timeout) || + options.timeout <= 0 || + options.timeout > 86400) + ) + return err( + new ExecutionError( + "spawn_error", + "Timeout must be positive seconds, at most 86400", + ), + ); + let argv = + typeof command === "string" ? ["/bin/sh", "-c", command] : [...command]; + if (!argv[0].startsWith("/")) { + try { + const found = await invoke( + ["/bin/sh", "-c", 'command -v "$1"', "resolve", argv[0]], + cwd, + { ...guestEnv, ...options?.env }, + new Uint8Array(), + 30000, + context, + ); + if (found.exitCode !== 0) + return err( + new ExecutionError("spawn_error", "Guest program not found"), + ); + } catch (e) { + return err( + e instanceof ExecutionError + ? e + : new ExecutionError( + "spawn_error", + "Unable to resolve guest program", + ), + ); + } + argv = ["/bin/sh", "-c", 'exec "$@"', "exec", ...argv]; + } + const decoders = { + stdout: new StreamDecoder(), + stderr: new StreamDecoder(), + }; + const captured: string[] = []; + const emit = (stream: "stdout" | "stderr", value: string) => { + if (value) { + captured.push(value); + options?.onOutput?.(value, context, { stream }); + } + }; + let outcome: Result; + try { + const result = await invoke( + argv, + cwd, + { ...guestEnv, ...options?.env }, + new Uint8Array(), + Math.min(2147483647, Math.ceil((options?.timeout ?? 120) * 1000)), + context, + (stream, bytes) => emit(stream, decoders[stream].decode(bytes)), + ( + options as + (ShellExecOptions & { [durableCommandId]?: string }) | undefined + )?.[durableCommandId], + ); + emit("stdout", decoders.stdout.decode()); + emit("stderr", decoders.stderr.decode()); + outcome = ok({ exitCode: result.exitCode } as { + exitCode: number; + spillPath?: string; + }); + } catch (e) { + const error = + e instanceof ExecutionError + ? e + : new ExecutionError( + context.abortSignal?.aborted + ? "aborted" + : (e as { code?: string })?.code === "spawn_error" + ? "spawn_error" + : "unknown", + e instanceof Error ? e.message : "Native execution failed", + ); + outcome = err(error); + } + const combined = captured.join(""); + if ( + options?.spill && + (encoder.encode(combined).length > options.spill.afterBytes || + combined.split("\n").length > options.spill.afterLines) + ) { + try { + const spill = await files.createTempFile( + { prefix: "shell-", suffix: ".log" }, + BACKGROUND_CONTEXT, + ); + if (!spill.ok) throw spill.error; + const bytes = encoder.encode(combined); + if (bytes.length > 16 * 1024 * 1024) + throw new Error("Decoded output exceeds the 16 MiB spill limit"); + // Spill artifacts may exceed the 8 MiB file-tool transfer bound, while + // remaining within the actual guest regular-file/native-input limits. + await guest( + ["/bin/sh", "-c", 'cat > "$1"', "spill", spill.value], + bytes, + BACKGROUND_CONTEXT, + ); + if (outcome.ok) outcome.value.spillPath = spill.value; + else outcome.error.spillPath = spill.value; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + if (outcome.ok) + outcome = err( + new ExecutionError( + "unknown", + `Command finished (exit ${outcome.value.exitCode}); full output could not be saved: ${message}`, + ), + ); + else + outcome.error.message += `; full output could not be saved: ${message}`; + } + } + + return outcome; + } + const watchers = new Set<() => void>(); + const originalReader = files.openBinaryReader.bind(files); + files.openBinaryReader = async (p, options, context) => { + if (context.abortSignal?.aborted) + return err(new FileError("aborted", "Operation cancelled")); + if (options?.noFollow) { + const absolute = await files.absolutePath(p, context); + if (!absolute.ok) return absolute; + try { + const result = await invoke( + ["/bin/sh", "-c", 'test ! -L "$1"', "nofollow", absolute.value], + "/workspace", + guestEnv, + new Uint8Array(), + 30000, + context, + ); + if (result.exitCode !== 0) + return err( + new FileError("invalid", "Final path is a symbolic link", p), + ); + } catch (e) { + return err(new FileError("unknown", String(e), p)); + } + } + return originalReader(p, undefined, context); + }; + files.flushFile = async (p, context) => { + const absolute = await files.absolutePath(p, context); + if (!absolute.ok) return absolute; + try { + await guest( + ["/bin/busybox", "sync", "-f", absolute.value], + undefined, + context, + ); + return ok(undefined); + } catch (e) { + return err( + e instanceof FileError ? e : new FileError("unknown", String(e), p), + ); + } + }; + async function snapshot(target: WatchTarget, context: Context) { + const absolute = await files.absolutePath(target.path, context); + if (!absolute.ok) throw absolute.error; + const script = + 'if ! [ -e "$1" ] && ! [ -L "$1" ]; then exit 0; fi; ' + + 'find "$1" ' + + (target.recursive ? "" : "-maxdepth 1 ") + + `-exec /bin/sh -c 'for p do if [ -f "$p" ]; then printf "f "; cksum < "$p"; elif [ -d "$p" ]; then printf d; else printf l; fi; printf "\\0%s\\0" "$p"; done' watch {} +`; + const output = await guest( + ["/bin/sh", "-c", script, "watch", absolute.value], + undefined, + context, + ); + if (output.length > 1024 * 1024) + throw new FileError("invalid", "Watch exceeds snapshot bound"); + const parts = text(output).split("\0"); + if (parts.length > 20001) + throw new FileError("invalid", "Watch exceeds entry bound"); + const entries: string[] = []; + for (let i = 0; i + 1 < parts.length; i += 2) { + const name = parts[i + 1]; + const relative = name.slice(absolute.value.length + 1); + if ( + name !== absolute.value && + relative + .split("/") + .some( + (segment) => + (target.exclude?.hidden && segment.startsWith(".")) || + target.exclude?.names?.includes(segment), + ) + ) + continue; + entries.push(JSON.stringify([name, parts[i]])); + } + return entries.sort().join("\n"); + } + files.watch = async (targets, onChange, context) => { + try { + context.abortSignal?.throwIfAborted(); + if (closed) throw new FileError("invalid", "Environment is closed"); + const signatures = await Promise.all( + targets.map((target) => snapshot(target, context)), + ); + let stopped = false, + pending: Promise | undefined; + const stop = () => { + stopped = true; + clearInterval(timer); + watchers.delete(stop); + }; + const timer = setInterval(() => { + if (stopped || pending) return; + pending = (async () => { + for (let i = 0; i < targets.length && !stopped; i++) { + const next = await snapshot(targets[i], BACKGROUND_CONTEXT); + if (next !== signatures[i]) { + signatures[i] = next; + const absolute = await files.absolutePath( + targets[i].path, + BACKGROUND_CONTEXT, + ); + if (absolute.ok && !stopped) + onChange({ paths: [absolute.value] }); + } + } + })() + .catch((error) => { + if (!stopped) { + stop(); + onChange({ + error: + error instanceof FileError + ? error + : new FileError("unknown", String(error)), + }); + } + }) + .finally(() => { + pending = undefined; + }); + }, 250); + watchers.add(stop); + return ok({ + mode: "polling" as const, + async close() { + stop(); + await pending; + }, + }); + } catch (e) { + return err( + e instanceof FileError + ? e + : new FileError( + context.abortSignal?.aborted ? "aborted" : "unknown", + String(e), + ), + ); + } + }; + const originalCleanup = files.cleanup.bind(files); + const environment = Object.assign(files, { exec }); + // Serialize complete file operations against shell writes, not only individual backend calls. + const wrapped = new Proxy(environment, { + get(target, key, receiver) { + if (key === "cleanup") + return (context: Context) => { + if (closing) return closing; + closed = true; + for (const stop of [...watchers]) stop(); + closing = (async () => { + let failure: unknown; + try { + await bridge.cleanup(workspaceId); + } catch (error) { + failure = error; + } + await tail.catch(() => {}); + await originalCleanup(context); + if (failure) throw failure; + })(); + return closing; + }; + const value = Reflect.get(target, key, receiver); + if (typeof value !== "function") return value; + return (...args: unknown[]) => { + const context = args.at(-1) as Context; + const run = tail + .catch(() => {}) + .then(async () => { + if (closed) + return key === "exec" + ? err( + new ExecutionError("spawn_error", "Environment is closed"), + ) + : err(new FileError("invalid", "Environment is closed")); + operationContext = context ?? BACKGROUND_CONTEXT; + return value.apply(target, args); + }); + tail = run; + return run; + }; + }, + }); + return wrapped; +} diff --git a/src/runtime/portable-entry.ts b/src/runtime/portable-entry.ts index 112ecd2..e738e1c 100644 --- a/src/runtime/portable-entry.ts +++ b/src/runtime/portable-entry.ts @@ -7,3 +7,5 @@ export { } from "../skills/skills"; export { createMobileModels } from "./providers"; export { CodexAuth, CodexAuthError } from "../auth/codex"; + +export { createIshWorkspace } from "./ish"; diff --git a/src/skills/skills.ts b/src/skills/skills.ts index a377adc..116f01a 100644 --- a/src/skills/skills.ts +++ b/src/skills/skills.ts @@ -104,15 +104,21 @@ const legacyPreamble = "You are Moose, a helpful mobile assistant. Reply in the user’s language. Skills provide instructions only. No shell, scripts, filesystem tools or external skill resources are available. Never claim to have executed them."; const mobilePreamble = "You are Moose, a helpful mobile assistant. Reply in the user’s language. Skills provide instructions only. Use only tools exposed in this conversation. Shell, script execution and external Skill resource loading are unavailable. Never claim to have executed unavailable tools."; -export function upgradeSkillPrompt(prompt: string) { - return prompt.startsWith(legacyPreamble) - ? mobilePreamble + prompt.slice(legacyPreamble.length) +const shellPreamble = + "You are Moose, a helpful mobile assistant. Reply in the user’s language. Skills provide instructions only. Use only tools exposed in this conversation. The local iSH guest provides a POSIX shell; network access and external Skill resource loading are unavailable. Never claim to have executed unavailable tools."; +export function upgradeSkillPrompt(prompt: string, shell = false) { + const previous = [legacyPreamble, mobilePreamble, shellPreamble].find((p) => + prompt.startsWith(p), + ); + return previous + ? (shell ? shellPreamble : mobilePreamble) + prompt.slice(previous.length) : prompt; } export function selectedSkillPrompt( skills: Skill[], selected: string[], + shell = false, ): string { const enabled = skills.filter((skill) => selected.includes(skill.id)); const total = enabled.reduce( @@ -122,7 +128,7 @@ export function selectedSkillPrompt( if (total > 65536) throw new Error("Selected skills exceed the 64 KiB prompt limit"); return [ - mobilePreamble, + shell ? shellPreamble : mobilePreamble, ...enabled.map( (skill) => `\n\n${skill.instructions}\n`, diff --git a/tests/ish-native.test.ts b/tests/ish-native.test.ts new file mode 100644 index 0000000..5fd1dcf --- /dev/null +++ b/tests/ish-native.test.ts @@ -0,0 +1,330 @@ +import { + createEnvConformance, + createExpectAssertions, +} from "@earendil-works/pi-durable/testing"; +import { runFileSmoke } from "../src/dev/files-smoke"; +import { DatabaseSync } from "node:sqlite"; +import { createModels } from "@earendil-works/pi-ai/models"; +import { + fauxProvider, + fauxAssistantMessage, + fauxToolCall, +} from "@earendil-works/pi-ai/providers/faux"; +import { MobileEngine } from "../src/runtime/engine"; +import { SerialSqliteDatabase } from "../src/storage/database"; +import { afterEach, describe, expect, it } from "vitest"; +import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process"; +import { cp, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + BACKGROUND_CONTEXT, + withAbortSignal, +} from "@earendil-works/chord/context"; +import { + createIshWorkspace, + type IshBridge, + type IshOutput, + type IshResult, + encodeBase64, + decodeBase64, +} from "../src/runtime/ish"; +import { getOrThrow } from "@earendil-works/pi-durable/env"; + +const core = process.env.MOOSE_ISH_CORE; +const cleanups: (() => Promise)[] = []; +afterEach(async () => { + for (const fn of cleanups.splice(0).reverse()) await fn(); +}); +async function fixture() { + const directory = await mkdtemp(join(tmpdir(), "moose-pi-ish-")); + cleanups.push(() => rm(directory, { recursive: true, force: true })); + const roots = new Set(), + listeners = new Set<(e: IshOutput) => void>(); + const jobs = new Map(); + const cancelledIds = new Set(); + let executionTail: Promise = Promise.resolve(); + const native: IshBridge = { + async prepare(id) { + if (!roots.has(id)) { + await cp(join(core!, "root"), join(directory, id), { recursive: true }); + roots.add(id); + } + }, + subscribe(fn) { + listeners.add(fn); + return () => { + listeners.delete(fn); + }; + }, + execute(id, command, argv, cwd, env, input, timeout) { + const pending = executionTail + .catch(() => {}) + .then( + () => + new Promise((resolve, reject) => { + if (cancelledIds.has(command)) { + resolve({ + exitCode: 137, + stdout: "", + stderr: "", + cancelled: true, + }); + return; + } + const child = spawn( + join(core!, "moose-ish-exec"), + [ + join(directory, id, "data"), + cwd, + ...Object.entries(env).map(([k, v]) => `${k}=${v}`), + "--", + ...argv, + ], + { env: { NODE_ENV: "test" }, stdio: ["pipe", "pipe", "pipe"] }, + ); + jobs.set(command, child); + const output = { stdout: [] as Buffer[], stderr: [] as Buffer[] }; + for (const stream of ["stdout", "stderr"] as const) + child[stream]!.on("data", (bytes: Buffer) => { + output[stream].push(bytes); + for (const fn of listeners) + fn({ + commandId: command, + stream, + data: bytes.toString("base64"), + }); + }); + let timedOut = false, + cancelled = false; + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGKILL"); + }, timeout); + child.once("error", reject); + child.once("close", (code, signal) => { + clearTimeout(timer); + jobs.delete(command); + if (signal && !timedOut && !cancelledIds.has(command)) { + reject(new Error(`Embedded core crashed: ${signal}`)); + return; + } + resolve({ + exitCode: code ?? 137, + stdout: Buffer.concat(output.stdout).toString("base64"), + stderr: Buffer.concat(output.stderr).toString("base64"), + timedOut, + cancelled: cancelledIds.has(command), + }); + }); + child.stdin!.on("error", () => {}); + child.stdin!.end(Buffer.from(decodeBase64(input))); + }), + ); + executionTail = pending; + return pending; + }, + async cancel(id) { + cancelledIds.add(id); + jobs.get(id)?.kill("SIGKILL"); + }, + async cleanup() { + for (const job of jobs.values()) job.kill("SIGKILL"); + }, + }; + const env = createIshWorkspace("test-1", native); + cleanups.push(() => env.cleanup(BACKGROUND_CONTEXT)); + return { env, native, directory }; +} +describe.skipIf(!core)("Pi adapter on actual embedded iSH", () => { + it("shares binary files and mutations with real guest shell", async () => { + const { env } = await fixture(); + getOrThrow(await env.createDir("notes", undefined, BACKGROUND_CONTEXT)); + getOrThrow( + await env.writeFile("notes/note.txt", "Hello Moose", BACKGROUND_CONTEXT), + ); + let output = ""; + getOrThrow( + await env.exec( + 'cat notes/note.txt; printf " updated" >> notes/note.txt', + { + onOutput: (t) => { + output += t; + }, + }, + BACKGROUND_CONTEXT, + ), + ); + expect(output).toBe("Hello Moose"); + expect( + getOrThrow(await env.readTextFile("notes/note.txt", BACKGROUND_CONTEXT)), + ).toBe("Hello Moose updated"); + getOrThrow(await env.flushFile("notes/note.txt", BACKGROUND_CONTEXT)); + const bytes = Uint8Array.from([0, 255, 128, 10, 0, 1]); + getOrThrow(await env.writeFile("binary", bytes, BACKGROUND_CONTEXT)); + expect( + getOrThrow(await env.readBinaryFile("binary", BACKGROUND_CONTEXT)), + ).toEqual(bytes); + expect( + getOrThrow(await env.listDir("notes", BACKGROUND_CONTEXT)).map( + (f) => f.name, + ), + ).toEqual(["note.txt"]); + }, 30000); + it("distinguishes timeout/abort and returns spawn errors for absent programs", async () => { + const { env } = await fixture(); + const missing = await env.exec( + ["program-not-installed"], + undefined, + BACKGROUND_CONTEXT, + ); + expect(missing.ok).toBe(false); + if (!missing.ok) expect(missing.error.code).toBe("spawn_error"); + const result = await env.exec( + "while :; do :; done", + { timeout: 0.1 }, + BACKGROUND_CONTEXT, + ); + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error.code).toBe("timeout"); + const controller = new AbortController(); + const pending = env.exec( + "sleep 30", + undefined, + withAbortSignal(controller.signal, BACKGROUND_CONTEXT), + ); + setTimeout(() => controller.abort(), 100); + const aborted = await pending; + expect(aborted.ok).toBe(false); + if (!aborted.ok) expect(aborted.error.code).toBe("aborted"); + expect( + getOrThrow(await env.exec("true", undefined, BACKGROUND_CONTEXT)) + .exitCode, + ).toBe(0); + }, 30000); + it("registers bash only for a shell-capable factory and runs write/bash/read through real Pi Harness", async () => { + const { native } = await fixture(); + const db = new DatabaseSync(":memory:"); + const storage = new SerialSqliteDatabase({ + execAsync: async (sql) => { + db.exec(sql); + }, + runAsync: async (sql, ...args) => { + db.prepare(sql).run(...args); + }, + getFirstAsync: async (sql, ...args) => + (db.prepare(sql).get(...args) as never) ?? null, + getAllAsync: async (sql, ...args) => + db.prepare(sql).all(...args) as never, + closeAsync: async () => { + db.close(); + }, + }); + const faux = fauxProvider(), + models = createModels(); + models.setProvider(faux.provider); + const call = (name: string, args: Parameters[1]) => + fauxAssistantMessage(fauxToolCall(name, args), { stopReason: "toolUse" }); + faux.setResponses([ + call("write", { path: "note", content: "Hello" }), + call("bash", { command: 'printf " guest" >> note; cat note' }), + call("read", { path: "note" }), + fauxAssistantMessage("Saved"), + ]); + const factory = Object.assign( + (id: string) => createIshWorkspace(id, native), + { shell: true }, + ); + const engine = await MobileEngine.open( + storage, + models, + { provider: "faux", modelId: "faux-1" }, + Date.now, + factory, + ); + cleanups.push(() => engine.close()); + const chat = await engine.newConversation("iSH"); + expect( + (await chat.agent(engine.context)).tools.map((t) => t.name).sort(), + ).toEqual(["bash", "edit", "read", "write"]); + const receipt = await ( + await engine.submit(chat.id, "Write and run a guest command", "ish-test") + ).wait(engine.context); + expect(receipt.status).toBe("done"); + const view = await chat.context(engine.context); + const outputs = view.entries + .flatMap((e) => e.model ?? []) + .filter((m) => m.role === "toolResult"); + expect(outputs).toHaveLength(3); + expect(outputs.every((m) => m.role === "toolResult" && !m.isError)).toBe( + true, + ); + expect(JSON.stringify(outputs)).toContain("Hello guest"); + expect((await chat.agent(engine.context)).instructions).not.toContain( + "Shell, script execution", + ); + }, 30000); + + it("runs controlled OpenAI SSE through generated Pi write/edit/bash/read and reopen", async () => { + const { native, directory } = await fixture(); + const openDatabase = async () => { + const db = new DatabaseSync(join(directory, "smoke.sqlite")); + return new SerialSqliteDatabase({ + execAsync: async (s) => { + db.exec(s); + }, + runAsync: async (s, ...p) => { + db.prepare(s).run(...p); + }, + getFirstAsync: async (s, ...p) => + (db.prepare(s).get(...p) as never) ?? null, + getAllAsync: async (s, ...p) => db.prepare(s).all(...p) as never, + closeAsync: async () => { + db.close(); + }, + }); + }; + const checks = await runFileSmoke( + openDatabase, + Object.assign((id: string) => createIshWorkspace(id, native), { + shell: true, + }), + ); + expect(checks).toContain("Moose → OpenAI SSE → Pi write/edit/bash/read"); + }, 30000); + + it("spills shell output above file-tool transfer size into a bounded guest artifact", async () => { + const { env } = await fixture(); + const result = getOrThrow( + await env.exec( + "head -c 9437184 /dev/zero", + { spill: { afterBytes: 1024, afterLines: 1000 } }, + BACKGROUND_CONTEXT, + ), + ); + expect(result.spillPath).toBeTruthy(); + let length = ""; + getOrThrow( + await env.exec( + ["/bin/sh", "-c", 'wc -c < "$1"', "size", result.spillPath!], + { + onOutput: (t) => { + length += t; + }, + }, + BACKGROUND_CONTEXT, + ), + ); + expect(Number(length.trim())).toBe(9437184); + }, 30000); + + for (const test of createEnvConformance({ + assertions: createExpectAssertions(expect), + shell: ["/bin/sh", "-c"], + withEnv: async (use) => { + const { env } = await fixture(); + await use(env); + }, + })) + it(test.name, test.run, test.timeoutMs ?? 30000); +}); diff --git a/tests/ish.test.ts b/tests/ish.test.ts new file mode 100644 index 0000000..3cd095f --- /dev/null +++ b/tests/ish.test.ts @@ -0,0 +1,108 @@ +import { expect, it } from "vitest"; +import { + BACKGROUND_CONTEXT, + withAbortSignal, +} from "@earendil-works/chord/context"; +import { + createIshWorkspace, + type IshBridge, + encodeBase64, + decodeBase64, +} from "../src/runtime/ish"; + +function bridge() { + let listener: Parameters[0] | undefined; + let cancel: (() => void) | undefined; + const calls: unknown[][] = []; + const native: IshBridge = { + prepare: async () => {}, + subscribe(fn) { + listener = fn; + return () => { + listener = undefined; + }; + }, + async execute(...args) { + calls.push(args); + listener?.({ commandId: args[1], stream: "stdout", data: "4oI=" }); + listener?.({ commandId: args[1], stream: "stderr", data: "ZXJyb3I=" }); + listener?.({ commandId: args[1], stream: "stdout", data: "rA==" }); + return { exitCode: 7, stdout: "4oKs", stderr: "ZXJyb3I=" }; + }, + cancel: async () => { + cancel?.(); + }, + cleanup: async () => {}, + }; + return { + native, + calls, + setCancel(fn: () => void) { + cancel = fn; + }, + }; +} +it("roundtrips arbitrary binary without Node globals", () => { + const bytes = Uint8Array.from({ length: 24577 }, (_, i) => i % 256); + expect(decodeBase64(encodeBase64(bytes))).toEqual(bytes); +}); +it("decodes stdout and stderr separately across chunk boundaries and preserves argv", async () => { + const { native, calls } = bridge(); + const env = createIshWorkspace("moose-chat-1", native); + const output: [string, string][] = []; + const result = await env.exec( + ["/bin/printf", "%s", "literal; $(false)"], + { + env: { PROBE: "value" }, + onOutput(text, _ctx, info) { + output.push([info.stream, text]); + }, + }, + BACKGROUND_CONTEXT, + ); + expect(result).toEqual({ ok: true, value: { exitCode: 7 } }); + expect(output).toEqual([ + ["stderr", "error"], + ["stdout", "€"], + ]); + expect(calls[0][2]).toEqual(["/bin/printf", "%s", "literal; $(false)"]); + expect(calls[0][4]).toMatchObject({ PROBE: "value" }); + await env.cleanup(BACKGROUND_CONTEXT); +}); +it("cancels a running native command and waits for its cleanup", async () => { + const fixture = bridge(); + let entered!: () => void; + const started = new Promise((r) => { + entered = r; + }); + fixture.native.execute = async () => { + entered(); + await new Promise((r) => fixture.setCancel(r)); + return { exitCode: 137, stdout: "", stderr: "", cancelled: true }; + }; + const env = createIshWorkspace("moose-chat-2", fixture.native); + const controller = new AbortController(); + const pending = env.exec( + "sleep 30", + undefined, + withAbortSignal(controller.signal, BACKGROUND_CONTEXT), + ); + await started; + controller.abort(); + const result = await pending; + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error.code).toBe("aborted"); + await env.cleanup(BACKGROUND_CONTEXT); +}); +it("preserves actionable native filesystem failures without starting a command", async () => { + const { native, calls } = bridge(); + native.prepare = async () => { + throw new Error("Native command journal capacity reached"); + }; + const env = createIshWorkspace("moose-chat-error", native); + const result = await env.readTextFile("note", BACKGROUND_CONTEXT); + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error.message).toContain("journal capacity"); + expect(calls).toHaveLength(0); + await env.cleanup(BACKGROUND_CONTEXT); +}); From 861fdeff5d9c9813d1123887a9429312641cf87a Mon Sep 17 00:00:00 2001 From: sugarforever Date: Fri, 23 Oct 2026 01:32:42 +0100 Subject: [PATCH 4/4] docs: record fresh iOS simulator iSH validation --- docs/ios-ish-setup-guide.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/ios-ish-setup-guide.md b/docs/ios-ish-setup-guide.md index 296a40a..591ef92 100644 --- a/docs/ios-ish-setup-guide.md +++ b/docs/ios-ish-setup-guide.md @@ -140,3 +140,19 @@ Record host tests, Hermes exports, simulator operations, physical-device operati ## Diagnostics There is currently no iOS equivalent of Termux's **Run self-check (no model calls)** or **View diagnostic records** settings buttons. Use the controlled developer smoke and native verification commands above. Preserve displayed errors and tool results; do not include API keys, OAuth tokens or private file contents when sharing a report. + +## Simulator validation, 2026-10-23 + +Rechecked code commit `07411e8` (Moose v0.7.0) on an Apple Silicon Mac with Xcode 26.2 (17C52), Node.js 26.8.1 and an iPhone 17 Pro simulator running iOS 26.2. The Mac already had the documented tools installed; this was not a clean-Mac prerequisite installation test. + +| Layer | Result from this run | +| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Dependency preparation | `npm run prepare:ish` completed using its default `/tmp/moose-ish-embedded` cache, producing device/simulator libraries, the XCFramework and rootfs. | +| Installed resources | The installed app's native source manifest matched the current core, policy patch and bridge sources; the guest rootfs and BusyBox were present. | +| Native Moose client | Rebuilt and installed with iSH enabled: build succeeded with zero errors and 11 warnings. The controlled Metro session used port 8097. | +| Host and Pi adapter verification | `npm run test:ish` passed the real host core and production bridge suites, plus all 33 adapter/boundary tests. These are host results, not simulator test counts. | +| UIKit core | Real same-kernel suite passed with roots, argv/env/streams, supervisor survival, descendant cancellation, timeouts and resource checks; the UIKit host survived. | +| UIKit production bridge | Passed binary stdin/output, 8 MiB transfer, event ordering, cancellation/timeout/flood bounds, command IDs, admission limits and cross-generation journal/invalidation. | +| React Native/Hermes app | With both iSH and file-smoke flags enabled, the app displayed `Moose filesystem smoke passed` after controlled SSE drove Pi write/edit/bash/read. Tool results reached the next controlled request; transcript persistence, checkpoint isolation, file/SQLite reopen and traversal rejection passed. | + +This validates the prepared native backend and its Pi integration in the simulator's controlled developer entry. It does not validate a real model/provider account, the ordinary chat UI's complete model-configuration flow, physical-iPhone signing/installation, or locked-screen/background execution. No real model credentials or billable requests were used by the controlled tests.