diff --git a/docs/macos-owner-contract.md b/docs/macos-owner-contract.md new file mode 100644 index 0000000..01fb191 --- /dev/null +++ b/docs/macos-owner-contract.md @@ -0,0 +1,281 @@ +# macOS lock owner and visibility contract (decision record) + +Status: **proposed, awaiting a user decision.** This record belongs to +[#127](https://github.com/abruption/codex-unlock/issues/127). It changes no +runtime behavior, platform support, JSON output, or signal path. It builds on +[macos-lock-attribution.md](macos-lock-attribution.md) (the disclosed +opener-correlation proposal and its evidence inventory), +[research-lock-attribution.md](https://github.com/abruption/codex-unlock/blob/3b89057c37e6a7028f88c7bef39c6eae9604c639/docs/research-lock-attribution.md) +(the #100 foreign-UID measurements), and the `lsof +fg` refusal-filter +research in draft [#139](https://github.com/abruption/codex-unlock/pull/139) +at [`56dc50d5`](https://github.com/abruption/codex-unlock/tree/56dc50d5f610d3356228aacee743cbd22a6706eb/experimental/126-guard-liveness) +(section M2). Facts measured there are summarized, not repeated. + +Evidence levels used below: + +- **Source-pinned**: read at a fixed revision. It explains behavior; it is not + a runtime measurement and a targeted read is not an exhaustive audit. +- **Measured**: observed with same-user synthetic fixtures on the stated + system only. Nothing here was measured with a real Codex process. +- **Not covered**: neither of the above. Treat as unverified. + +## 1. Fixed-source facts + +### Codex writer lock lifetimes (source-pinned) + +Codex `rust-v0.162.1` is commit +[`092d3acd`](https://github.com/openai/codex/blob/092d3acd6bec3e3a14bdc7e7a2810ab628ab759d/codex-rs/rollout/src/writer_lock.rs), +built with Rust 1.95.0 +([`rust-toolchain.toml`](https://github.com/openai/codex/blob/092d3acd6bec3e3a14bdc7e7a2810ab628ab759d/codex-rs/rust-toolchain.toml)). + +| Path | Handle lifetime at that revision | +| --- | --- | +| [`acquire`](https://github.com/openai/codex/blob/092d3acd6bec3e3a14bdc7e7a2810ab628ab759d/codex-rs/rollout/src/writer_lock.rs#L43-L86) | Blocking coordination lock, then opens the thread file and `try_lock`s it. On `WouldBlock` or error the file local is dropped on return, before the earlier-declared coordination local, so a failed acquirer's thread FD is closed while coordination is still held. On success the file moves into the guard. | +| [`try_acquire_for_publication`](https://github.com/openai/codex/blob/092d3acd6bec3e3a14bdc7e7a2810ab628ab759d/codex-rs/rollout/src/writer_lock.rs#L91-L106) | Under coordination, opens and `try_lock`s the thread file. Both a successful (transient holder) and a failed (transient nonholder) probe drop the file at the end of the match arm; coordination is returned to the caller. | +| [`remove_stale_thread_locks`](https://github.com/openai/codex/blob/092d3acd6bec3e3a14bdc7e7a2810ab628ab759d/codex-rs/rollout/src/writer_lock.rs#L124-L171) | Under coordination: a transient holder for each free lock file, dropped before the pathname is removed; a busy one is closed without removal. | +| [`Drop for WriterLockGuard`](https://github.com/openai/codex/blob/092d3acd6bec3e3a14bdc7e7a2810ab628ab759d/codex-rs/rollout/src/writer_lock.rs#L174-L196) | Takes coordination (blocking), closes the file, removes the pathname, releases coordination. If taking coordination fails, it returns early: the pathname is not removed, and the `file` field is still closed afterwards by ordinary field drop, without coordination. | + +Rust 1.95.0 std ([`59807616`](https://github.com/rust-lang/rust/blob/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/sys/fs/unix.rs)) +opens every `File` with +[`O_CLOEXEC`](https://github.com/rust-lang/rust/blob/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/sys/fs/unix.rs#L1349-L1359); +on Apple targets +[`lock`](https://github.com/rust-lang/rust/blob/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/sys/fs/unix.rs#L1434-L1449) +is `flock(LOCK_EX)` and +[`try_lock`](https://github.com/rust-lang/rust/blob/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/sys/fs/unix.rs#L1520-L1543) +is `flock(LOCK_EX | LOCK_NB)`. + +What this source reading **does not** exclude: + +- Another code path in the same process holding a duplicate of the guard's + description (`try_clone`, `dup`, raw-FD use). No such use of this type was + checked beyond the file above. +- A `fork` without exec anywhere in the process or its dependencies; a forked + child shares the description, its lock, and its flags. `O_CLOEXEC` only + closes the FD at a successful exec, so a fork-then-exec launcher also has a + short shared window. Whether Codex's spawn paths use `posix_spawn` or + fork-then-exec was not audited. +- Descriptor passing (`SCM_RIGHTS`, Mach fileports) to another process. +- Packaging, wrappers, or a different build of Codex than the pinned source. + +These are listed because a targeted search found no such path, and a targeted +search is not a proof of "no call" or "no inheritance". The 0.162.1 +`WriterLockGuard::drop` and `O_CLOEXEC` statements above are source-only. + +Observation interleavings: the guarded probe takes the same coordination lock, +so it never observes the transient holders and nonholders above. `lsof` opener +discovery runs outside the synchronous guard (it must, to keep the guard +invariant), so it can sample those transient openers from other Codex +processes. Each sample being individually valid does not make the combination +an atomic holder observation. #139 M2 Part 2 shows that a wrong eligible +target needs the transient descriptor open in both the opener-scan and the +lock-set-scan sample windows; no real-environment frequency was measured. + +### XNU flock ownership and `FWASLOCKED` (source-pinned) + +The pinned public source read here is +[`ac9718fb` (xnu-12377.121.6)](https://github.com/apple-oss-distributions/xnu/tree/ac9718fb1af618d5ce8678d0dc6e8a58f252216f). +It is not claimed to be the newest public release. The measured kernel +(`xnu-13432.1.9`) is a later version whose source was not found in this +search, so these statements are about the pinned source, not the running +kernel. + +- flock locks are created with + [`lf_owner = NULL`](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/kern/kern_lockf.c#L313-L317) + (only `F_POSIX` locks, and confined OFD locks, record an owning process), and + [`F_GETLK` reports `l_pid = -1`](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/kern/kern_lockf.c#L1172-L1180) + when the owner is NULL. By construction the kernel lock record of a flock + names an open file description, not a process. +- `FWASLOCKED` (`0x4000`, ["has or has had an advisory fcntl lock"](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/sys/fcntl.h#L136)) + is set on the description after a successful + [`flock`](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/kern/kern_descrip.c#L6034-L6039) + and after `open` with + [`O_EXLOCK`/`O_SHLOCK`](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/vfs/vfs_syscalls.c#L5028-L5032). + `LOCK_UN` does not clear it. It is consulted at + [close](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/vfs/vfs_vnops.c#L1886-L1899) + to decide whether to issue `F_UNLCK`. +- [`fill_fileinfo`](https://github.com/apple-oss-distributions/xnu/blob/ac9718fb1af618d5ce8678d0dc6e8a58f252216f/bsd/kern/proc_info.c#L2651-L2653) + copies the description's `fg_flag` into libproc's `fi_openflags`; this is + where `lsof +fg` obtains the `G` field. It is a per-description flag word, + not a per-process lock record. + +"Never cleared" is from a targeted read of `kern_descrip.c`, `kern_lockf.c`, +`vfs_syscalls.c`, `vfs_vnops.c`, `proc_info.c`, `uipc_usrreq.c`, and +`sys/fcntl.h`, not a whole-tree audit. The sticky behavior is **measured** on +the running kernel (#139 M2 and the 2026-10-13 inventory), which is the +evidence actually relied on. + +No public macOS API found in this research maps a flock to its holding +process. The tested candidates (`F_GETLK`, `F_GETLKPID`, `F_OFD_GETLK`, +`lsof -F l`, `lsof +fg`, `proc_listpidspath`, `fuser`, `NOTE_FUNLOCK`) are +recorded in [macos-lock-attribution.md](macos-lock-attribution.md#evidence-inventory). +DTrace and Endpoint Security were not exercised. This is a bounded finding, +not a proof that no such primitive exists. + +## 2. Counterexamples to opener-based attribution + +All rows are measured with synthetic same-user fixtures unless marked. + +| Counterexample | What the observer sees | Source | +| --- | --- | --- | +| Hidden holder + visible nonlocking opener | Held probe, one clean visible opener that is not the holder | Foreign-UID CI run (#100); same-user modelled omission (#139 M2) | +| Failed acquirer retaining its FD | Visible opener without `FWASLOCKED` | #139 M2 matrix | +| Never-locked opener | Visible opener without `FWASLOCKED` | #139 M2 matrix | +| Former holder (`LOCK_UN`, FD kept) + actual holder | Both descriptions carry `FWASLOCKED`; indistinguishable | #139 M2 matrix | +| Former holder alone, lock free | `FWASLOCKED` still set | #139 M2 matrix | +| Shared description via fork without exec | Both processes carry the bit; one lock | This record, section 5 | +| Shared description via exec of an inheritable FD | Both carry the bit; lock survives the original's close | This record, section 5 | +| Description passed with `SCM_RIGHTS` to a sibling process (not the locker's descendant) | Receiver never called `flock`, carries the bit, and becomes the sole holder | This record, section 5 | + +Consequence: the bit can exclude a descriptor that never held a lock. It can +never identify the current holder, and terminating one process that shares a +description does not prove the lock will be released. + +## 3. Options + +| Option | What it does | Removes | Does not remove | Cost and risk | +| --- | --- | --- | --- | --- | +| **(a)** Keep current correlation with documented limits | No runtime change; [macos-lock-attribution.md](macos-lock-attribution.md) states that `live_owner` is opener correlation, not holder proof | Nothing new | Hidden holder + eligible visible nonholder (failed acquirer, never-locked opener, former holder) | No new code. The residual wrong-target risk is accepted only as disclosed, and that acceptance is a user decision | +| **(b)** Add a refusal-only `lsof +fg` filter | Before eligibility, require the candidate to have at least one description on the lock with `FWASLOCKED`. Decision values only `unknown` / `reject` / `not_excluded`; `unknown` and `reject` refuse; `not_excluded` only means "not refused by this filter" | Failed-acquirer and never-locked wrong targets, including the #100 shape (#139 M2 Part 2) | Former holder retaining an FD; shared descriptions; clean whole-record omission | An extra `lsof` invocation per candidate at the existing fixed path (no new native dependency). Relies on an undocumented libproc flag word; parser and format drift must fail as `unknown`. It is never holder evidence and must not be reported as such | +| **(c)** Restrict macOS to diagnostics | `list`, `inspect` and `check-update` keep working; `unlock` refuses on macOS | Every macOS wrong-target path | Nothing on Linux (separate fdinfo track) | Removes a supported recovery feature and changes the platform-support contract and JSON-visible behavior | + +Option (b) adds only refusals, but it still changes which owners `unlock` will +signal, adds an observable refusal reason, and must be specified for JSON v1 +compatibility. Option (c) removes macOS unlock support. **Both options change +the supported-platform or permission contract and require an explicit user +decision; neither is implemented here.** No option introduces a new success +or authorization claim: no available macOS evidence supports one. + +### Recommendation + +1. Keep (a)'s current behavior unchanged now, with the limits in + [macos-lock-attribution.md](macos-lock-attribution.md) and this record. + Keeping the behavior is not approval of its residual wrong-target risk. + Accepting that risk is a separate user decision that this record does not + make. +2. Propose (b) as the next separately reviewed change. It is refusal-only. + In #139 M2 it rejects the failed-acquirer and never-locked-opener wrong + targets, including the #100 failed-acquirer shape. It does **not** remove + the former holder that keeps its FD, which #139 M2 also reproduced as a + wrong eligible target (`B_former`). That residual must be documented with + (b), and the [gap plan](#4-gaps-and-fixture-plan) rows marked + "before (b)" must be measured first. +3. Choose (c) instead only if the user judges the residual former-holder and + shared-description risk unacceptable even after (b). That trade-off is a + policy decision, not one research can settle. +4. A stronger, holder-positive mode remains gated on a current-holder + primitive or an authenticated cooperative owner release, per the + acceptance gates in [macos-lock-attribution.md](macos-lock-attribution.md#acceptance-gates-for-a-stronger-contract) + and the [upstream handoff proposal](upstream-handoff-proposal.md). + +Under every option the invariants in `AGENTS.md` are unchanged: independent +nonblocking guarded probe, lock order lease → native guard → thread probe, no +native file creation or deletion, no force mode, no `SIGKILL`, and a signal +only to the exact revalidated same-user owner with a stable transcript ending +in `task_complete`. + +### Transfer to the Rust experiment (#122) + +Rust's `File::lock`/`try_lock` on macOS is the same `flock`, so a Rust +implementation inherits exactly these limits. A Rust port could read +`proc_pidfdinfo` directly instead of parsing `lsof`, which removes the text +format but not the semantics: the flag is still per description and sticky, +and same-user libproc visibility still does not reveal hidden holders. Rust +does not resolve holder attribution. Any chosen option transfers as the same +decision table and the same `unknown`/`reject`/`not_excluded` vocabulary; the +#122 lab must keep `safeToUnlock` false for anything it cannot attribute. + +## 4. Gaps and fixture plan + +"Measured" means same-user synthetic fixtures on APFS, macOS 27.0.1. Each plan +uses only processes the fixture starts, bounded waits, stdin-EOF children with +their own lifetime, zero signals, and asserts zero SIGTERM for refused cases. + +| Area | Item | Status | Narrow fixture or test plan | Needed before | +| --- | --- | --- | --- | --- | +| Inherited FD | fork without exec, both alive | Measured (section 5) | — | — | +| Inherited FD | exec of inheritable FD; `O_CLOEXEC` across exec | Measured with Python's launcher (section 5) | Repeat with a Rust `std::process::Command` child to cover the launcher Codex uses | (b) | +| Inherited FD | `SCM_RIGHTS` passing to a sibling process | Measured (section 5) | — | — | +| Inherited FD | Mach fileport passing | Not covered | C fixture: `fileport_makeport` → `fileport_makefd` in a separately started same-user actor; record bit and probe | (b) | +| Inherited FD | Shared description with a visible Codex-looking sharer and a second sharer hidden | Not covered | Combine `scm_rights` with the M2 modelled omission; product must not report `safeToUnlock: true` with only one sharer killed. Expected today: possible wrong target (document, do not "fix" in the test) | (b) | +| Inherited FD | Real Codex inheritance / fork paths | Not covered (source-only) | Out of scope without a real Codex binary; requires a separately approved, isolated, synthetic-home run | Holder-positive mode | +| Process identity | Holder execs in place (same PID, new image) while keeping the FD | Not covered on macOS | Actor `execve`s itself with an inheritable lock FD between two observations; assert refusal via changed command/arguments | (b) | +| Process identity | PID reuse, start-time resolution of `ps lstart` | Covered by deterministic revalidation tests ([safety-race-matrix.md](safety-race-matrix.md)); real PID reuse not forced | Unit test with an injected process table where PID and second-resolution start time match but PPID/arguments differ | — | +| Process identity | Credential change of a holder after locking (setuid helper) | Not covered. The existing `--privileged-fixtures` CI job (`scripts/research-lock-attribution.py`) measures a hidden holder that runs under a different UID from the start (#100); it does not change a holder's credentials after it locks | A post-lock credential-change fixture needs a privileged helper; it could be added to that existing job | (c) decision input | +| Permission | Same-user process whose libproc FD info is denied (sandboxed or hardened target) | Not covered | Holder launched under `sandbox-exec` with a deny-default profile; record lsof status, stderr, and whether the record is omitted cleanly | (b) | +| Permission | codex-unlock itself running sandboxed (e.g. inside a Codex shell tool) | Not covered | Run the fixtures' observer under `sandbox-exec`; any reduced visibility must surface as `unknown`, not as a clean scan | (b) | +| Permission | `lsof +fg` field drift (Homebrew lsof, future versions) | Only lsof 4.91 at `/usr/sbin/lsof` measured | Golden-output parser tests from multiple lsof versions; unknown fields → `unknown` (the #139 parser does this) | (b) | +| Filesystem | APFS | Measured | — | — | +| Filesystem | HFS+, ExFAT, MS-DOS local volumes | Not covered | `hdiutil create -fs ` image in the scratch root, attached with `-nobrowse -mountpoint` under the scratch; needs approval because it changes the mount table | (b) | +| Filesystem | SMB/NFS/AFP and FUSE (macFUSE, rclone) | Not covered | Network/FUSE flock may be local-only or emulated. Plan: report these as unsupported for unlock unless measured; detect by `statfs` type | (c) decision input | +| OS version | macOS 15 CI (kernel 24.x) sticky-bit behavior | Not covered (only foreign-UID visibility measured there) | Run `experimental/127-macos-contract/inherit.py` and #139 M2 in a research-only CI job on `macos-15` and `macos-15-intel` | (b) | + +## 5. Synthetic validation (#127 fixtures) + +Script: [`experimental/127-macos-contract/inherit.py`](../experimental/127-macos-contract/README.md). +Environment: Darwin 27.0.0 arm64 (macOS 27.0.1, `xnu-13432.1.9`), Python +3.15.0, lsof 4.91 (`/usr/sbin/lsof`), APFS, same user, worktree base +`3bc40919`, 2026-10-16. Raw results are private (they contain local paths and +PIDs). + +Three script revisions were used in runs 1–6. Results of one revision do not +validate another. Later integrating-session corrections require a separate +run and are not validated by runs 1–6. + +- **Original revision** (runs 1–4). Its `ok` checked only failures and + incomplete waits, not the expected table. Its raw results do not record a + script SHA-256, so the exact script is *not recorded* there. It was the + script in the frozen READY-B input. The table below was checked against + each raw result's per-stage `probe`, `roles`, `scan_valid` and + `unexpected_pids` fields after the fact. The original revision used + `subprocess.run` timeouts for `mount` and `lsof -v`. Those could send + SIGKILL if they timed out. No raw result shows a timeout, because one would + have aborted the run before a result was written. Its `signals_sent: 0` + field was a constant, not a measurement. +- **Expected-table revisions** (runs 5 and 6, distinct scripts). Each records its own `script_sha256`, + evaluates an expected table in code (`collected`, `measured`, `ok`), parses + `lsof` strictly, and makes no kill calls or killing timeouts. + +| Run | Script | Outcome | Raw result SHA-256 | +| --- | --- | --- | --- | +| 1 | original (SHA not recorded) | `ok`; all stages match the table below | `8c2460c52c9b302e0106512dc0cc3f9f71d79ef9652b3877cb9af1875ebb1d71` | +| 2 | original (SHA not recorded) | same | `9d5fed5f5eebd86ab16ec6f7512a6df23f0e7578fd41168993ec5dc8601a9e18` | +| 3 | original (SHA not recorded) | same | `925afe905d4e75f3ee2c70e49df8c635bba15e3921c7eb56be9977dc973fb341` | +| 4 | original (SHA not recorded); independent rerun by the integrating session | same; every actor exit 0 | `f4911363a73dc5ad06297ce37b1fc1d5258b8623acf03d6167318f54f4b67d2f` | +| 5 | `743269e2d8efb821f650cbe22c5fd598747e5e9fb56866cb23a7d1582fb7573c` | exit 1: `collected` true, `measured` false. Five no-opener stages were `scan_invalid`, caused by a parser bug in that revision (empty `lsof` output counted as one unterminated record). Probes and roles matched the table | `b2c44156447d9813e2e2ce0350fa7b39cc0ef60cc8e71e672375a4384aba2457` | +| 6 | `3a24810af3090b7cd60f24063134ba1f19f5ca9118a8536feeed43aeed070fb1` (bug fixed, test added) | exit 0: `ok`, `collected`, `measured`; no mismatches, no pending diagnostics | `55172c87451b32427dbc85d33ae28f7531b8c6898dc3b53671f5f771b8fb5f49` | +| 7 | `44215d458e53b7ddd038db32a65746b2fdf1402258563cac4f48f682dee51bac` (integrating-session corrections) | exit 0: `ok`, `collected`, `measured`; no failures, mismatches, incomplete waits or pending diagnostics; start/end script hashes match | `097a33f60b49e7461983d7d521286c4fa666dad83b09900413c42c847559ffdd` | + +Run 5's fixture directory is kept, as designed for a failed run. +Run 7 separately validates the line-cap/partial-EOF, cleanup and script-identity +corrections. The focused helper suite passed 23 tests with no skipped tests +and ResourceWarnings treated as errors. This remains one machine's synthetic +APFS measurement, not a native Codex or production-holder attribution result. + +| Scenario / stage | Probe | Bit per process | +| --- | --- | --- | +| `fork_shared` both alive | held | parent bit, forked child bit | +| `fork_shared` parent closed its FD (alive) | held | parent absent, child bit | +| `exec_inherit` both alive | held | parent bit, exec'd child bit | +| `exec_inherit` parent closed its FD (alive) | held | parent absent, child bit | +| `exec_cloexec` both alive | held | parent bit, child absent (child confirmed the FD number did not name the fixture) | +| `exec_cloexec` parent closed its FD | **free** | both absent | +| `scm_rights` both hold the description | held | sender bit, receiver bit | +| `scm_rights` sender closed its FD / exited | held | sender absent, receiver (never called `flock`) bit | +| every scenario, all closed | free | no openers | + +Interpretation, within that scope: + +- One kernel lock can be held simultaneously by several processes that share + a description, and lsof shows the same bit on each. A unique-opener rule + refuses the "both alive" stages only because two openers are visible. +- After the original locker closes, the remaining sharer is the real holder + even though it never called `flock`. With `SCM_RIGHTS` it is a sibling + started by the same fixture parent, not the locker's descendant. Unrelated + lineage in general was not tested. These results show that one description + can be shared by several processes. They do not show that real Codex takes + any such path. Process lineage is not a holder criterion. +- The `exec_cloexec` result shows the OS mechanism the Rust std `O_CLOEXEC` + argument depends on, for Python's launcher. It does not show that Codex + never inherits or passes its lock FD. +- Not done here: other filesystems, other macOS versions, fileports, sandboxed + observers, and any real Codex binary. diff --git a/experimental/127-macos-contract/README.md b/experimental/127-macos-contract/README.md new file mode 100644 index 0000000..e69fd0d --- /dev/null +++ b/experimental/127-macos-contract/README.md @@ -0,0 +1,80 @@ +# #127 synthetic description-sharing fixtures (macOS) + +Unpublished research for [#127](https://github.com/abruption/codex-unlock/issues/127). +It is outside the npm `files` list and is not run by `npm test`. Results and +their interpretation are recorded in +[docs/macos-owner-contract.md](../../docs/macos-owner-contract.md#5-synthetic-validation-127-fixtures). + +`inherit.py` starts only its own same-user Python actors on a fresh fixture file +and never touches a Codex home or process. It measures, per stage, an +independent nonblocking flock probe and the `lsof +fg` `FWASLOCKED` bit for: + +| Scenario | Shape | +| --- | --- | +| `fork_shared` | Holder forks without exec; both processes share one description | +| `exec_inherit` | Holder marks its FD inheritable and execs a child that keeps it | +| `exec_cloexec` | Holder keeps `O_CLOEXEC` and execs a child with `close_fds=False` | +| `scm_rights` | Holder passes its FD over a socketpair to a sibling actor (started by the script, not the holder's descendant), then closes it | + +Results are reported in three parts: + +- `collected`: no actor failure, no incomplete wait and no exception. +- `measured`: every stage matches the expected table in `EXPECTED_STAGES`. + The table covers probe, per-role bit, `scan_valid`, zero unexpected PIDs, + ready, closed and keeper events, received FD count, and child exits. +- `ok`: both. + +Any mismatch, invalid scan or incomplete wait exits nonzero and keeps the +fixture directory. + +Framing and parsing: actor output is read from the raw FD with newline +framing, a fixed deadline, and per-line and per-stream byte caps. A partial +line is never returned as an event. A malformed event or an unexpected event +name (for example a keeper or receiver `failed` event) is recorded as a +failure, never wrapped as ready. `lsof -F0pfG` is parsed strictly. A +missing `G`, a field-order change, non-hex flags or an unterminated record +makes the scan invalid. It is never read as `no_bit`. + +Lifecycle: every actor is registered when it starts. Scenario exceptions enter +cleanup, which attempts stdin EOF, a bounded output read and a bounded exit +wait for each registered actor before writing the result. These attempts do +not guarantee cleanup or result writing after arbitrary I/O failures. Actors end themselves with +exit 3 after 20 s. **The script makes no kill calls and uses no killing +subprocess timeouts.** A diagnostic (`lsof`, `mount`) that exceeds its bound +is recorded as incomplete with its PID. It is polled again at the end +(`pending_diagnostics`) and never stopped. Its pipes stay open until the +script exits, after which the kernel may deliver SIGPIPE on its next write. +Nothing here guarantees that a child actually ended. + +The result file is created with `O_CREAT|O_EXCL|O_NOFOLLOW`, mode 0600, before +the fixture directory exists, so an existing path or a symlink is refused and +cleanup never removes it. It records `script_sha256` and compares start/end +script hashes (`script_identity`); a change makes the measurement fail. +The hostname is not +collected. The filesystem type is taken from `mount` for the canonical path. + +```sh +root="$(mktemp -d "/codex-127-XXXXXX")" && chmod 700 "$root" +python3 -I experimental/127-macos-contract/inherit.py --scratch-root "$root" --result "/result.json" +RESEARCH_TMP_ROOT="" python3 -I experimental/127-macos-contract/test_inherit.py +``` + +`test_inherit.py` checks the following deterministically with finite +synthetic children only: + +- framing: two lines in one chunk, a partial line at a deadline and at EOF, + and line and stream caps; +- event validation; +- strict `lsof` parsing, including empty output and eleven malformed shapes; +- expected-table counterexamples: a wrong bit, an invalid scan, an unexpected + PID, a probe error, a wrong or missing keeper, a wrong received count, a + missing closed event, a nonzero child exit, a missing scenario, the stage + order and the same PID; +- a diagnostic timeout recorded as pending and later observed as exited; +- an output cap, early actor exit, a `failed` instead of `ready` event, a + partial line followed by silence, and closing every registered actor; +- result-file refusal of an existing path and a symlink. + +The JSON result and console output can contain local paths and PIDs; keep +both private. Run against a frozen source file; a measurement does not certify +another script revision. diff --git a/experimental/127-macos-contract/inherit.py b/experimental/127-macos-contract/inherit.py new file mode 100644 index 0000000..850fdc4 --- /dev/null +++ b/experimental/127-macos-contract/inherit.py @@ -0,0 +1,829 @@ +#!/usr/bin/env python3 +"""#127 research: macOS open-file-description sharing and FWASLOCKED. + +Same-user synthetic fixtures only. Never targets a Codex process or home. +Every actor is started by this script and registered before use. Actors close +their descriptors on stdin EOF (the forked child on its control pipe's EOF) +and end themselves with exit 3 after ACTOR_LIFETIME seconds. + +Signals: this script never calls kill or a subprocess timeout that kills. +Diagnostics (lsof, mount) are waited for with a bound; one that does not +finish is recorded as incomplete with its PID and polled again at the end. +It is not stopped. Its pipes stay open until this script exits, after which +the kernel may deliver SIGPIPE to it on its next write. Nothing here +guarantees that any child actually ended. + +Scenarios (each uses a fresh fixture file): + fork_shared actor flocks, forks WITHOUT exec; both share one description + exec_inherit actor flocks an inheritable FD, execs a child that keeps it + exec_cloexec actor flocks an O_CLOEXEC FD, execs a child with close_fds off + scm_rights actor A flocks and passes the FD over a socketpair to sibling + actor B (started by this script, not A's descendant), then A + closes its FD + +Per stage: an independent nonblocking flock probe (held/free) and +`lsof +fg -F0pfG` per-PID FWASLOCKED bits. The bit is recorded only; it is +never treated as holder evidence. Collection (`collected`) and the expected +table (`measured`) are reported separately; `ok` needs both. +""" + +import argparse +import array +import errno +import fcntl +import hashlib +import json +import os +import platform +import re +import select +import socket +import subprocess +import sys +import tempfile +import time +from pathlib import Path + + +FWASLOCKED = 0x4000 +ACTOR_LIFETIME = 20.0 +READY_TIMEOUT = 5.0 +EXIT_TIMEOUT = 5.0 +DIAGNOSTIC_TIMEOUT = 5.0 +LINE_LIMIT = 16 << 10 +STREAM_LIMIT = 256 << 10 +LSOF = "/usr/sbin/lsof" +MOUNT = "/sbin/mount" +SCRIPT = str(Path(__file__).resolve()) + + +# ------------------------------------------------------------------ framing + +class Framed: + """Newline framing over a raw fd with a fixed deadline and byte caps. + + read_line returns ("line", bytes), ("eof", None), ("timeout", None) or + ("limit", None) or ("unterminated", None). Bytes after a returned line + stay buffered for the next call; a partial line at a deadline is kept, + never returned. EOF with a partial line is a framing failure. + """ + + def __init__(self, fd, line_limit=LINE_LIMIT, stream_limit=STREAM_LIMIT): + self.fd = fd + self.buffer = b"" + self.total = 0 + self.line_limit = line_limit + self.stream_limit = stream_limit + self.eof = False + self.overflow = False + + def read_line(self, deadline): + while True: + if self.overflow: + return ("limit", None) + if b"\n" in self.buffer: + if self.buffer.index(b"\n") > self.line_limit: + self.overflow = True + return ("limit", None) + line, self.buffer = self.buffer.split(b"\n", 1) + return ("line", line) + if len(self.buffer) > self.line_limit: + self.overflow = True + return ("limit", None) + if self.eof: + return ("unterminated" if self.buffer else "eof", None) + remaining = deadline - time.monotonic() + if remaining <= 0: + return ("timeout", None) + ready, _, _ = select.select([self.fd], [], [], remaining) + if not ready: + continue + chunk = os.read(self.fd, 4096) + if not chunk: + self.eof = True + continue + # Bytes past the stream cap are dropped, never buffered. + room = self.stream_limit - self.total + if len(chunk) > room: + self.overflow = True + chunk = chunk[:max(0, room)] + self.total += len(chunk) + self.buffer += chunk + + +def parse_event(line): + """A JSON object with a string `event`, or None.""" + try: + value = json.loads(line.decode("utf-8")) + except (UnicodeDecodeError, ValueError): + return None + if not isinstance(value, dict) or not isinstance(value.get("event"), str): + return None + return value + + +# ---------------------------------------------------------------- actor side + +def emit(value): + data = (json.dumps(value, sort_keys=True) + "\n").encode() + while data: + data = data[os.write(1, data):] + + +def wait_readable(fd, deadline): + """True when fd is readable (data or EOF), False at deadline.""" + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + ready, _, _ = select.select([fd], [], [], remaining) + if ready: + return True + + +def run_commands(deadline, handlers): + """Serve stdin commands until EOF (True) or lifetime/limit (False).""" + framed = Framed(0) + while True: + status, line = framed.read_line(deadline) + if status == "eof": + return True + if status != "line": + return False + handler = handlers.get(line.decode("utf-8", "replace").strip()) + emit(handler() if handler else {"event": "error", "error": "unknown command"}) + + +def lock_exclusive(fd): + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + + +def bounded_waitpid(pid, deadline): + while time.monotonic() < deadline: + done, status = os.waitpid(pid, os.WNOHANG) + if done: + return os.waitstatus_to_exitcode(status) + time.sleep(0.01) + return None + + +def actor_fork_shared(lock_path, deadline): + fd = os.open(lock_path, os.O_RDWR | os.O_CLOEXEC | os.O_NOFOLLOW) + lock_exclusive(fd) + control_r, control_w = os.pipe() + child = os.fork() + if child == 0: + # Forked child, no exec: shares the parent's description of fd. + os.close(control_w) + code = 0 if wait_readable(control_r, deadline) else 3 + os.close(fd) + os._exit(code) + os.close(control_r) + state = {"fd": fd} + + def close_parent_fd(): + os.close(state["fd"]) + state["fd"] = None + return {"event": "closed", "who": "parent"} + + try: + emit({"event": "ready", "pid": os.getpid(), "child_pid": child}) + clean = run_commands(deadline, {"close_parent_fd": close_parent_fd}) + finally: + os.close(control_w) + if state["fd"] is not None: + os.close(state["fd"]) + child_exit = bounded_waitpid(child, time.monotonic() + EXIT_TIMEOUT) + emit({"event": "child_exit", "code": child_exit}) + return 0 if clean and child_exit == 0 else 3 + + +def actor_exec(lock_path, deadline, inheritable): + fd = os.open(lock_path, os.O_RDWR | os.O_CLOEXEC | os.O_NOFOLLOW) + lock_exclusive(fd) + if inheritable: + os.set_inheritable(fd, True) + # close_fds=False isolates the O_CLOEXEC flag: nothing else closes fd. + child = subprocess.Popen( + [sys.executable, "-I", SCRIPT, "actor", "keeper", lock_path, str(fd)], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=False, bufsize=0) + state = {"fd": fd} + + def close_parent_fd(): + os.close(state["fd"]) + state["fd"] = None + return {"event": "closed", "who": "parent"} + + clean = False + try: + status, line = Framed(child.stdout.fileno()).read_line(time.monotonic() + READY_TIMEOUT) + keeper = parse_event(line) if status == "line" else None + if not keeper or keeper.get("event") != "keeper_ready" or not isinstance(keeper.get("has_fd"), bool): + # A keeper that is not ready is a failure, never a ready actor. + emit({"event": "failed", "reason": f"keeper_not_ready:{status}", "child_pid": child.pid}) + else: + emit({"event": "ready", "pid": os.getpid(), "child_pid": child.pid, "keeper": keeper}) + clean = run_commands(deadline, {"close_parent_fd": close_parent_fd}) + finally: + child.stdin.close() + if state["fd"] is not None: + os.close(state["fd"]) + child_exit = bounded_waitpid(child.pid, time.monotonic() + EXIT_TIMEOUT) + child.returncode = child_exit + if child_exit is not None: + child.stdout.close() + emit({"event": "child_exit", "code": child_exit}) + return 0 if clean and child_exit == 0 else 3 + + +def actor_keeper(lock_path, fd_text, deadline): + """Exec'd child: report whether the inherited fd is the fixture file.""" + fd = int(fd_text) + try: + st = os.fstat(fd) + expected = os.stat(lock_path) + has_fd = (st.st_dev, st.st_ino) == (expected.st_dev, expected.st_ino) + except OSError as error: + if error.errno != errno.EBADF: + raise + has_fd = False + emit({"event": "keeper_ready", "pid": os.getpid(), "has_fd": has_fd}) + clean = run_commands(deadline, {}) + if has_fd: + os.close(fd) + return 0 if clean else 3 + + +def actor_scm_sender(lock_path, sock_fd, deadline): + fd = os.open(lock_path, os.O_RDWR | os.O_CLOEXEC | os.O_NOFOLLOW) + lock_exclusive(fd) + sock = socket.socket(fileno=sock_fd) + sock.sendmsg([b"x"], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, + array.array("i", [fd]))]) + state = {"fd": fd} + + def close_own_fd(): + os.close(state["fd"]) + state["fd"] = None + return {"event": "closed", "who": "sender"} + + try: + emit({"event": "ready", "pid": os.getpid()}) + clean = run_commands(deadline, {"close_own_fd": close_own_fd}) + finally: + if state["fd"] is not None: + os.close(state["fd"]) + sock.close() + return 0 if clean else 3 + + +def actor_scm_receiver(sock_fd, deadline): + sock = socket.socket(fileno=sock_fd) + if not wait_readable(sock.fileno(), time.monotonic() + READY_TIMEOUT): + emit({"event": "failed", "reason": "not_received"}) + return 3 + _, ancdata, _, _ = sock.recvmsg(1, socket.CMSG_SPACE(array.array("i").itemsize)) + fds = array.array("i") + for level, kind, data in ancdata: + if level == socket.SOL_SOCKET and kind == socket.SCM_RIGHTS: + fds.frombytes(data[:len(data) - (len(data) % fds.itemsize)]) + try: + if len(fds) != 1: + emit({"event": "failed", "reason": f"received:{len(fds)}"}) + return 3 + emit({"event": "ready", "pid": os.getpid(), "received": len(fds)}) + clean = run_commands(deadline, {}) + finally: + for fd in fds: + os.close(fd) + sock.close() + return 0 if clean else 3 + + +def actor_main(argv): + deadline = time.monotonic() + ACTOR_LIFETIME + role = argv[0] + if role == "fork_shared": + return actor_fork_shared(argv[1], deadline) + if role == "exec_inherit": + return actor_exec(argv[1], deadline, True) + if role == "exec_cloexec": + return actor_exec(argv[1], deadline, False) + if role == "keeper": + return actor_keeper(argv[1], argv[2], deadline) + if role == "scm_sender": + return actor_scm_sender(argv[1], int(argv[2]), deadline) + if role == "scm_receiver": + return actor_scm_receiver(int(argv[1]), deadline) + raise SystemExit(f"unknown role {role}") + + +# ---------------------------------------------------------- diagnostics + +def run_diagnostic(run, argv, timeout=DIAGNOSTIC_TIMEOUT, limit=STREAM_LIMIT): + """Run a diagnostic with a bound and byte caps, never signalling it. + + Returns {"status", "stdout", "stderr", "problems"}; on timeout the process + is registered in run.pending and reported with status None. + """ + proc = subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, bufsize=0, + env={"LC_ALL": "C", "LANG": "C", "PATH": "/usr/bin:/bin:/usr/sbin:/sbin"}) + out_fd, err_fd = proc.stdout.fileno(), proc.stderr.fileno() + streams = {out_fd: bytearray(), err_fd: bytearray()} + problems = [] + deadline = time.monotonic() + timeout + open_fds = set(streams) + while open_fds: + remaining = deadline - time.monotonic() + if remaining <= 0: + run.pending.append({"argv0": argv[0], "pid": proc.pid, "proc": proc}) + run.incomplete.append(f"diagnostic_timeout:{os.path.basename(argv[0])}:{proc.pid}") + return {"status": None, "stdout": b"", "stderr": b"", "problems": ["timeout"]} + ready, _, _ = select.select(list(open_fds), [], [], remaining) + for fd in ready: + chunk = os.read(fd, 65536) + if not chunk: + open_fds.discard(fd) + continue + room = limit - len(streams[fd]) + if len(chunk) > room and "output_limit" not in problems: + problems.append("output_limit") + streams[fd] += chunk[:max(0, room)] + status = None + while time.monotonic() < deadline and status is None: + status = proc.poll() + if status is None: + time.sleep(0.005) + if status is None: + run.pending.append({"argv0": argv[0], "pid": proc.pid, "proc": proc}) + run.incomplete.append(f"diagnostic_exit_wait:{os.path.basename(argv[0])}:{proc.pid}") + problems.append("exit_wait") + proc.stdout.close() + proc.stderr.close() + return {"status": status, "stdout": bytes(streams[out_fd]), "stderr": bytes(streams[err_fd]), "problems": problems} + + +FLAGS = re.compile(r"^0x[0-9a-f]+;0x[0-9a-f]+$") + + +def parse_fg(out): + """Strict `lsof -F0pfG` parse. Any deviation is a problem, never no_bit.""" + problems = [] + pids = {} + pid = None + if not out: + return {"problems": problems, "pids": {}} + if not out.endswith(b"\n"): + problems.append("unterminated_output") + for number, record in enumerate(out.split(b"\n")[:-1] if out.endswith(b"\n") else out.split(b"\n")): + try: + text = record.decode("ascii") + except UnicodeDecodeError: + problems.append(f"non_ascii:{number}") + continue + if not text.endswith("\0"): + problems.append(f"unterminated_record:{number}") + continue + fields = text[:-1].split("\0") + tags = "".join(f[:1] for f in fields) + if tags == "p" and re.fullmatch(r"[1-9][0-9]*", fields[0][1:]): + pid = int(fields[0][1:]) + if pid in pids: + problems.append(f"duplicate_pid:{pid}") + pids.setdefault(pid, []) + elif tags == "fG" and pid is not None and re.fullmatch(r"[0-9]+", fields[0][1:]) and FLAGS.match(fields[1][1:]): + head = fields[1][1:].split(";", 1)[0] + pids[pid].append({"fd": fields[0][1:], "flags": fields[1][1:], + "was_locked": bool(int(head, 16) & FWASLOCKED)}) + else: + problems.append(f"record:{number}:{tags}") + for p, descriptors in pids.items(): + if not descriptors: + problems.append(f"pid_without_fd:{p}") + return {"problems": problems, "pids": {str(k): v for k, v in pids.items()}} + + +def fg_scan(run, lock_path): + result = run_diagnostic(run, [LSOF, "-nP", "+fg", "-F0pfG", "--", lock_path]) + problems = list(result["problems"]) + if result["stderr"]: + problems.append("stderr") + status, out = result["status"], result["stdout"] + # lsof exits 1 with no output when nothing has the file open. + if status is not None and (status not in (0, 1) or (status == 1 and out) or (status == 0 and not out)): + problems.append(f"status:{status}") + parsed = parse_fg(out) if status is not None else {"problems": [], "pids": {}} + problems += parsed["problems"] + return {"valid": status is not None and not problems, "problems": problems, "status": status, "pids": parsed["pids"]} + + +def probe(lock_path): + fd = os.open(lock_path, os.O_RDONLY | os.O_CLOEXEC | os.O_NOFOLLOW) + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as error: + if error.errno in (errno.EWOULDBLOCK, errno.EAGAIN): + return "held" + return f"error:{errno.errorcode.get(error.errno, error.errno)}" + finally: + os.close(fd) + return "free" + + +def classify(scan, roles): + seen = {} + for role, pid in roles.items(): + descriptors = scan["pids"].get(str(pid)) + if descriptors is None: + seen[role] = "absent" + else: + seen[role] = "bit" if any(d["was_locked"] for d in descriptors) else "no_bit" + other = sorted(set(scan["pids"]) - {str(p) for p in roles.values()}) + return seen, len(other) + + +def observe(run, lock_path, stage, roles): + scan = fg_scan(run, lock_path) + seen, unexpected = classify(scan, roles) + return {"stage": stage, "probe": probe(lock_path), "roles": seen, + "unexpected_pids": unexpected, "scan_valid": scan["valid"], + "scan_problems": scan["problems"], "raw": scan} + + +# --------------------------------------------------------- orchestrator + +class Run: + def __init__(self, scratch): + self.scratch = scratch + self.incomplete = [] + self.failures = [] + self.pending = [] + self.actors = [] + + def start(self, label, argv, pass_fds=()): + proc = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE, + pass_fds=pass_fds, close_fds=True, bufsize=0) + actor = {"label": label, "proc": proc, "framed": Framed(proc.stdout.fileno()), "finished": None} + self.actors.append(actor) + return actor + + def start_actor(self, label, args, pass_fds=()): + return self.start(label, [sys.executable, "-I", SCRIPT, "actor", *args], pass_fds) + + def read_event(self, actor, expected): + """Next event; it must be `expected`. Anything else is recorded.""" + status, line = actor["framed"].read_line(time.monotonic() + READY_TIMEOUT) + label = actor["label"] + if status == "timeout": + self.incomplete.append(f"{label}:no_event") + return None + if status != "line": + self.failures.append(f"{label}:{status}") + return None + event = parse_event(line) + if event is None: + self.failures.append(f"{label}:malformed_event") + return None + if event["event"] != expected: + self.failures.append(f"{label}:unexpected_event:{event['event']}:{event.get('reason', '')}") + return None + return event + + def command(self, actor, text, expected): + actor["proc"].stdin.write(text.encode() + b"\n") + actor["proc"].stdin.flush() + return self.read_event(actor, expected) + + def finish(self, actor): + """stdin EOF, bounded read of remaining events, bounded exit wait.""" + if actor["finished"] is not None: + return actor["finished"] + proc, label = actor["proc"], actor["label"] + try: + proc.stdin.close() + except OSError: + pass + events = [] + deadline = time.monotonic() + EXIT_TIMEOUT + while True: + status, line = actor["framed"].read_line(deadline) + if status == "line": + event = parse_event(line) + if event is None: + self.failures.append(f"{label}:malformed_event") + else: + events.append(event) + continue + if status == "timeout": + self.incomplete.append(f"{label}:output_wait") + elif status == "limit": + self.failures.append(f"{label}:output_limit") + elif status != "eof": + self.failures.append(f"{label}:output_{status}") + break + code = None + while time.monotonic() < deadline and code is None: + code = proc.poll() + if code is None: + time.sleep(0.01) + if code is None: + self.incomplete.append(f"{label}:exit_wait:{proc.pid}") + elif code != 0: + self.failures.append(f"{label}:exit:{code}") + if code is not None: + proc.stdout.close() + actor["finished"] = {"exit": code, "events": events} + return actor["finished"] + + def finish_all(self): + for actor in self.actors: + try: + self.finish(actor) + except Exception as error: + # A broken actor's output must not skip EOF for later actors. + self.failures.append(f"{actor['label']}:finish_error:{type(error).__name__}") + try: + actor["proc"].stdin.close() + except OSError: + pass + self.incomplete.append(f"{actor['label']}:finish_unconfirmed:{actor['proc'].pid}") + + def poll_pending(self): + states = [] + for item in self.pending: + code = item["proc"].poll() + states.append({"argv0": item["argv0"], "pid": item["pid"], + "state": "running" if code is None else f"exited:{code}"}) + return states + + +def fixture(run, name): + path = run.scratch / f"{name}.lock" + path.write_bytes(b"") + path.chmod(0o600) + return str(path) + + +def scenario_fork_shared(run): + lock = fixture(run, "fork_shared") + actor = run.start_actor("fork_shared", ["fork_shared", lock]) + ready = run.read_event(actor, "ready") + if not ready: + return {"ready": None, "finish": run.finish(actor)} + roles = {"parent": ready["pid"], "forked_child": ready["child_pid"]} + stages = [observe(run, lock, "both_alive", roles)] + closed = run.command(actor, "close_parent_fd", "closed") + stages.append(observe(run, lock, "parent_closed_fd_alive", roles)) + finish = run.finish(actor) + stages.append(observe(run, lock, "all_closed", {})) + return {"ready": ready, "closed": closed, "stages": stages, "finish": finish} + + +def scenario_exec(run, role): + lock = fixture(run, role) + actor = run.start_actor(role, [role, lock]) + ready = run.read_event(actor, "ready") + if not ready: + return {"ready": None, "finish": run.finish(actor)} + roles = {"parent": ready["pid"], "exec_child": ready["child_pid"]} + stages = [observe(run, lock, "both_alive", roles)] + closed = run.command(actor, "close_parent_fd", "closed") + stages.append(observe(run, lock, "parent_closed_fd_alive", roles)) + finish = run.finish(actor) + stages.append(observe(run, lock, "all_closed", {})) + return {"ready": ready, "keeper": ready.get("keeper"), "closed": closed, "stages": stages, "finish": finish} + + +def scenario_scm_rights(run): + lock = fixture(run, "scm_rights") + left, right = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM) + try: + receiver = run.start_actor("scm_receiver", ["scm_receiver", str(right.fileno())], (right.fileno(),)) + sender = run.start_actor("scm_sender", ["scm_sender", lock, str(left.fileno())], (left.fileno(),)) + finally: + left.close() + right.close() + sent = run.read_event(sender, "ready") + received = run.read_event(receiver, "ready") + if not sent or not received: + return {"sent": sent, "received": received, "finish": [run.finish(sender), run.finish(receiver)]} + roles = {"sender_flock_caller": sent["pid"], "receiver_never_flocked": received["pid"]} + stages = [observe(run, lock, "both_hold_description", roles)] + closed = run.command(sender, "close_own_fd", "closed") + stages.append(observe(run, lock, "sender_closed_fd_alive", roles)) + finish_sender = run.finish(sender) + stages.append(observe(run, lock, "sender_exited", roles)) + finish_receiver = run.finish(receiver) + stages.append(observe(run, lock, "all_closed", {})) + return {"sent": sent, "received": received, "closed": closed, "stages": stages, + "finish": [finish_sender, finish_receiver]} + + +# ------------------------------------------------------- expected table + +# (stage, probe, {role: bit | no_bit | absent}) in order, per scenario. +EXPECTED_STAGES = { + "fork_shared": [ + ("both_alive", "held", {"parent": "bit", "forked_child": "bit"}), + ("parent_closed_fd_alive", "held", {"parent": "absent", "forked_child": "bit"}), + ("all_closed", "free", {}), + ], + "exec_inherit": [ + ("both_alive", "held", {"parent": "bit", "exec_child": "bit"}), + ("parent_closed_fd_alive", "held", {"parent": "absent", "exec_child": "bit"}), + ("all_closed", "free", {}), + ], + "exec_cloexec": [ + ("both_alive", "held", {"parent": "bit", "exec_child": "absent"}), + ("parent_closed_fd_alive", "free", {"parent": "absent", "exec_child": "absent"}), + ("all_closed", "free", {}), + ], + "scm_rights": [ + ("both_hold_description", "held", {"sender_flock_caller": "bit", "receiver_never_flocked": "bit"}), + ("sender_closed_fd_alive", "held", {"sender_flock_caller": "absent", "receiver_never_flocked": "bit"}), + ("sender_exited", "held", {"sender_flock_caller": "absent", "receiver_never_flocked": "bit"}), + ("all_closed", "free", {}), + ], +} +KEEPER_HAS_FD = {"exec_inherit": True, "exec_cloexec": False} +CLOSED_WHO = {"fork_shared": "parent", "exec_inherit": "parent", "exec_cloexec": "parent", "scm_rights": "sender"} + + +def is_pid(value): + return isinstance(value, int) and not isinstance(value, bool) and value > 0 + + +def evaluate(scenarios): + """Mismatches between collected scenarios and the expected table.""" + mismatches = [] + for name, expected in EXPECTED_STAGES.items(): + result = scenarios.get(name) or {} + stages = result.get("stages") + if not stages: + mismatches.append(f"{name}:no_stages") + continue + if [s["stage"] for s in stages] != [e[0] for e in expected]: + mismatches.append(f"{name}:stage_list") + continue + for stage, (label, want_probe, want_roles) in zip(stages, expected): + where = f"{name}/{label}" + if not stage["scan_valid"]: + mismatches.append(f"{where}:scan_invalid") + if stage["unexpected_pids"]: + mismatches.append(f"{where}:unexpected_pids:{stage['unexpected_pids']}") + if stage["probe"] != want_probe: + mismatches.append(f"{where}:probe:{stage['probe']}") + if stage["roles"] != want_roles: + mismatches.append(f"{where}:roles") + closed = result.get("closed") + if not closed or closed.get("who") != CLOSED_WHO[name]: + mismatches.append(f"{name}:closed_event") + if name == "scm_rights": + sent, received = result.get("sent") or {}, result.get("received") or {} + if not is_pid(sent.get("pid")) or not is_pid(received.get("pid")) or received.get("received") != 1: + mismatches.append(f"{name}:ready_events") + finishes = result.get("finish") or [] + if len(finishes) != 2 or any(f.get("exit") != 0 or f.get("events") for f in finishes): + mismatches.append(f"{name}:finish") + else: + ready = result.get("ready") or {} + if not is_pid(ready.get("pid")) or not is_pid(ready.get("child_pid")) or ready["pid"] == ready["child_pid"]: + mismatches.append(f"{name}:ready_event") + if name in KEEPER_HAS_FD: + keeper = ready.get("keeper") or {} + if keeper.get("event") != "keeper_ready" or keeper.get("has_fd") is not KEEPER_HAS_FD[name] or \ + keeper.get("pid") != ready.get("child_pid"): + mismatches.append(f"{name}:keeper") + finish = result.get("finish") or {} + if finish.get("exit") != 0 or finish.get("events") != [{"code": 0, "event": "child_exit"}]: + mismatches.append(f"{name}:finish") + return mismatches + + +# ------------------------------------------------------------- results + +def filesystem_of(run, path): + """Filesystem type of the mount containing the canonical path.""" + canonical = os.path.realpath(path) + result = run_diagnostic(run, [MOUNT]) + if result["status"] != 0 or result["problems"]: + return "unknown" + best = ("", "unknown") + for line in result["stdout"].decode("utf-8", "replace").splitlines(): + if " on " not in line or " (" not in line: + continue + point = line.split(" on ", 1)[1].rsplit(" (", 1)[0] + kind = line.rsplit(" (", 1)[1].split(",", 1)[0].rstrip(")") + if (canonical + "/").startswith(point.rstrip("/") + "/") and len(point) >= len(best[0]): + best = (point, kind) + return best[1] + + +def open_result(path): + """Create the result file exclusively, 0600, refusing symlinks.""" + return os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW | os.O_CLOEXEC, 0o600) + + +def script_sha256(): + return hashlib.sha256(Path(SCRIPT).read_bytes()).hexdigest() + + +def script_identity(start_sha256): + """Compare the on-disk script at the start and end of a run.""" + end_sha256 = script_sha256() + return {"start_sha256": start_sha256, "end_sha256": end_sha256, + "unchanged": start_sha256 == end_sha256} + + +def main(): + if len(sys.argv) > 1 and sys.argv[1] == "actor": + return actor_main(sys.argv[2:]) + parser = argparse.ArgumentParser() + parser.add_argument("--scratch-root", required=True, + help="existing directory; a private child is created") + parser.add_argument("--result", required=True, + help="new JSON result path, outside the scratch directory") + args = parser.parse_args() + if platform.system() != "Darwin": + raise SystemExit("macOS only: FWASLOCKED and lsof +fg are XNU-specific") + # Created before the fixture directory, so it is never inside it and is + # never removed by fixture cleanup. + result_fd = open_result(args.result) + scratch = Path(tempfile.mkdtemp(prefix="codex-127-fixture-", dir=args.scratch_root)) + scratch.chmod(0o700) + run = Run(scratch) + scenarios = {} + errors = [] + environment = {} + start_sha256 = None + try: + # Actors exec SCRIPT later, so bracket the run rather than relying on + # an import-time hash or hashing only after measurements complete. + start_sha256 = script_sha256() + scenarios["fork_shared"] = scenario_fork_shared(run) + scenarios["exec_inherit"] = scenario_exec(run, "exec_inherit") + scenarios["exec_cloexec"] = scenario_exec(run, "exec_cloexec") + scenarios["scm_rights"] = scenario_scm_rights(run) + except Exception as error: # recorded, then reported nonzero + errors.append(f"{type(error).__name__}: {error}") + finally: + run.finish_all() + try: + version = run_diagnostic(run, [LSOF, "-v"]) + environment = { + # Hostname deliberately omitted. + "system": platform.system(), "release": platform.release(), "machine": platform.machine(), + "mac_ver": platform.mac_ver()[0], "python": platform.python_version(), + "lsof": next((l.strip() for l in version["stderr"].decode("utf-8", "replace").splitlines() + if "revision:" in l and "latest" not in l), None), + "filesystem": filesystem_of(run, scratch), + } + except Exception as error: + errors.append(f"environment: {type(error).__name__}: {error}") + mismatches = evaluate(scenarios) + identity = {"start_sha256": start_sha256, "end_sha256": None, "unchanged": False} + try: + if start_sha256 is not None: + identity = script_identity(start_sha256) + except Exception as error: + errors.append(f"script_identity: {type(error).__name__}: {error}") + if not identity["unchanged"]: + mismatches.append("script_identity:changed_or_unavailable") + pending = run.poll_pending() + collected = not run.failures and not run.incomplete and not errors + measured = not mismatches + result = { + "ok": collected and measured, + "collected": collected, + "measured": measured, + "failures": run.failures, + "incomplete": run.incomplete, + "errors": errors, + "mismatches": mismatches, + "pending_diagnostics": pending, + "signals_sent_by_script": "none: no kill calls and no killing timeouts; see module docstring", + "script_sha256": identity["end_sha256"], + "script_identity": identity, + "environment": environment, + "scenarios": scenarios, + } + with os.fdopen(result_fd, "w") as handle: + handle.write(json.dumps(result, indent=2, sort_keys=True, default=str)) + for name, scenario in scenarios.items(): + for stage in scenario.get("stages", []): + roles = ", ".join(f"{k}={v}" for k, v in stage["roles"].items()) + print(f"{name}/{stage['stage']}: probe={stage['probe']} {roles} " + f"unexpected={stage['unexpected_pids']} scan_valid={stage['scan_valid']}") + print(json.dumps({"ok": result["ok"], "collected": collected, "measured": measured, + "failures": run.failures, "incomplete": run.incomplete, "errors": errors, + "mismatches": mismatches, "filesystem": environment.get("filesystem")})) + if result["ok"] and not pending: + for child in scratch.iterdir(): + child.unlink() + scratch.rmdir() + return 0 if result["ok"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/experimental/127-macos-contract/test_inherit.py b/experimental/127-macos-contract/test_inherit.py new file mode 100644 index 0000000..b5dcf8d --- /dev/null +++ b/experimental/127-macos-contract/test_inherit.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Deterministic tests for inherit.py helpers (#127 research). + + RESEARCH_TMP_ROOT= python3 -I experimental/127-macos-contract/test_inherit.py + +Uses only finite synthetic children started here. No signals are sent. +""" + +import copy +import json +import os +import shutil +import sys +import tempfile +import time +import unittest +from unittest import mock +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import inherit # noqa: E402 + +ROOT = os.environ.get("RESEARCH_TMP_ROOT") +PY = sys.executable + + +def scratch(): + if not ROOT: + raise unittest.SkipTest("RESEARCH_TMP_ROOT is required") + path = Path(tempfile.mkdtemp(prefix="codex-127-test-", dir=ROOT)) + path.chmod(0o700) + return path + + +class FramedTest(unittest.TestCase): + def pipe(self, data, close=True): + r, w = os.pipe() + os.write(w, data) + if close: + os.close(w) + w = None + return r, w + + def test_two_lines_in_one_chunk_are_both_returned(self): + r, _ = self.pipe(b'{"a":1}\n{"b":2}\n') + f = inherit.Framed(r) + self.assertEqual(f.read_line(time.monotonic() + 1), ("line", b'{"a":1}')) + self.assertEqual(f.read_line(time.monotonic() + 1), ("line", b'{"b":2}')) + self.assertEqual(f.read_line(time.monotonic() + 1), ("eof", None)) + os.close(r) + + def test_partial_line_times_out_and_is_kept(self): + r, w = self.pipe(b'{"a":', close=False) + f = inherit.Framed(r) + self.assertEqual(f.read_line(time.monotonic() + 0.1), ("timeout", None)) + os.write(w, b"1}\n") + self.assertEqual(f.read_line(time.monotonic() + 1), ("line", b'{"a":1}')) + os.close(w) + os.close(r) + + def test_partial_line_at_eof_is_not_a_line(self): + r, _ = self.pipe(b'{"a":1}') + self.assertEqual(inherit.Framed(r).read_line(time.monotonic() + 1), ("unterminated", None)) + os.close(r) + + def test_complete_line_boundary(self): + for size in (63, 64, 65): + with self.subTest(size=size): + r, _ = self.pipe(b"x" * size + b"\n") + try: + expected = ("line", b"x" * size) if size <= 64 else ("limit", None) + self.assertEqual(inherit.Framed(r, line_limit=64).read_line(time.monotonic() + 1), expected) + finally: + os.close(r) + + def test_newline_in_next_chunk_cannot_bypass_line_cap(self): + for tail, expected in ((b"\n", "line"), (b"x\n", "limit")): + with self.subTest(tail=tail): + framed = inherit.Framed(123, line_limit=64) + with mock.patch.object(inherit.select, "select", return_value=([123], [], [])), \ + mock.patch.object(inherit.os, "read", side_effect=[b"x" * 64, tail]): + self.assertEqual(framed.read_line(time.monotonic() + 1)[0], expected) + + def test_stream_overflow_rejects_buffered_valid_event(self): + event = b'{"event":"ready"}\n' + r, _ = self.pipe(event + b"x") + try: + framed = inherit.Framed(r, stream_limit=len(event)) + self.assertEqual(framed.read_line(time.monotonic() + 1), ("limit", None)) + self.assertEqual(framed.read_line(time.monotonic() + 1), ("limit", None)) + finally: + os.close(r) + + def test_line_and_stream_limits(self): + r, _ = self.pipe(b"x" * 200) + self.assertEqual(inherit.Framed(r, line_limit=64).read_line(time.monotonic() + 1), ("limit", None)) + os.close(r) + r, _ = self.pipe(b"ok\n" * 100) + f = inherit.Framed(r, line_limit=64, stream_limit=10) + statuses = [f.read_line(time.monotonic() + 1)[0] for _ in range(5)] + self.assertIn("limit", statuses) + os.close(r) + + def test_events_must_be_objects_with_a_string_event(self): + self.assertEqual(inherit.parse_event(b'{"event":"ready","pid":1}'), {"event": "ready", "pid": 1}) + for bad in (b"", b"nope", b"[1]", b'{"pid":1}', b'{"event":3}', b"\xff"): + self.assertIsNone(inherit.parse_event(bad), bad) + + +class ParseFgTest(unittest.TestCase): + GOOD = b"p100\0\nf3\0G0x4003;0x3\0\np200\0\nf4\0G0x3;0x0\0\n" + + def test_valid_output(self): + parsed = inherit.parse_fg(self.GOOD) + self.assertEqual(parsed["problems"], []) + self.assertTrue(parsed["pids"]["100"][0]["was_locked"]) + self.assertFalse(parsed["pids"]["200"][0]["was_locked"]) + + def test_empty_output_has_no_records(self): + # lsof exits 1 with no output when nothing has the file open. + self.assertEqual(inherit.parse_fg(b""), {"problems": [], "pids": {}}) + + def test_deviations_are_problems_never_no_bit(self): + cases = { + "missing G": b"p100\0\nf3\0\n", + "G before f": b"p100\0\nG0x4003;0x3\0f3\0\n", + "non-hex flags": b"p100\0\nf3\0G0xzz;0x3\0\n", + "flags without second field": b"p100\0\nf3\0G0x4003\0\n", + "unknown tag": b"p100\0\nn/tmp/x\0\n", + "non-numeric pid": b"pabc\0\nf3\0G0x4003;0x3\0\n", + "fd before pid": b"f3\0G0x4003;0x3\0\n", + "unterminated record": b"p100\0\nf3\0G0x4003;0x3\n", + "unterminated output": b"p100\0\nf3\0G0x4003;0x3\0", + "pid without fd": b"p100\0\n", + "duplicate pid": b"p100\0\nf3\0G0x3;0x0\0\np100\0\nf4\0G0x3;0x0\0\n", + } + for name, out in cases.items(): + with self.subTest(name): + self.assertNotEqual(inherit.parse_fg(out)["problems"], [], name) + + +def stage(name, probe, roles, valid=True, unexpected=0): + return {"stage": name, "probe": probe, "roles": roles, "scan_valid": valid, "unexpected_pids": unexpected} + + +def nominal(): + scenarios = {} + for name, expected in inherit.EXPECTED_STAGES.items(): + stages = [stage(*e) for e in expected] + closed = {"event": "closed", "who": inherit.CLOSED_WHO[name]} + if name == "scm_rights": + scenarios[name] = {"stages": stages, "closed": closed, "sent": {"event": "ready", "pid": 10}, + "received": {"event": "ready", "pid": 11, "received": 1}, + "finish": [{"exit": 0, "events": []}, {"exit": 0, "events": []}]} + else: + ready = {"event": "ready", "pid": 10, "child_pid": 11} + if name in inherit.KEEPER_HAS_FD: + ready["keeper"] = {"event": "keeper_ready", "pid": 11, "has_fd": inherit.KEEPER_HAS_FD[name]} + scenarios[name] = {"stages": stages, "closed": closed, "ready": ready, + "finish": {"exit": 0, "events": [{"code": 0, "event": "child_exit"}]}} + return scenarios + + +class EvaluateTest(unittest.TestCase): + def test_script_identity_compares_start_and_end_hashes(self): + with mock.patch.object(inherit, "script_sha256", return_value="a" * 64): + self.assertTrue(inherit.script_identity("a" * 64)["unchanged"]) + changed = inherit.script_identity("b" * 64) + self.assertFalse(changed["unchanged"]) + self.assertEqual(changed["start_sha256"], "b" * 64) + self.assertEqual(changed["end_sha256"], "a" * 64) + + def test_nominal_table_passes(self): + self.assertEqual(inherit.evaluate(nominal()), []) + + def mutate(self, fn): + scenarios = copy.deepcopy(nominal()) + fn(scenarios) + return inherit.evaluate(scenarios) + + def test_counterexamples_fail(self): + def bit(s): + s["fork_shared"]["stages"][0]["roles"]["forked_child"] = "no_bit" + + def invalid(s): + s["scm_rights"]["stages"][1]["scan_valid"] = False + + def unexpected(s): + s["exec_inherit"]["stages"][0]["unexpected_pids"] = 1 + + def probe_error(s): + s["exec_cloexec"]["stages"][1]["probe"] = "error:EIO" + + def keeper(s): + s["exec_cloexec"]["ready"]["keeper"]["has_fd"] = True + + def keeper_missing(s): + del s["exec_inherit"]["ready"]["keeper"] + + def received(s): + s["scm_rights"]["received"]["received"] = 2 + + def closed(s): + s["fork_shared"]["closed"] = None + + def child_exit(s): + s["exec_inherit"]["finish"]["events"] = [{"code": 3, "event": "child_exit"}] + + def missing(s): + del s["scm_rights"] + + def stage_order(s): + s["fork_shared"]["stages"].reverse() + + def same_pid(s): + s["fork_shared"]["ready"]["child_pid"] = 10 + + for fn in (bit, invalid, unexpected, probe_error, keeper, keeper_missing, received, closed, + child_exit, missing, stage_order, same_pid): + with self.subTest(fn.__name__): + self.assertNotEqual(self.mutate(fn), [], fn.__name__) + + +class ProcessTest(unittest.TestCase): + def setUp(self): + self.dir = scratch() + self.run = inherit.Run(self.dir) + + def tearDown(self): + shutil.rmtree(self.dir) + + def test_diagnostic_timeout_is_incomplete_and_unsignalled(self): + result = inherit.run_diagnostic(self.run, [PY, "-I", "-c", "import time; time.sleep(1.0)"], timeout=0.2) + self.assertIsNone(result["status"]) + self.assertTrue(self.run.incomplete[0].startswith("diagnostic_timeout:")) + self.assertEqual(self.run.poll_pending()[0]["state"], "running") + time.sleep(1.5) + self.assertEqual(self.run.poll_pending()[0]["state"], "exited:0") + for item in self.run.pending: + item["proc"].stdout.close() + item["proc"].stderr.close() + + def test_diagnostic_output_is_capped(self): + result = inherit.run_diagnostic(self.run, [PY, "-I", "-c", "import sys; sys.stdout.write('x' * 300000)"], limit=1000) + self.assertEqual(result["status"], 0) + self.assertEqual(len(result["stdout"]), 1000) + self.assertIn("output_limit", result["problems"]) + + def test_early_exit_is_a_failure_not_ready(self): + actor = self.run.start("early", [PY, "-I", "-c", "pass"]) + self.assertIsNone(self.run.read_event(actor, "ready")) + self.assertEqual(self.run.failures, ["early:eof"]) + self.assertEqual(self.run.finish(actor)["exit"], 0) + + def test_not_ready_event_is_not_ready(self): + body = "import json,sys; print(json.dumps({'event':'failed','reason':'keeper_not_ready:timeout'})); sys.stdout.flush(); sys.stdin.read()" + actor = self.run.start("keeper", [PY, "-I", "-c", body]) + self.assertIsNone(self.run.read_event(actor, "ready")) + self.assertEqual(self.run.failures, ["keeper:unexpected_event:failed:keeper_not_ready:timeout"]) + self.assertEqual(self.run.finish(actor)["exit"], 0) + + def test_partial_line_then_silence_is_incomplete(self): + body = "import sys,time; sys.stdout.write('{\"event\":'); sys.stdout.flush(); sys.stdin.read()" + actor = self.run.start("partial", [PY, "-I", "-c", body]) + old = inherit.READY_TIMEOUT + inherit.READY_TIMEOUT = 0.3 + try: + self.assertIsNone(self.run.read_event(actor, "ready")) + finally: + inherit.READY_TIMEOUT = old + self.assertEqual(self.run.incomplete, ["partial:no_event"]) + self.assertEqual(self.run.finish(actor)["exit"], 0) + + def test_finish_all_closes_every_registered_actor(self): + body = "import sys; sys.stdin.read()" + actors = [self.run.start(f"a{i}", [PY, "-I", "-c", body]) for i in range(2)] + self.run.finish_all() + self.assertEqual([a["finished"]["exit"] for a in actors], [0, 0]) + + def test_valid_event_followed_by_partial_tail_fails_finish(self): + body = "import sys; sys.stdout.write('{\"event\":\"ready\"}\\n{\"event\":'); sys.stdout.flush(); sys.stdin.read()" + actor = self.run.start("tail", [PY, "-I", "-c", body]) + self.assertEqual(self.run.read_event(actor, "ready"), {"event": "ready"}) + self.assertEqual(self.run.finish(actor)["exit"], 0) + self.assertIn("tail:output_unterminated", self.run.failures) + + def test_finish_all_continues_after_actor_exception(self): + body = "import sys; sys.stdin.read()" + actors = [self.run.start(f"a{i}", [PY, "-I", "-c", body]) for i in range(2)] + finish = self.run.finish + + def faulty(actor): + if actor is actors[0]: + raise OSError("synthetic broken output") + return finish(actor) + + try: + with mock.patch.object(self.run, "finish", side_effect=faulty): + self.run.finish_all() + self.assertEqual(actors[1]["finished"]["exit"], 0) + self.assertIn("a0:finish_error:OSError", self.run.failures) + self.assertTrue(self.run.incomplete[0].startswith("a0:finish_unconfirmed:")) + finally: + finish(actors[0]) + + def test_result_file_refuses_existing_and_symlink(self): + existing = self.dir / "existing.json" + existing.write_text("{}") + with self.assertRaises(FileExistsError): + inherit.open_result(str(existing)) + link = self.dir / "link.json" + link.symlink_to(self.dir / "target.json") + with self.assertRaises(OSError): + inherit.open_result(str(link)) + self.assertFalse((self.dir / "target.json").exists()) + fd = inherit.open_result(str(self.dir / "new.json")) + os.close(fd) + self.assertEqual((self.dir / "new.json").stat().st_mode & 0o777, 0o600) + + +if __name__ == "__main__": + unittest.main(verbosity=1)