From 62ec00aee110f814bdbc007149c3905191c55bc2 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 12:53:08 +0900 Subject: [PATCH 01/29] Write down who owns the relay's state and its daemon The relay was proven with an explicit temporary database and a narrow driver, so nothing recorded what an operator is entitled to assume in real use: which store was actually selected, who may start or stop the daemon, and what a stuck delivery means. docs/operations.md states that contract. Every section carries a status, so a reader can tell the agreed behaviour from the shipped behaviour rather than inferring it: the state precedence and reconciliation exist today, the selection diagnostics, service lifecycle and ownership land in this PR, and the delivery phase taxonomy lands in the dependent one. It also records the trap that --state alone does not move the bridge adapter's transport ledger, which reads CODEX_SESSION_RELAY_STATE independently. --- packages/codex-session-relay/README.md | 2 + .../codex-session-relay/docs/operations.md | 145 ++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 packages/codex-session-relay/docs/operations.md diff --git a/packages/codex-session-relay/README.md b/packages/codex-session-relay/README.md index 64ad2e9..5b22355 100644 --- a/packages/codex-session-relay/README.md +++ b/packages/codex-session-relay/README.md @@ -336,3 +336,5 @@ These are recorded because behaviour depends on them. - `docs/protocol-v1.md` — the wire and record protocol, derived from the frozen contract. - `docs/invariants.md` — every invariant and the code that enforces it. +- `docs/operations.md` — where the state lives, who owns the daemon, and how to read a + stuck delivery. Each section says whether the behaviour is implemented or planned. diff --git a/packages/codex-session-relay/docs/operations.md b/packages/codex-session-relay/docs/operations.md new file mode 100644 index 0000000..d2508e0 --- /dev/null +++ b/packages/codex-session-relay/docs/operations.md @@ -0,0 +1,145 @@ +# Operating the relay + +Who owns the state, who owns the daemon, and what an operator can ask it. The delivery +contract itself is in [protocol-v1.md](protocol-v1.md) and the refusals that enforce it are +in [invariants.md](invariants.md). + +Each section carries a status, because this document is written alongside the work that +makes it true: **implemented** means the behaviour exists and is tested in this package, +**planned** means the contract is agreed and the code lands in a named pull request. +Nothing here describes an installed runtime, a registered service or a live App Server; +source in this package changes none of those. + +## Where the state lives + +Every participant of one assignment must read and write the same SQLite store. The +directory is chosen by the first rule that applies: + +| Precedence | Source | Notes | +|---|---|---| +| 1 | `--state ` | explicit, wins over everything | +| 2 | `CODEX_SESSION_RELAY_STATE` | also read independently by the bridge adapter for its transport ledger | +| 3 | `XDG_STATE_HOME/codex-session-relay/` | the scope is a hash of the App Server socket path | +| 4 | `~/.local/state/codex-session-relay/` | the default | + +The store is `/relay.sqlite3`. `codex-session-relay doctor` reports which rule won, the +value that won, the resolved database path and the measured read/write access, so a +participant never has to infer its own configuration. + +Setting `--state` alone is not enough for an isolated run. The bridge adapter resolves its +transport ledger from `CODEX_SESSION_RELAY_STATE` independently, so a run that overrides only +the flag splits the relay store from the ledger that carries send idempotency. Set both. + +Status: planned in PR-A. Today `state_dir()` implements the precedence but nothing reports +the selection. + +## Proving two participants share one store + +A path string is not proof: symlinks, bind mounts and per-sandbox mounts all make equal +paths unequal and unequal paths equal. A stored identifier alone is not proof either, +because copying the database copies the identifier. + +| Evidence | Verdict | +|---|---| +| a nonce written by one participant is readable by the other | proven | +| equal store id and equal device/inode | proven | +| equal store id, different inode, no nonce | unproven - a copy is possible | +| different store id | mismatch | + +`doctor --expect-store ` exits non-zero on a mismatch. An unproven result is never +reported as healthy. + +Status: planned in PR-A. + +## The service + +Two process roles. A **worker** is the existing bounded daemon: it runs a set number of ticks +or until a deadline and then exits, and cannot be constructed unbounded (I-64). A +**supervisor** owns the locks and launches successive workers, which is what carries an +assignment past any single process bound. + +| Command | Effect | +|---|---| +| `service status` | intent, liveness, ownership, store, conflicts, projects, observation health | +| `service enable` / `disable` | records the owner's intent; disable also stops a running service | +| `service start` | refuses if already running, if another store owns the operating scope, or if intent is disabled | +| `service stop` | refuses unless this installation owns the process; confirms both supervisor and worker exited | +| `service restart` | stop then start, preserving the recorded intent | +| `service run` | the foreground supervisor; what `start` launches | + +An update tool should read `service status`, act on `stop` or `restart`, and rely on one +guarantee: **a service that was disabled is never enabled as a side effect**. Intent lives in +`service.json` and absence means never configured, which is not enabled. + +Status: planned in PR-A. Today only the bounded foreground `daemon` command exists. + +### Ownership + +Single ownership is enforced at two levels, because either alone has a hole. + +- `/daemon.lock` prevents two daemons per state directory. +- A per-user scope registry, keyed by the App Server socket and located independently of the + state directory, prevents two daemons **with different state directories** from serving the + same App Server. A file lock in one state directory cannot see a rival that chose another. + +Both locks are held by the supervisor and inherited by its worker, so ownership persists +until every process of the service has exited. A dead supervisor with a live worker does not +release the scope. + +Termination is bound to a process handle rather than a pid, and identity includes the boot +id, the installation and the store, so a reused pid is never signalled by mistake. Where a +stable handle is unavailable, ownership reports `unverifiable` and stop refuses. + +Status: planned in PR-A. + +## One service, several projects + +A single supervisor and a single store serve every assignment on that host, user and App +Server, across projects and repositories. Each delivery is checked against its own +assignment before any transport call: the recipient must be that assignment's own parent for +a completion, or its own child for a revision. Membership in the authorized recipient list +alone is not sufficient, because two assignments may legitimately list the same recipient. +`service status` groups by project so one service carrying several projects is visible. + +Cancelling, pausing or archiving one assignment removes it from the loop. It does not stop +the shared service: only the supervisor's own bound, an explicit stop, or disabled intent +does that. + +Status: the per-assignment refusal is partly implemented (I-13, I-74); the direction check, +project grouping and shared-service guarantees are planned in PR-A. + +## Reading a stuck delivery + +`status` distinguishes the stages that all previously read as one withheld state: + +| Phase | Meaning | +|---|---| +| `awaiting_receipt` | the child has not produced a completion receipt yet | +| `parent_busy` | the parent is mid-turn; it is never interrupted | +| `settings_rejected` | the host would not confirm the authorized execution settings | +| `turn_accepted` | the transport started a turn | +| `awaiting_ack` | delivered, acknowledgement outstanding | +| `channel_closed` | the push channel itself is unavailable; stored, not woken | +| `superseded` | a newer generation or revision replaced this one | + +Each carries the most recent failed operation, its concrete error, the exact settings +difference where there is one, and the next retry time. + +Health is separate from liveness. A running process with a growing observation backlog is +reported as stalled: staged event age, when each current anchor was last successfully +polled, and the backlog per assignment are all exposed, and a live pid is never counted as +working. + +Status: planned in PR-B. + +## What a restart preserves + +Assignments, generations and anchors, queued and deferred deliveries, attempt history and +frozen message bytes, acknowledgements, verdicts and the Linear outbox all live in the store +and survive any process boundary. On start the relay reconciles unresolved attempts before +doing anything else, and reconciliation establishes what happened without sending: an +attempt whose response was lost stays uncertain until evidence resolves it, and is never +resent on the strength of elapsed time. + +Status: the durable state and reconciliation exist; supervised restart and lease recovery +are planned in PR-A. From 1e64a122309e172e5699952ce674d64d0dd2a4ea Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 13:09:46 +0900 Subject: [PATCH 02/29] Let every participant prove which store it is actually using The relay required all participants to share one database and gave them no way to check. A path string cannot answer it: symlinks and per-sandbox mounts make equal paths unequal and unequal paths equal. Worse, doctor built a Store before it could describe anything, so on the one host that needed a diagnosis, the diagnosis raised instead. State selection now reports itself. resolve_state_dir returns which of the four rules won and the exact value that won it, and doctor prints that alongside the resolved database, its device and inode, and read/write access measured rather than assumed. Identity is a store_id minted once, plus a store_challenge nonce. The nonce is what makes it evidence: copying relay.sqlite3 copies the identifier too, so an identifier match alone is graded unproven, a nonce written after the copy is proven, and a conflict is a mismatch that exits 2 while still printing the whole diagnosis. Unproven also exits 2, because a caller that asked whether this is the same store must not read exit 0 as yes. Services now builds every dependency on first use. That is what lets doctor answer for a missing or read-only state directory without creating or opening it, and the ack property wires its outbox before publishing the service so record_verdict cannot silently drop the obligation. doctor also reports where the transport ledger will land. bridge_adapter resolves it from CODEX_SESSION_RELAY_STATE independently, so --state alone splits the store from the ledger that carries send idempotency, and that split is now visible instead of silent. --- packages/codex-session-relay/README.md | 8 +- .../src/codex_session_relay/cli.py | 272 ++++++++++++--- .../src/codex_session_relay/store.py | 314 +++++++++++++++++- .../codex-session-relay/tests/test_cli.py | 138 ++++++++ .../codex-session-relay/tests/test_store.py | 186 ++++++++++- 5 files changed, 857 insertions(+), 61 deletions(-) diff --git a/packages/codex-session-relay/README.md b/packages/codex-session-relay/README.md index 5b22355..624ec28 100644 --- a/packages/codex-session-relay/README.md +++ b/packages/codex-session-relay/README.md @@ -56,13 +56,19 @@ activate a service on another host or establish that an earlier test process is Runtime state lives outside any repository, in `$XDG_STATE_HOME/codex-session-relay//` (mode 0700), holding `relay.sqlite3`, -`daemon.lock`, `daemon.pid` and `daemon.log`. `CODEX_SESSION_RELAY_STATE` overrides the directory. +`daemon.lock`, `daemon.pid` and `daemon.log`. Precedence, highest first: `--state`, +`CODEX_SESSION_RELAY_STATE`, `XDG_STATE_HOME`, then `~/.local/state`. **Every process must point at the same state directory.** The child emitting, the parent acknowledging and the daemon delivering share one store; a mismatched `--state` means they simply do not see each other. The endpoint hash is derived from the socket path, so passing the same `--socket` is enough. +`doctor` reports which rule won, the database it resolved to and the access this process really +has. To prove two participants share one store rather than two copies of one, write a nonce with +`store-challenge --write` and check it from the other side with `doctor --expect-nonce`; an +identifier alone is copied along with the file. See [docs/operations.md](docs/operations.md). + ## Authorized execution settings A send carries the settings the recipient task was actually created with. This is not defensive diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index cd0e051..451ce44 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -28,7 +28,9 @@ from .reconcile import Reconciler from .registry import Registry, contract_record as relationship_record, record_settings from .settings import REQUIRED as REQUIRED_SETTINGS -from .store import Store, state_dir +from .store import ( + Store, compare_store, nonce_lookup, probe, resolve_state_dir, state_dir, +) from .sync import SyncOutbox, render_progress_summary EXIT_OK, EXIT_REFUSED, EXIT_HOST, EXIT_USAGE = 0, 2, 3, 4 @@ -40,7 +42,7 @@ "ack", "ack-proof", "admit-turn", "assignment-show", "claim", "criteria-register", "criteria-show", "doctor", "emit", "generation-bind", "generation-open", "register", "relationship-resume", "relationship-status", "revision-head", "settings-record", - "settings-show", "show", "status", "verdict", + "settings-show", "show", "status", "store-challenge", "store-identity", "verdict", ) @@ -55,30 +57,106 @@ def __getattr__(self, name): class Services: + """Everything a command might need, built only when the command actually needs it. + + Nothing is constructed here on purpose. doctor exists to describe a host where the store + cannot be opened, and a Store built during construction opens the file O_RDWR, switches on + WAL and runs the schema script - so it would raise before doctor could report why. Each + dependency is cached after first use, so laziness never means two stores in one process. + """ + def __init__(self, args): - directory = Path(args.state) if args.state else state_dir(args.socket) - self.store = Store(directory / "relay.sqlite3") + self.selection = resolve_state_dir(getattr(args, "state", None), args.socket) self.clock = SystemClock() self.socket_path = args.socket self.adapter_requested = bool(args.socket) + self._store = None self._adapter = None - self.criteria = CriteriaService(self.store, self.clock) - self.registry = Registry(self.store, self.clock) - self.intake = ReceiptIntake( - self.store, self.registry, self.clock, - admission=AnchorOrExplicit(_LazyAdapter(self) if self.adapter_requested else None), - ) - self.delivery = DeliveryService(self.store, self.registry, self.intake, self.clock) - self.ack = AckService( - self.store, self.registry, self.intake, self.delivery, self.clock, - criteria=self.criteria, - ) - self.reconciler = Reconciler(self.store, self.registry, self.delivery, self.clock) - self.sync = SyncOutbox(self.store, self.clock) - self.ack.sync = self.sync - self.assignments = AssignmentView( - self.store, self.registry, self.clock, criteria=self.criteria - ) + self._criteria = None + self._registry = None + self._intake = None + self._delivery = None + self._ack = None + self._reconciler = None + self._sync = None + self._assignments = None + + @property + def state_directory(self): + return self.selection.path + + @property + def store(self): + if self._store is None: + self._store = Store(self.selection.db_path) + return self._store + + @property + def criteria(self): + if self._criteria is None: + self._criteria = CriteriaService(self.store, self.clock) + return self._criteria + + @property + def registry(self): + if self._registry is None: + self._registry = Registry(self.store, self.clock) + return self._registry + + @property + def intake(self): + if self._intake is None: + self._intake = ReceiptIntake( + self.store, self.registry, self.clock, + admission=AnchorOrExplicit( + _LazyAdapter(self) if self.adapter_requested else None + ), + ) + return self._intake + + @property + def delivery(self): + if self._delivery is None: + self._delivery = DeliveryService( + self.store, self.registry, self.intake, self.clock + ) + return self._delivery + + @property + def ack(self): + if self._ack is None: + service = AckService( + self.store, self.registry, self.intake, self.delivery, self.clock, + criteria=self.criteria, + ) + # Wired BEFORE the service is published. record_verdict skips its outbox + # obligation when sync is absent, so an ack handed out unwired would drop the + # obligation silently rather than fail. + service.sync = self.sync + self._ack = service + return self._ack + + @property + def reconciler(self): + if self._reconciler is None: + self._reconciler = Reconciler( + self.store, self.registry, self.delivery, self.clock + ) + return self._reconciler + + @property + def sync(self): + if self._sync is None: + self._sync = SyncOutbox(self.store, self.clock) + return self._sync + + @property + def assignments(self): + if self._assignments is None: + self._assignments = AssignmentView( + self.store, self.registry, self.clock, criteria=self.criteria + ) + return self._assignments @property def adapter(self): @@ -107,7 +185,11 @@ def close(self): except Exception: pass self._adapter = None - self.store.close() + # Only a store that was actually built is closed. Reading the property here would + # construct one during teardown, on the very host where constructing it fails. + if self._store is not None: + self._store.close() + self._store = None def _require_adapter(services): @@ -121,6 +203,20 @@ def __init__(self, message, code): self.code = code +class PayloadExit(Exception): + """A completed answer that is still a refusal. + + doctor has to print its whole diagnosis AND exit non-zero when it cannot prove two + participants share a store. A plain refusal would throw the diagnosis away, and a plain + return would let exit 0 be read as yes. + """ + + def __init__(self, payload, code): + super().__init__(payload.get("detail", "refused")) + self.payload = payload + self.code = code + + # --------------------------------------------------------------------- commands @@ -583,7 +679,7 @@ def cmd_daemon(services, args) -> dict: _require_adapter(services) from .daemon import RelayDaemon, SingleInstance - directory = Path(args.state) if args.state else state_dir(args.socket) + directory = services.state_directory daemon = RelayDaemon( services.store, services.registry, services.intake, services.delivery, services.ack, services.reconciler, services.adapter, clock=services.clock, @@ -597,24 +693,96 @@ def cmd_daemon(services, args) -> dict: return {"ticks": [report.as_dict() for report in reports]} -def cmd_doctor(services, args) -> dict: - import os +def cmd_store_identity(services, args) -> dict: + """One line each participant can emit, for a comparison to consume.""" + return {"stateSelection": services.selection.to_record(), "store": services.store.locate()} + +def cmd_store_challenge(services, args) -> dict: + """Write a nonce here, or look for one another participant wrote. + + This is the only evidence that survives a copied database: the identifier inside a copy is + identical, but a value written AFTER the copy exists in exactly one of the two files. + """ + if args.write: + return services.store.write_challenge(actor=args.actor or "cli") + if not args.read: + raise SystemExit2("store-challenge needs --write or --read ", EXIT_USAGE) + return services.store.read_challenge(args.read) + + +def _ledger_location(services) -> dict: + """Where the transport ledger will actually live, which --state does not move. + + bridge_adapter._build resolves it with state_dir(socket_path), reading the environment + only, so a run that overrides --state alone splits the relay store from the ledger that + carries send idempotency. Mirrors codex_thread_bridge.ledger.open_endpoint_ledger, which + cannot be called here because opening it is a side effect. + """ + import hashlib + + if not services.socket_path: + return {"configured": False, "directory": None, "path": None, "split": False} + directory = Path(state_dir(services.socket_path)).expanduser() + canonical = Path(services.socket_path).expanduser().absolute().resolve() + endpoint = hashlib.sha256(str(canonical).encode()).hexdigest()[:16] + split = directory.resolve() != services.selection.path.resolve() return { - "stateDirectory": str(services.store.path.parent), - "procAvailable": os.path.isdir("/proc/self/fd"), - "adapter": "bridge" if services.adapter_requested else "none (read-only, no --socket)", - "relationships": services.store.one( - "SELECT COUNT(*) AS c FROM relationships" - )["c"], - "openAttempts": len(Reconciler( - services.store, services.registry, services.delivery, services.clock - ).open_attempts()), - "actorReachability": _reachability(services), + "configured": True, "directory": str(directory), + "path": str(directory / f"operations-{endpoint}.sqlite3"), "split": split, } -def _reachability(services) -> dict: +def _contents(services, report) -> dict: + """Counts, but only when the store can actually be opened for them.""" + if not report["access"]["dbReadable"]: + return {"available": False, "relationships": None, "openAttempts": None, + "detail": "the database is not readable from this process"} + try: + relationships = services.store.one("SELECT COUNT(*) AS c FROM relationships")["c"] + open_attempts = len(services.reconciler.open_attempts()) + except Exception as error: # noqa: BLE001 - doctor reports, it does not fail + return {"available": False, "relationships": None, "openAttempts": None, + "detail": f"{type(error).__name__}: {error}"} + return {"available": True, "relationships": relationships, + "openAttempts": open_attempts, "detail": None} + + +def cmd_doctor(services, args) -> dict: + """What THIS process can actually do here, measured rather than assumed. + + Constructs no Store: probe() answers from stat, a read-only connection and a rolled-back + write transaction, so a missing, unreadable or read-only state directory is an answer + instead of the failure that would otherwise replace it. + """ + import os + + report = probe(services.selection) + report["procAvailable"] = os.path.isdir("/proc/self/fd") + report["adapter"] = ( + "bridge" if services.adapter_requested else "none (read-only, no --socket)" + ) + report["ledger"] = _ledger_location(services) + report["actorReachability"] = _reachability(services, report) + report["contents"] = _contents(services, report) + + nonce = nonce_lookup(services.selection, args.expect_nonce) if args.expect_nonce else None + report["nonce"] = nonce + comparison = compare_store( + report["store"], expect_store=args.expect_store, expect_inode=args.expect_inode, + nonce=nonce, + ) + asked = any((args.expect_store, args.expect_inode, args.expect_nonce)) + report.update(comparison) + if asked and comparison["sameStore"] != "proven": + # A caller that asked whether this is the same store and got no proof must not read + # exit 0 as yes. Unproven is refused for the same reason a mismatch is: the criterion + # is that a different database is never reported as healthy. + raise PayloadExit(report, EXIT_REFUSED) + return report + + +def _reachability(services, report) -> dict: """What THIS process can actually do here, measured rather than assumed. A workspace-write task cannot write the default state directory and cannot connect to the @@ -624,15 +792,10 @@ def _reachability(services) -> dict: thread, which is why doctor can answer even where the bridge itself could not load. """ import socket - import tempfile - directory = services.store.path.parent - writable, detail = True, None - try: - with tempfile.NamedTemporaryFile(dir=directory, prefix=".reach-"): - pass - except OSError as error: - writable, detail = False, f"{type(error).__name__}: {error}" + # Reuses the probe's measurement rather than repeating it, so one command cannot report + # two different answers about the same directory. + access = report["access"] connect = "not configured" if services.socket_path: @@ -647,8 +810,8 @@ def _reachability(services) -> dict: probe.close() return { - "stateDirectoryWritable": writable, - "stateDirectoryDetail": detail, + "stateDirectoryWritable": access["directoryWritable"], + "stateDirectoryDetail": access["detail"], "socketConfigured": bool(services.socket_path), "socketConnect": connect, "offlineCommands": list(OFFLINE_COMMANDS), @@ -916,7 +1079,19 @@ def build_parser() -> argparse.ArgumentParser: daemon.add_argument("--deadline", type=float) daemon.set_defaults(handler=cmd_daemon) - subparsers.add_parser("doctor").set_defaults(handler=cmd_doctor) + doctor = subparsers.add_parser("doctor") + doctor.add_argument("--expect-store", help="the store id another participant reported") + doctor.add_argument("--expect-inode", help="the device:inode another participant reported") + doctor.add_argument("--expect-nonce", help="a nonce another participant wrote here") + doctor.set_defaults(handler=cmd_doctor) + + subparsers.add_parser("store-identity").set_defaults(handler=cmd_store_identity) + + challenge = subparsers.add_parser("store-challenge") + challenge.add_argument("--write", action="store_true") + challenge.add_argument("--read") + challenge.add_argument("--actor") + challenge.set_defaults(handler=cmd_store_challenge) return parser @@ -939,6 +1114,9 @@ def main(argv=None) -> int: except SystemExit2 as error: print(json.dumps({"error": "usage", "detail": str(error)}, indent=2)) return error.code + except PayloadExit as error: + print(json.dumps(error.payload, indent=2, default=str)) + return error.code except Exception as error: print(json.dumps({ "error": "host", "detail": f"{type(error).__name__}: {error}" diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 6cc0996..c80a60f 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -13,7 +13,11 @@ import json import os import sqlite3 +import tempfile +import uuid from contextlib import contextmanager +from dataclasses import dataclass +from datetime import datetime, timezone from pathlib import Path SCHEMA_VERSION = 1 @@ -409,6 +413,12 @@ detail TEXT ); +CREATE TABLE IF NOT EXISTS store_challenge ( + nonce TEXT PRIMARY KEY, + written_by TEXT NOT NULL, + written_at TEXT NOT NULL +); + CREATE INDEX IF NOT EXISTS deliveries_state ON deliveries (state, next_eligible_at); CREATE INDEX IF NOT EXISTS attempts_open ON attempts (internal_state); CREATE INDEX IF NOT EXISTS events_relationship ON events (relationship_id, execution_generation); @@ -420,19 +430,76 @@ """ -def state_dir(socket_path: str | None = None) -> Path: - """Runtime state lives outside every repository, mirroring the bridge's convention.""" - override = os.environ.get("CODEX_SESSION_RELAY_STATE") +STATE_ENV = "CODEX_SESSION_RELAY_STATE" +PRECEDENCE = ("flag", "env", "xdg", "home") + + +def _now_iso() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") + + +def socket_scope(socket_path) -> str: + if not socket_path: + return "default" + return hashlib.sha256(str(Path(socket_path).expanduser()).encode()).hexdigest()[:16] + + +@dataclass(frozen=True) +class StateSelection: + """Which rule chose the state directory, and the exact value that won. + + Four rules can decide where the store lives, and a participant that cannot say which one + applied to it cannot be compared with another participant. The reason travels with the + path for that reason alone. + """ + + path: Path + source: str + detail: str + socket_scope: str | None + + @property + def db_path(self) -> Path: + return self.path / "relay.sqlite3" + + def to_record(self) -> dict: + return { + "path": str(self.path), "dbPath": str(self.db_path), "source": self.source, + "detail": self.detail, "socketScope": self.socket_scope, + "precedence": list(PRECEDENCE), + } + + +def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: + """Runtime state lives outside every repository, mirroring the bridge's convention. + + Highest precedence first: an explicit --state, then the environment override, then + XDG_STATE_HOME, then the home default. Only the last two carry a socket scope; an + explicit directory is used exactly as given, because the caller has already decided + which participants share it. + """ + if explicit: + return StateSelection( + Path(explicit).expanduser().absolute(), "flag", f"--state {explicit}", None + ) + override = os.environ.get(STATE_ENV) if override: - return Path(override).expanduser() - base = Path( - os.environ.get("XDG_STATE_HOME") or (Path.home() / ".local" / "state") - ).expanduser() - root = base / "codex-session-relay" - if socket_path: - endpoint = hashlib.sha256(str(Path(socket_path).expanduser()).encode()).hexdigest()[:16] - return root / endpoint - return root / "default" + return StateSelection( + Path(override).expanduser().absolute(), "env", f"{STATE_ENV}={override}", None + ) + xdg = os.environ.get("XDG_STATE_HOME") + if xdg: + base, source, detail = Path(xdg).expanduser(), "xdg", f"XDG_STATE_HOME={xdg}" + else: + base, source = Path.home() / ".local" / "state", "home" + detail = f"default under {Path.home() / '.local' / 'state'}" + scope = socket_scope(socket_path) + return StateSelection((base / "codex-session-relay" / scope).absolute(), source, detail, scope) + + +def state_dir(socket_path: str | None = None) -> Path: + """The directory the environment alone would choose. Kept for callers that have no flag.""" + return resolve_state_dir(None, socket_path).path class Store: @@ -451,9 +518,71 @@ def __init__(self, path): self.db.execute( "INSERT OR IGNORE INTO schema_meta VALUES ('version', ?)", (str(SCHEMA_VERSION),) ) + # Minted once and never rewritten, so reopening a store - or restarting the daemon on + # it - cannot look like a new store. It is deliberately NOT proof on its own: copying + # the file copies the identifier too, which is what store_challenge exists for. + self.db.execute( + "INSERT OR IGNORE INTO schema_meta VALUES ('store_id', ?)", (uuid.uuid4().hex,) + ) + self.db.execute( + "INSERT OR IGNORE INTO schema_meta VALUES ('store_created_at', ?)", (_now_iso(),) + ) # Tests set this to prove a transition rolls back; nothing in production assigns it. self.fault_hook = None + # ------------------------------------------------------------------ identity + + def meta(self, key: str): + row = self.one("SELECT value FROM schema_meta WHERE key = ?", (key,)) + return row["value"] if row is not None else None + + @property + def identity(self): + return self.meta("store_id") + + def locate(self) -> dict: + """The physical facts a same-store comparison needs, not just the path we were given. + + The device and inode come from the RESOLVED path, so a symlink or a bind mount that + reaches the same bytes compares equal while two genuinely different files do not. + """ + try: + real = self.path.resolve() + info = os.stat(real) + device, inode, real_path = info.st_dev, info.st_ino, str(real) + except OSError: + device = inode = real_path = None + return { + "exists": True, + "storeId": self.identity, + "createdAt": self.meta("store_created_at"), + "dbPath": str(self.path), + "realPath": real_path, + "device": device, + "inode": inode, + "schemaVersion": self.meta("version"), + } + + def write_challenge(self, *, actor: str) -> dict: + """Leave a value only a participant reading THIS file can find.""" + nonce = uuid.uuid4().hex + written_at = _now_iso() + with self.transaction() as db: + db.execute( + "INSERT INTO store_challenge (nonce, written_by, written_at) VALUES (?,?,?)", + (nonce, actor, written_at), + ) + return {"nonce": nonce, "writtenBy": actor, "writtenAt": written_at} + + def read_challenge(self, nonce: str) -> dict: + row = self.one("SELECT * FROM store_challenge WHERE nonce = ?", (nonce,)) + if row is None: + return {"nonce": nonce, "found": False, "writtenBy": None, "writtenAt": None} + return { + "nonce": nonce, "found": True, "writtenBy": row["written_by"], + "writtenAt": row["written_at"], + } + @contextmanager def transaction(self): """BEGIN IMMEDIATE, then commit or roll back. Never a partial record.""" @@ -485,3 +614,164 @@ def all(self, sql: str, params=()): def close(self) -> None: self.db.close() + + +PROVEN, UNPROVEN, MISMATCH = "proven", "unproven", "mismatch" + + +def probe(selection: StateSelection) -> dict: + """Describe the selected state WITHOUT constructing a Store. + + Store.__init__ creates the directory, opens the file O_RDWR, switches on WAL and runs the + schema script. Constructing one in order to find out whether that works fails before it + can report anything, on exactly the host that needed the report. Every step below owns its + error and becomes a field instead, so this function never raises. + """ + directory, db_path = selection.path, selection.db_path + notes = [] + access = { + "directoryExists": False, "directoryReadable": False, "directoryWritable": False, + "dbExists": False, "dbReadable": False, "dbWritable": False, "detail": None, + } + store = { + "exists": False, "storeId": None, "createdAt": None, "dbPath": str(db_path), + "realPath": None, "device": None, "inode": None, "schemaVersion": None, + } + + try: + access["directoryExists"] = directory.is_dir() + except OSError as error: + notes.append(f"directory stat failed: {type(error).__name__}: {error}") + if access["directoryExists"]: + access["directoryReadable"] = os.access(directory, os.R_OK | os.X_OK) + # os.access answers for the REAL uid and can disagree with the kernel under a + # privileged runner or an unusual mount, so writability is measured by writing. + try: + with tempfile.NamedTemporaryFile(dir=directory, prefix=".probe-"): + pass + access["directoryWritable"] = True + except OSError as error: + notes.append(f"directory write failed: {type(error).__name__}: {error}") + + try: + info = os.stat(db_path) + access["dbExists"] = store["exists"] = True + store["realPath"] = str(db_path.resolve()) + store["device"], store["inode"] = info.st_dev, info.st_ino + except OSError as error: + if access["directoryExists"]: + notes.append(f"database stat failed: {type(error).__name__}: {error}") + + if access["dbExists"]: + try: + connection = sqlite3.connect(f"{db_path.as_uri()}?mode=ro", uri=True, timeout=5) + try: + connection.row_factory = sqlite3.Row + access["dbReadable"] = True + for key, field in ( + ("store_id", "storeId"), ("store_created_at", "createdAt"), + ("version", "schemaVersion"), + ): + row = connection.execute( + "SELECT value FROM schema_meta WHERE key = ?", (key,) + ).fetchone() + # A store written before identity existed has no row here. Absence is + # reported as absence and never defaulted, because a default could later + # compare equal to another store's and be read as proof. + store[field] = row["value"] if row is not None else None + finally: + connection.close() + except (OSError, sqlite3.Error, TypeError, ValueError) as error: + notes.append(f"database read failed: {type(error).__name__}: {error}") + + try: + connection = sqlite3.connect( + f"{db_path.as_uri()}?mode=rw", uri=True, timeout=5, isolation_level=None + ) + try: + connection.execute("BEGIN IMMEDIATE") + connection.execute("ROLLBACK") + # Acquiring a write transaction is evidence that this process can write NOW. + # It is not a promise that a later commit succeeds; a full disk still fails. + access["dbWritable"] = True + finally: + connection.close() + except (OSError, sqlite3.Error) as error: + notes.append(f"database write probe failed: {type(error).__name__}: {error}") + + access["detail"] = "; ".join(notes) or None + return {"stateSelection": selection.to_record(), "store": store, "access": access} + + +def nonce_lookup(selection: StateSelection, nonce: str) -> dict: + """Look for a challenge nonce read-only, so a comparison never writes to the store.""" + db_path = selection.db_path + try: + connection = sqlite3.connect(f"{db_path.as_uri()}?mode=ro", uri=True, timeout=5) + except (OSError, sqlite3.Error) as error: + return {"nonce": nonce, "found": False, "readable": False, + "detail": f"{type(error).__name__}: {error}"} + try: + connection.row_factory = sqlite3.Row + row = connection.execute( + "SELECT written_by, written_at FROM store_challenge WHERE nonce = ?", (nonce,) + ).fetchone() + except sqlite3.Error as error: + return {"nonce": nonce, "found": False, "readable": True, + "detail": f"{type(error).__name__}: {error}"} + finally: + connection.close() + if row is None: + return {"nonce": nonce, "found": False, "readable": True, "detail": None} + return {"nonce": nonce, "found": True, "readable": True, "detail": None, + "writtenBy": row["written_by"], "writtenAt": row["written_at"]} + + +def compare_store(store: dict, *, expect_store=None, expect_inode=None, nonce=None) -> dict: + """Grade the evidence that this participant and another share ONE store. + + Conflicting evidence is decided before agreeing evidence, so an easier comparison that + happened to succeed can never talk a mismatch down. Absence is never agreement: a store + that cannot state its identity is unproven, not proven, because the criterion is that a + different database must never be reported as healthy. + """ + reasons = [] + if expect_store is not None: + if store.get("storeId") is None: + reasons.append((UNPROVEN, "this store states no identity, so it cannot be compared")) + elif store["storeId"] != expect_store: + reasons.append((MISMATCH, f"store id {store['storeId']} is not {expect_store}")) + else: + reasons.append((None, "store id matches")) + if expect_inode is not None: + want = str(expect_inode).split(":") + here = (store.get("device"), store.get("inode")) + if len(want) != 2 or None in here: + reasons.append((UNPROVEN, "physical identity is not comparable here")) + elif (str(here[0]), str(here[1])) != (want[0], want[1]): + reasons.append(( + MISMATCH, f"device:inode {here[0]}:{here[1]} is not {expect_inode}", + )) + else: + reasons.append((PROVEN, "same device and inode")) + if nonce is not None: + if nonce.get("found"): + reasons.append((PROVEN, "a nonce written by another participant is readable here")) + else: + reasons.append((MISMATCH, "a nonce written by another participant is not here")) + + if not reasons: + return {"sameStore": UNPROVEN, "detail": "no expectation was supplied to compare against"} + for verdict in (MISMATCH, UNPROVEN, PROVEN): + matched = [detail for grade, detail in reasons if grade == verdict] + if matched: + if verdict is PROVEN and any(g == UNPROVEN for g, _ in reasons): + continue + return {"sameStore": verdict, "detail": "; ".join(matched)} + # Every expectation agreed, but none of them was physical or live evidence: an identical + # store id alone is satisfied by a copy of the file, so this is not proof. + return { + "sameStore": UNPROVEN, + "detail": "only the store id was compared, and copying a database copies it too;" + " supply --expect-inode or a nonce for proof", + } diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index 45eb4d2..ae91823 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -1,9 +1,12 @@ """The command surface, driven end to end with no host and no socket.""" +import argparse import json import os +import shutil import subprocess import sys +import tempfile import unittest from .support import CHILD, DISPATCH_TURN, HOST, ISSUE, PARENT, RelayTestCase @@ -233,3 +236,138 @@ def test_an_incomplete_record_is_refused_with_a_machine_readable_reason(self): ) self.assertEqual(refused["reason"], "settings_incomplete") self.assertIn("environments", refused["detail"]) + + +class Diagnosis(unittest.TestCase): + """doctor has to answer ON the host it is describing, including a broken one.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-doctor-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.home = os.path.join(self.tmp, "home") + os.makedirs(self.home) + + def cli(self, *args, state=None, socket=None, expect=0, env=None): + environment = dict( + os.environ, PYTHONPATH=os.path.join(REPO, "src"), HOME=self.home, + ) + environment.pop("CODEX_SESSION_RELAY_STATE", None) + environment.pop("XDG_STATE_HOME", None) + environment.update(env or {}) + completed = subprocess.run( + [sys.executable, "-m", "codex_session_relay.cli", + *(["--state", state] if state else []), + *(["--socket", socket] if socket else []), *args], + capture_output=True, text=True, env=environment, timeout=60, + ) + self.assertEqual( + completed.returncode, expect, + f"exit {completed.returncode}: {completed.stdout}{completed.stderr}", + ) + return json.loads(completed.stdout) + + def test_doctor_answers_for_a_state_directory_that_does_not_exist_yet(self): + absent = os.path.join(self.tmp, "absent") + report = self.cli("doctor", state=absent) + self.assertEqual(report["stateSelection"]["source"], "flag") + self.assertFalse(report["access"]["directoryExists"]) + self.assertFalse(report["contents"]["available"]) + # The old doctor built a Store first, which created the directory it was asked about. + self.assertFalse(os.path.exists(absent)) + + def test_doctor_names_the_rule_that_chose_the_directory(self): + chosen = os.path.join(self.tmp, "chosen") + by_env = self.cli("doctor", env={"CODEX_SESSION_RELAY_STATE": chosen}) + self.assertEqual(by_env["stateSelection"]["source"], "env") + self.assertEqual(by_env["stateSelection"]["path"], chosen) + by_flag = self.cli("doctor", state=chosen, env={ + "CODEX_SESSION_RELAY_STATE": os.path.join(self.tmp, "ignored"), + }) + self.assertEqual(by_flag["stateSelection"]["source"], "flag") + self.assertEqual(by_flag["stateSelection"]["path"], chosen) + + def test_a_different_store_is_refused_rather_than_reported_healthy(self): + a, b = os.path.join(self.tmp, "a"), os.path.join(self.tmp, "b") + mine = self.cli("store-identity", state=a)["store"] + theirs = self.cli("store-identity", state=b)["store"] + self.assertNotEqual(mine["storeId"], theirs["storeId"]) + refused = self.cli( + "doctor", "--expect-store", mine["storeId"], state=b, expect=2, + ) + self.assertEqual(refused["sameStore"], "mismatch") + # The whole diagnosis survives the refusal; it is not replaced by an error envelope. + self.assertIn("stateSelection", refused) + self.assertIn("access", refused) + + def test_a_nonce_proves_one_store_and_disproves_a_copy(self): + a, b = os.path.join(self.tmp, "a"), os.path.join(self.tmp, "b") + mine = self.cli("store-identity", state=a)["store"] + os.makedirs(b, exist_ok=True) + for suffix in ("", "-wal", "-shm"): + source = os.path.join(a, f"relay.sqlite3{suffix}") + if os.path.exists(source): + shutil.copy(source, os.path.join(b, f"relay.sqlite3{suffix}")) + nonce = self.cli("store-challenge", "--write", "--actor", "parent", state=a)["nonce"] + proven = self.cli( + "doctor", "--expect-store", mine["storeId"], "--expect-nonce", nonce, state=a, + ) + self.assertEqual(proven["sameStore"], "proven") + copied = self.cli( + "doctor", "--expect-store", mine["storeId"], "--expect-nonce", nonce, state=b, + expect=2, + ) + self.assertEqual(copied["sameStore"], "mismatch") + # Identifier alone cannot separate them, which is why it is graded unproven. + weak = self.cli("doctor", "--expect-store", mine["storeId"], state=b, expect=2) + self.assertEqual(weak["sameStore"], "unproven") + + def test_doctor_reports_that_state_and_the_transport_ledger_have_split(self): + state = os.path.join(self.tmp, "state") + socket = os.path.join(self.tmp, "app.sock") + split = self.cli("doctor", state=state, socket=socket) + self.assertTrue(split["ledger"]["configured"]) + # --state moved the store; the adapter resolves its ledger from the environment. + self.assertTrue(split["ledger"]["split"]) + together = self.cli( + "doctor", state=state, socket=socket, + env={"CODEX_SESSION_RELAY_STATE": state}, + ) + self.assertFalse(together["ledger"]["split"]) + + +class LazyServices(unittest.TestCase): + """Building dependencies on demand must not drop the wiring __init__ used to do.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-lazy-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + + def services(self): + from codex_session_relay.cli import Services + + built = Services(argparse.Namespace(state=self.tmp, socket=None)) + self.addCleanup(built.close) + return built + + def test_the_first_ack_is_already_wired_to_the_outbox(self): + built = self.services() + # Touching ack FIRST is the regression: record_verdict skips its outbox obligation + # when sync is absent, so an unwired ack would lose it silently. + self.assertIsNotNone(built.ack.sync) + self.assertIs(built.ack.sync, built.sync) + + def test_every_dependency_shares_one_store_and_is_built_once(self): + built = self.services() + self.assertIs(built.registry.store, built.store) + self.assertIs(built.delivery.store, built.store) + self.assertIs(built.ack.store, built.store) + self.assertIs(built.registry, built.registry) + self.assertIs(built.delivery, built.delivery) + + def test_closing_without_ever_using_the_store_creates_nothing(self): + from codex_session_relay.cli import Services + + empty = os.path.join(self.tmp, "untouched") + built = Services(argparse.Namespace(state=empty, socket=None)) + built.close() + self.assertFalse(os.path.exists(empty)) diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 34c9519..b460d8c 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -3,7 +3,14 @@ import os import unittest -from codex_session_relay.store import SCHEMA_VERSION, Store, state_dir +import shutil +import tempfile +from pathlib import Path +from unittest import mock + +from codex_session_relay.store import ( + SCHEMA_VERSION, Store, compare_store, nonce_lookup, probe, resolve_state_dir, state_dir, +) from .support import RelayTestCase @@ -105,5 +112,182 @@ def test_a_socket_gets_its_own_endpoint_directory(self): self.assertIn("codex-session-relay", str(first)) +class Precedence(unittest.TestCase): + """Four rules can choose the directory, and a participant has to be able to say which one.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-precedence-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + # Never read the real home or the real XDG state during a test. + self.home = os.path.join(self.tmp, "home") + os.makedirs(self.home) + patch = mock.patch.dict(os.environ, {"HOME": self.home}, clear=False) + patch.start() + self.addCleanup(patch.stop) + for name in ("CODEX_SESSION_RELAY_STATE", "XDG_STATE_HOME"): + os.environ.pop(name, None) + + def test_each_rule_wins_in_order_and_says_so(self): + home = resolve_state_dir(None, "/run/x.sock") + self.assertEqual(home.source, "home") + self.assertTrue(str(home.path).startswith(self.home)) + + os.environ["XDG_STATE_HOME"] = os.path.join(self.tmp, "xdg") + xdg = resolve_state_dir(None, "/run/x.sock") + self.assertEqual(xdg.source, "xdg") + self.assertIn("XDG_STATE_HOME=", xdg.detail) + self.assertEqual(xdg.socket_scope, home.socket_scope) + + os.environ["CODEX_SESSION_RELAY_STATE"] = os.path.join(self.tmp, "env") + env = resolve_state_dir(None, "/run/x.sock") + self.assertEqual(env.source, "env") + self.assertIn("CODEX_SESSION_RELAY_STATE=", env.detail) + + flag = resolve_state_dir(os.path.join(self.tmp, "flag"), "/run/x.sock") + self.assertEqual(flag.source, "flag") + self.assertIn("--state ", flag.detail) + self.assertEqual(flag.db_path.name, "relay.sqlite3") + + def test_a_relative_flag_resolves_to_the_same_store_as_its_absolute_form(self): + target = os.path.join(self.tmp, "rel") + os.makedirs(target) + Store(Path(target) / "relay.sqlite3").close() + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = probe(resolve_state_dir("rel")) + finally: + os.chdir(here) + absolute = probe(resolve_state_dir(target)) + self.assertEqual(relative["store"]["storeId"], absolute["store"]["storeId"]) + self.assertEqual(relative["store"]["inode"], absolute["store"]["inode"]) + + +class Identity(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-identity-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.a = os.path.join(self.tmp, "a") + os.makedirs(self.a) + self.store = Store(Path(self.a) / "relay.sqlite3") + self.addCleanup(self.store.close) + + def copy_store(self, into): + """Copy the store the way a person copying a state directory would. + + The sidecars matter: in WAL mode a recent write still lives in relay.sqlite3-wal, so + copying the main file alone produces a store that has lost it. That would make this + test pass for the wrong reason - an empty copy has no identifier to collide with. + """ + os.makedirs(into, exist_ok=True) + for suffix in ("", "-wal", "-shm"): + source = os.path.join(self.a, f"relay.sqlite3{suffix}") + if os.path.exists(source): + shutil.copy(source, os.path.join(into, f"relay.sqlite3{suffix}")) + return into + + def test_identity_is_minted_once_and_survives_reopening(self): + first = self.store.locate() + self.assertTrue(first["storeId"]) + self.store.close() + again = Store(Path(self.a) / "relay.sqlite3") + self.addCleanup(again.close) + self.assertEqual(again.identity, first["storeId"]) + self.assertEqual(again.meta("store_created_at"), first["createdAt"]) + + def test_a_copy_keeps_the_identifier_and_is_not_the_same_store(self): + """The case a stored uuid gets wrong: copying the file copies the identifier.""" + mine = self.store.locate() + b = self.copy_store(os.path.join(self.tmp, "b")) + theirs = probe(resolve_state_dir(b))["store"] + self.assertEqual(theirs["storeId"], mine["storeId"]) + self.assertNotEqual(theirs["inode"], mine["inode"]) + + # The identifier alone is satisfied by the copy, so it is never proof on its own. + self.assertEqual( + compare_store(theirs, expect_store=mine["storeId"])["sameStore"], "unproven", + ) + self.assertEqual( + compare_store( + theirs, expect_store=mine["storeId"], + expect_inode=f"{mine['device']}:{mine['inode']}", + )["sameStore"], + "mismatch", + ) + + def test_a_nonce_written_after_the_copy_separates_them(self): + b = self.copy_store(os.path.join(self.tmp, "b")) + self.assertEqual( + probe(resolve_state_dir(b))["store"]["storeId"], self.store.identity, + "the copy must be a real copy, or the nonce result proves nothing", + ) + # Written AFTER the copy on purpose: a nonce that already existed would have been + # copied too, and finding it would prove nothing. + written = self.store.write_challenge(actor="parent") + here = nonce_lookup(resolve_state_dir(self.a), written["nonce"]) + there = nonce_lookup(resolve_state_dir(b), written["nonce"]) + self.assertTrue(here["found"]) + self.assertFalse(there["found"]) + self.assertEqual(compare_store(self.store.locate(), nonce=here)["sameStore"], "proven") + self.assertEqual( + compare_store(probe(resolve_state_dir(b))["store"], nonce=there)["sameStore"], + "mismatch", + ) + + def test_a_symlinked_directory_is_the_same_store(self): + alias = os.path.join(self.tmp, "alias") + os.symlink(self.a, alias) + mine = self.store.locate() + through_alias = probe(resolve_state_dir(alias))["store"] + self.assertNotEqual(through_alias["dbPath"], mine["dbPath"]) + self.assertEqual( + compare_store( + through_alias, expect_store=mine["storeId"], + expect_inode=f"{mine['device']}:{mine['inode']}", + )["sameStore"], + "proven", + ) + + def test_a_store_with_no_identity_is_never_proven_equal(self): + """Absence must not become agreement; an old store predates the identity rows.""" + self.assertEqual( + compare_store({"storeId": None}, expect_store="whatever")["sameStore"], "unproven", + ) + + +class Probe(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-probe-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + + def test_a_missing_state_directory_is_an_answer_and_is_not_created(self): + absent = os.path.join(self.tmp, "nothing-here") + report = probe(resolve_state_dir(absent)) + self.assertFalse(report["access"]["directoryExists"]) + self.assertFalse(report["store"]["exists"]) + self.assertFalse(os.path.exists(absent), "probe must not create what it describes") + + def test_reported_writability_matches_what_this_process_can_really_do(self): + """Asserted against a measured attempt, because a privileged runner ignores mode bits.""" + locked = os.path.join(self.tmp, "locked") + os.makedirs(locked) + Store(Path(locked) / "relay.sqlite3").close() + os.chmod(locked, 0o500) + self.addCleanup(os.chmod, locked, 0o700) + try: + probe_file = os.path.join(locked, ".really-writable") + with open(probe_file, "w", encoding="utf-8"): + pass + os.unlink(probe_file) + really_writable = True + except OSError: + really_writable = False + report = probe(resolve_state_dir(locked)) + self.assertEqual(report["access"]["directoryWritable"], really_writable) + self.assertTrue(report["access"]["dbReadable"]) + if not really_writable: + self.assertIsNotNone(report["access"]["detail"]) + + if __name__ == "__main__": unittest.main() From 035834ca31e93239dd9a3793c319a962d87c261e Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 13:10:11 +0900 Subject: [PATCH 03/29] Mark the state-selection and store-identity contract as shipped Those two sections described behaviour that now exists, and a status line that lags the code is worse than no status line. --- packages/codex-session-relay/docs/operations.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/packages/codex-session-relay/docs/operations.md b/packages/codex-session-relay/docs/operations.md index d2508e0..ee304a2 100644 --- a/packages/codex-session-relay/docs/operations.md +++ b/packages/codex-session-relay/docs/operations.md @@ -30,8 +30,8 @@ Setting `--state` alone is not enough for an isolated run. The bridge adapter re transport ledger from `CODEX_SESSION_RELAY_STATE` independently, so a run that overrides only the flag splits the relay store from the ledger that carries send idempotency. Set both. -Status: planned in PR-A. Today `state_dir()` implements the precedence but nothing reports -the selection. +Status: implemented. `resolve_state_dir` returns the winning rule and `doctor` reports it with +the resolved database and the measured access. ## Proving two participants share one store @@ -49,7 +49,9 @@ because copying the database copies the identifier. `doctor --expect-store ` exits non-zero on a mismatch. An unproven result is never reported as healthy. -Status: planned in PR-A. +Status: implemented. Unproven also exits non-zero, because a caller that asked whether this is +the same store must not read exit 0 as yes. Each participant runs the check in its own sandbox; +one invocation cannot establish another participant's access. ## The service From 98cc235803d53f7af2587d0e5a1239e1098f6383 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 13:39:15 +0900 Subject: [PATCH 04/29] Give the relay an owner, and make it refuse to guess who that is A flock on /daemon.lock proves one daemon per state directory and nothing more. A second installation that picked a different state directory for the same App Server took its own lock and believed it was alone, so two services could serve one host quite happily. Stopping one was no better: the code had a pid, and a pid is not a process. Verifying /proc/ and then calling kill leaves a window where that pid belongs to something else. Ownership now lives in a scope registry keyed by the App Server socket, in a directory neither installation chose: the passwd home of the effective uid, read from the passwd database rather than $HOME so it cannot move with the launch environment. Liveness is the lock, never a recorded pid, so a dead record does not release a scope. The record outlives the process on purpose, because a stopped registration on a different store is still a duplicate registration and a live-only scan would never see it. An environment override would have put the hole back one level down, so it does not silently become the authority: it marks the registry isolated, namespaces the key, and is refused unless --allow-isolated-scope says it was deliberate. Termination goes through a pidfd opened before the identity check and held until the process is gone, so the thing verified and the thing signalled are the same thing. Identity is boot id, start ticks, installation and store together. Where pidfd is unavailable this refuses rather than falling back to kill; a signal that cannot be aimed is worse than none. A zombie counts as gone, or the grace period would wait out a process that had already exited and then report it as still running. Intent is durable and separate from liveness. service.json absent means never configured, which is not enabled; start never writes it; restart re-reads it after stopping so an owner who disables the service mid-restart does not get a replacement. The standalone daemon command takes the same scope claim, or two bounded runs with different state directories would still serve one socket unseen. It does not require the managed service's enable intent, because an explicit bounded run is an operator action. --- .../codex-session-relay/docs/operations.md | 8 +- .../src/codex_session_relay/cli.py | 111 ++- .../src/codex_session_relay/service.py | 680 ++++++++++++++++++ .../tests/test_daemon_cadence.py | 16 + .../codex-session-relay/tests/test_service.py | 265 +++++++ 5 files changed, 1063 insertions(+), 17 deletions(-) create mode 100644 packages/codex-session-relay/src/codex_session_relay/service.py create mode 100644 packages/codex-session-relay/tests/test_service.py diff --git a/packages/codex-session-relay/docs/operations.md b/packages/codex-session-relay/docs/operations.md index ee304a2..652236c 100644 --- a/packages/codex-session-relay/docs/operations.md +++ b/packages/codex-session-relay/docs/operations.md @@ -73,7 +73,9 @@ An update tool should read `service status`, act on `stop` or `restart`, and rel guarantee: **a service that was disabled is never enabled as a side effect**. Intent lives in `service.json` and absence means never configured, which is not enabled. -Status: planned in PR-A. Today only the bounded foreground `daemon` command exists. +Status: the commands above are implemented. `service run` currently serves ONE bounded +segment and exits; supervision across successive worker processes, which is what carries an +assignment past a single process lifetime, is still planned in PR-A. ### Ownership @@ -92,7 +94,9 @@ Termination is bound to a process handle rather than a pid, and identity include id, the installation and the store, so a reused pid is never signalled by mistake. Where a stable handle is unavailable, ownership reports `unverifiable` and stop refuses. -Status: planned in PR-A. +Status: implemented, except that the inherited-descriptor arrangement only matters once the +supervisor launches workers, which is still planned. Today the single service process holds +both locks itself. ## One service, several projects diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 451ce44..4b8613b 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -37,12 +37,16 @@ # Which commands need to reach the App Server, and which do not. Reported by doctor, because a # caller should learn this from one command instead of from a failure halfway through. -HOST_REQUIRED_COMMANDS = ("daemon", "deliver", "reconcile", "recover", "verify-acks") +HOST_REQUIRED_COMMANDS = ( + "daemon", "deliver", "reconcile", "recover", "service run", "service start", + "service restart", "verify-acks", +) OFFLINE_COMMANDS = ( "ack", "ack-proof", "admit-turn", "assignment-show", "claim", "criteria-register", "criteria-show", "doctor", "emit", "generation-bind", "generation-open", "register", "relationship-resume", "relationship-status", "revision-head", "settings-record", "settings-show", "show", "status", "store-challenge", "store-identity", "verdict", + "service status", "service enable", "service disable", "service stop", ) @@ -677,20 +681,10 @@ def wait(seconds): def cmd_daemon(services, args) -> dict: _require_adapter(services) - from .daemon import RelayDaemon, SingleInstance - - directory = services.state_directory - daemon = RelayDaemon( - services.store, services.registry, services.intake, services.delivery, services.ack, - services.reconciler, services.adapter, clock=services.clock, - ) - deadline = services.clock.now() + args.deadline if args.deadline else None - with SingleInstance(directory): - reports = daemon.run( - max_ticks=args.max_ticks, deadline=deadline, - sleep=_scheduler_wait(services.clock, deadline), - ) - return {"ticks": [report.as_dict() for report in reports]} + # A bounded daemon run is an explicit operator action, so it does NOT require the managed + # service's enable intent - but it does take the same scope claim, or two standalone runs + # with different state directories could serve one App Server and never see each other. + return _run_bounded(services, _service_for(services), args, require_intent=False) def cmd_store_identity(services, args) -> dict: @@ -782,6 +776,77 @@ def cmd_doctor(services, args) -> dict: return report +def _service_for(services): + """Built from the probe, so status stays an offline command that constructs no Store.""" + from .service import RelayService + + return RelayService( + services.selection, socket_path=services.socket_path, + store_id=probe(services.selection)["store"]["storeId"], + ) + + +def _refuse_unless_ok(payload: dict) -> dict: + if payload.get("ok"): + return payload + raise PayloadExit(payload, EXIT_REFUSED) + + +def _run_bounded(services, service, args, *, require_intent: bool) -> dict: + """Hold ownership for exactly as long as this process serves, then let it go. + + The daemon is constructed INSIDE the claim so a run that loses the race never opens a + transport connection it is about to abandon. + """ + from .daemon import RelayDaemon + from .service import ServiceRefused, owned_service + + deadline = services.clock.now() + args.deadline if args.deadline else None + allow_isolated = getattr(args, "allow_isolated_scope", False) + try: + with owned_service( + service, allow_isolated=allow_isolated, require_intent=require_intent, + ) as record: + daemon = RelayDaemon( + services.store, services.registry, services.intake, services.delivery, + services.ack, services.reconciler, services.adapter, clock=services.clock, + ) + reports = daemon.run( + max_ticks=args.max_ticks, deadline=deadline, + sleep=_scheduler_wait(services.clock, deadline), + ) + except ServiceRefused as refusal: + raise PayloadExit( + {"ok": False, "reason": refusal.reason, "detail": refusal.detail}, EXIT_REFUSED, + ) from refusal + return {"ok": True, "reason": None, "pid": record["pid"], + "ticks": [report.as_dict() for report in reports]} + + +def cmd_service(services, args) -> dict: + service = _service_for(services) + action = args.service_command + if action == "status": + return service.status() + if action == "enable": + return service.enable(actor=args.actor or "cli") + if action == "disable": + return _refuse_unless_ok(service.disable(actor=args.actor or "cli")) + if action == "stop": + return _refuse_unless_ok(service.stop(actor=args.actor or "cli")) + if action in ("start", "restart"): + _require_adapter(services) + call = service.start if action == "start" else service.restart + return _refuse_unless_ok(call( + allow_isolated=args.allow_isolated_scope, max_ticks=args.max_ticks, + deadline=args.deadline, actor=args.actor or "cli", + )) + if action == "run": + _require_adapter(services) + return _run_bounded(services, service, args, require_intent=True) + raise SystemExit2(f"unknown service action {action!r}", EXIT_USAGE) + + def _reachability(services, report) -> dict: """What THIS process can actually do here, measured rather than assumed. @@ -1077,8 +1142,24 @@ def build_parser() -> argparse.ArgumentParser: daemon = subparsers.add_parser("daemon") daemon.add_argument("--max-ticks", type=int) daemon.add_argument("--deadline", type=float) + daemon.add_argument("--allow-isolated-scope", action="store_true") daemon.set_defaults(handler=cmd_daemon) + service = subparsers.add_parser("service") + actions = service.add_subparsers(dest="service_command", required=True) + for name in ("status", "enable", "disable", "stop"): + offline = actions.add_parser(name) + offline.add_argument("--actor") + for name in ("start", "restart", "run"): + hosted = actions.add_parser(name) + hosted.add_argument("--actor") + hosted.add_argument("--allow-isolated-scope", action="store_true") + hosted.add_argument("--max-ticks", type=int) + # One bounded segment by default. Supervision ACROSS segments is a separate change; + # an unbounded run is not constructible here and must not become one by omission. + hosted.add_argument("--deadline", type=float, default=3600.0) + service.set_defaults(handler=cmd_service) + doctor = subparsers.add_parser("doctor") doctor.add_argument("--expect-store", help="the store id another participant reported") doctor.add_argument("--expect-inode", help="the device:inode another participant reported") diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py new file mode 100644 index 0000000..49cae0a --- /dev/null +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -0,0 +1,680 @@ +"""Who owns the daemon, and which process may be told to stop. + +Two questions the state directory cannot answer on its own. + +A flock on /daemon.lock proves one daemon per state directory. It says nothing about a +second installation that chose a DIFFERENT state directory for the same App Server, and those +two would each hold their own lock and each believe they were alone. Single ownership +therefore lives in a scope registry keyed by the operating scope - the socket - in a location +neither installation chose. + +And a pid is not a process. Verifying /proc/ and then calling os.kill leaves a window in +which that process exits and its number is reused, so the signal lands somewhere else. Every +signal here goes through a pidfd opened BEFORE the identity check and kept open until the +process is gone, so the thing verified and the thing signalled are the same thing. +""" + +import errno +import fcntl +import hashlib +import json +import os +import pwd +import signal +import time +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path + +SCOPE_ENV = "CODEX_SESSION_RELAY_SCOPE_DIR" +PRODUCTION, ISOLATED = "production", "isolated" + +DAEMON_LOCK = "daemon.lock" +DAEMON_RECORD = "daemon.json" +SERVICE_INTENT = "service.json" +DAEMON_LOG = "daemon.log" + +OURS, FOREIGN, UNVERIFIABLE, NONE = "ours", "foreign", "unverifiable", "none" + + +def _now() -> str: + from .store import _now_iso + + return _now_iso() + + +def boot_id(): + """Without it a pid recorded before a reboot can match a live unrelated pid today.""" + try: + return Path("/proc/sys/kernel/random/boot_id").read_text(encoding="utf-8").strip() + except OSError: + return None + + +def _stat_fields(pid: int): + """The fields of /proc//stat after the comm. + + The comm can itself contain spaces and brackets, so the split starts after the LAST + closing bracket rather than at the second whitespace-separated field. + """ + try: + raw = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8") + return raw[raw.rindex(")") + 2:].split() + except (OSError, ValueError): + return None + + +def start_ticks(pid: int): + """Field 22. Pids repeat; a pid together with its start time does not.""" + fields = _stat_fields(pid) + try: + return int(fields[19]) + except (TypeError, ValueError, IndexError): + return None + + +def process_state(pid: int): + """Field 3: R, S, D, Z, T and friends.""" + fields = _stat_fields(pid) + return fields[0] if fields else None + + +def production_scope_root() -> Path: + """The one ownership authority a real launch may use. + + Read from the passwd database rather than $HOME, so it does not move with the launch + environment - which is the whole point, since the environment is what distinguishes two + installations. Deliberately not under ~/.codex, which this package never writes. + """ + return Path(pwd.getpwuid(os.geteuid()).pw_dir) / ".codex-session-relay" / "scopes" + + +def resolve_scope_root(): + """(root, authority). An override never silently becomes the authority. + + An environment override alone would put the hole back one level down: two real launches + could set it to different directories, or one could set it and the other not, and both + would claim the same socket while passing every ownership check. So an override marks the + registry ISOLATED, and the commands that start a daemon refuse an isolated authority + unless the caller passes --allow-isolated-scope and means it. + """ + override = os.environ.get(SCOPE_ENV) + if override: + return Path(override).expanduser().absolute(), ISOLATED + return production_scope_root(), PRODUCTION + + +class ScopeUnavailable(Exception): + """The ownership record cannot be reached, so ownership cannot be established.""" + + +@dataclass +class ScopeRegistry: + """One claim per operating scope, in a place neither state directory chose.""" + + root: Path + authority: str = PRODUCTION + _handle: object = field(default=None, repr=False) + + def key(self, socket_path) -> str: + canonical = str(Path(socket_path).expanduser().absolute()) + digest = hashlib.sha256(canonical.encode()).hexdigest()[:16] + if self.authority == PRODUCTION: + return digest + # Namespaced so an isolated record can never be read as the production one. This is + # NOT socket isolation: two deliberately isolated services still share the socket. + salt = hashlib.sha256(str(self.root).encode()).hexdigest()[:8] + return f"isolated-{salt}-{digest}" + + def prepare(self) -> Path: + """Create and validate the directory, or refuse. There is no second location.""" + try: + self.root.mkdir(mode=0o700, parents=True, exist_ok=True) + info = os.stat(self.root) + except OSError as error: + raise ScopeUnavailable(f"{self.root}: {type(error).__name__}: {error}") from error + if info.st_uid != os.geteuid(): + raise ScopeUnavailable(f"{self.root} is owned by uid {info.st_uid}, not this user") + if info.st_mode & 0o022: + raise ScopeUnavailable(f"{self.root} is group or world writable") + return self.root + + def record_path(self, socket_path) -> Path: + return self.root / f"{self.key(socket_path)}.json" + + def read(self, socket_path): + try: + return json.loads(self.record_path(socket_path).read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + def claim(self, socket_path, record: dict): + """Take the scope lock and write the record, or report who already holds it. + + Liveness is the LOCK, never a recorded pid. A supervisor that dies leaving a worker + alive still owns the scope through the descriptor the worker inherited, and a dead + pid in the record is not permission to take it. + """ + self.prepare() + path = self.root / f"{self.key(socket_path)}.lock" + handle = open(path, "a+") + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as error: + handle.close() + if error.errno not in (errno.EACCES, errno.EAGAIN): + raise ScopeUnavailable(f"{path}: {type(error).__name__}: {error}") from error + return {"ok": False, "reason": "scope_owned_by_other_store", + "held_by": self.read(socket_path), "scopeKey": self.key(socket_path)} + self._handle = handle + payload = dict(record, scopeKey=self.key(socket_path), scopeAuthority=self.authority, + socketPath=str(socket_path), registeredAt=_now()) + self.record_path(socket_path).write_text(json.dumps(payload, indent=2), encoding="utf-8") + return {"ok": True, "reason": None, "record": payload, "lockFd": handle.fileno()} + + def release(self, socket_path, *, shared: bool = False) -> None: + """Keep the record, drop the claim. + + The record outlives the process on purpose: a STOPPED duplicate registration on a + different store is still a duplicate, and live-only scanning could not see it. + """ + existing = self.read(socket_path) + if existing is not None: + existing.update(pid=None, workerPid=None, startedAt=None, releasedAt=_now()) + try: + self.record_path(socket_path).write_text( + json.dumps(existing, indent=2), encoding="utf-8") + except OSError: + pass + handle, self._handle = self._handle, None + if handle is not None: + # Closing releases it when this is the last descriptor. Never LOCK_UN: unlocking + # through any duplicate would release a lock a supervised worker still relies on. + if not shared: + try: + fcntl.flock(handle, fcntl.LOCK_UN) + except OSError: + pass + handle.close() + + def conflicts(self, socket_path, *, store_id=None, state_dir=None) -> list: + """Every record for this scope that names a different store, alive or not.""" + out = [] + try: + entries = sorted(self.root.glob("*.json")) + except OSError: + return out + mine = self.key(socket_path) + for entry in entries: + try: + found = json.loads(entry.read_text(encoding="utf-8")) + except (OSError, ValueError): + continue + if found.get("scopeKey") != mine: + continue + same_store = store_id is not None and found.get("storeId") == store_id + same_dir = state_dir is not None and found.get("stateDir") == str(state_dir) + if same_store and same_dir: + continue + out.append({ + "stateDir": found.get("stateDir"), "storeId": found.get("storeId"), + "installationId": found.get("installationId"), "pid": found.get("pid"), + "live": _is_live(found), "reason": "same_scope_different_store", + }) + return out + + +def _is_live(record) -> bool: + pid, recorded_boot = record.get("pid"), record.get("bootId") + if not pid: + return False + if recorded_boot is not None and recorded_boot != boot_id(): + # A pid from a previous boot cannot be this record's process. + return False + if process_state(int(pid)) in (None, "Z"): + return False + ticks = start_ticks(int(pid)) + if ticks is None: + return False + recorded = record.get("startTicks") + return recorded is None or int(recorded) == ticks + + +def installation_id(state_dir) -> str: + """This checkout plus this state directory. A second of either is a second install.""" + package = Path(__file__).resolve().parent + material = f"{package}\0{Path(state_dir).expanduser().absolute().resolve()}" + return hashlib.sha256(material.encode()).hexdigest()[:16] + + +class ProcessHandle: + """A handle to a specific process, not to a number. + + Opened FIRST, verified second, signalled third, all through the same descriptor, and kept + open until the process is gone - including across the grace period before SIGKILL. That + ordering is the point: a pid verified and then signalled can belong to a different process + by the time the signal lands. Holding the descriptor also reserves the pid, so /proc stays + truthful for us while we wait. + + Where pidfd is unavailable this refuses rather than falling back to os.kill. A signal that + cannot be aimed is worse than no signal. + """ + + def __init__(self, pid): + self.pid = int(pid) + self.fd = None + self.detail = None + self.already_gone = False + opener = getattr(os, "pidfd_open", None) + if opener is None or getattr(signal, "pidfd_send_signal", None) is None: + self.detail = "this build has no pidfd support, so a signal cannot be aimed" + return + try: + self.fd = opener(self.pid) + except ProcessLookupError: + self.already_gone = True + self.detail = "the process is already gone" + except OSError as error: + self.detail = f"{type(error).__name__}: {error}" + + @property + def usable(self) -> bool: + return self.fd is not None + + def alive(self) -> bool: + """A zombie is not alive. + + Its /proc entry survives until its parent reaps it, and holding a pidfd keeps the pid + reserved, so a liveness check that only asks whether /proc/ exists would wait out + the whole grace period and then report a terminated process as still running. + """ + state = process_state(self.pid) + return state is not None and state != "Z" + + def send(self, sig) -> bool: + if self.fd is None: + return False + try: + signal.pidfd_send_signal(self.fd, sig) + except ProcessLookupError: + return True + except OSError as error: + self.detail = f"{type(error).__name__}: {error}" + return False + return True + + def close(self) -> None: + if self.fd is not None: + os.close(self.fd) + self.fd = None + + +@dataclass +class ServiceIntent: + """What the owner asked for, independent of whether anything is running. + + Absent means never configured, which is NOT enabled. Nothing here writes enabled=True as a + side effect of starting; only an explicit enable does. + """ + + path: Path + + def read(self) -> dict: + try: + data = json.loads(self.path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {"enabled": False, "configured": False, "changedAt": None, "changedBy": None} + return {"enabled": bool(data.get("enabled")), "configured": True, + "changedAt": data.get("changedAt"), "changedBy": data.get("changedBy")} + + def write(self, *, enabled: bool, actor: str) -> dict: + self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + payload = {"enabled": bool(enabled), "changedAt": _now(), "changedBy": actor} + self.path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + return dict(payload, configured=True) + + +class RelayService: + """Start, stop and describe the daemon that belongs to THIS installation.""" + + def __init__(self, selection, *, socket_path=None, scope=None, store_id=None): + self.selection = selection + self.socket_path = socket_path + self.store_id = store_id + if scope is None: + root, authority = resolve_scope_root() + scope = ScopeRegistry(root, authority) + self.scope = scope + self.intent = ServiceIntent(selection.path / SERVICE_INTENT) + self.installation_id = installation_id(selection.path) + + @property + def record_path(self) -> Path: + return self.selection.path / DAEMON_RECORD + + def record(self): + try: + return json.loads(self.record_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + def write_record(self, payload: dict) -> dict: + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + self.record_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + return payload + + def new_record(self, *, pid, token=None, worker_pid=None) -> dict: + return { + "pid": pid, "startTicks": start_ticks(pid), "bootId": boot_id(), + "workerPid": worker_pid, "token": token, + "storeId": self.store_id, "installationId": self.installation_id, + "stateDir": str(self.selection.path), "socketPath": self.socket_path, + "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), + "startedAt": _now(), "restarts": 0, "consecutiveFailures": 0, "lastExit": None, + "nextRestartAt": None, + } + + def ownership(self, record=None): + """ours | foreign | unverifiable | none, decided through a real process handle.""" + record = self.record() if record is None else record + if not record or not record.get("pid"): + return NONE, None, "no daemon record" + handle = ProcessHandle(record["pid"]) + if handle.already_gone: + return NONE, handle, "the recorded process is gone" + if not handle.usable: + return UNVERIFIABLE, handle, handle.detail + mismatches = [] + if record.get("bootId") not in (None, boot_id()): + mismatches.append("recorded before a different boot") + ticks = start_ticks(record["pid"]) + if record.get("startTicks") is not None and ticks != record["startTicks"]: + mismatches.append("the pid was reused by a different process") + if record.get("installationId") != self.installation_id: + mismatches.append("another installation owns it") + if self.store_id is not None and record.get("storeId") not in (None, self.store_id): + mismatches.append("it is using a different store") + if mismatches: + return FOREIGN, handle, "; ".join(mismatches) + return OURS, handle, None + + def lock_is_held(self) -> bool: + """Probed by trying to take it: the lock is the only honest liveness signal.""" + path = self.selection.path / DAEMON_LOCK + if not path.exists(): + return False + try: + handle = open(path, "a+") + except OSError: + return False + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + return True + else: + fcntl.flock(handle, fcntl.LOCK_UN) + return False + finally: + handle.close() + + # -------------------------------------------------------------- operations + + def authority_check(self, *, allow_isolated: bool): + if self.scope.authority == ISOLATED and not allow_isolated: + return { + "ok": False, "reason": "isolated_scope_not_allowed", + "detail": f"{SCOPE_ENV} is set, which moves the ownership record away from the" + " one authority every launch shares. Pass --allow-isolated-scope if" + " that is deliberate; single ownership is not enforced across" + " differently isolated roots.", + } + return {"ok": True, "reason": None} + + def enable(self, *, actor: str = "cli") -> dict: + return {"ok": True, "reason": None, + "intent": self.intent.write(enabled=True, actor=actor)} + + def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: + written = self.intent.write(enabled=False, actor=actor) + stopped = self.stop(actor=actor, timeout=timeout) + # A stop that had nothing to stop is not a failure; the intent is what disable owns. + failed = not stopped["ok"] and stopped["reason"] != "not_running" + return {"ok": not failed, "reason": stopped["reason"] if failed else None, + "intent": written, "stop": stopped} + + def status(self) -> dict: + record = self.record() + owner, handle, detail = self.ownership(record) + if handle is not None: + handle.close() + intent = self.intent.read() + held = self.lock_is_held() + return { + "enabled": intent["enabled"], "intentConfigured": intent["configured"], + "intentChangedAt": intent["changedAt"], "intentChangedBy": intent["changedBy"], + # Liveness is the lock, not the recorded pid: a supervisor that died leaving a + # worker alive still holds it through the descriptor the worker inherited. + "running": held, + "ownership": owner, "ownershipDetail": detail, + "pid": (record or {}).get("pid"), "workerPid": (record or {}).get("workerPid"), + "startedAt": (record or {}).get("startedAt"), + "storeId": (record or {}).get("storeId"), "socketPath": self.socket_path, + "installationId": self.installation_id, + "stateDirectory": str(self.selection.path), + "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), + "lock": "held" if held else "free", + "staleRecord": bool(record and record.get("pid") and not held), + "restarts": (record or {}).get("restarts"), + "consecutiveFailures": (record or {}).get("consecutiveFailures"), + "lastExit": (record or {}).get("lastExit"), + "nextRestartAt": (record or {}).get("nextRestartAt"), + "conflicts": self.conflicts(), + } + + def conflicts(self) -> list: + if not self.socket_path: + return [] + try: + return self.scope.conflicts( + self.socket_path, store_id=self.store_id, state_dir=self.selection.path, + ) + except ScopeUnavailable: + return [] + + def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) -> dict: + """Signal only a process this installation owns, through a handle to that process.""" + record = self.record() + owner, handle, detail = self.ownership(record) + try: + if owner == NONE: + return {"ok": False, "reason": "not_running", "detail": detail, + "supervisor": "gone", "worker": "gone"} + if owner == FOREIGN: + return {"ok": False, "reason": "not_ours", "detail": detail, + "supervisor": "untouched", "worker": "untouched"} + if owner == UNVERIFIABLE: + # Refusing is the point. Signalling on a pid match alone is how an unrelated + # process gets killed after its number is reused. + return {"ok": False, "reason": "ownership_unverifiable", "detail": detail, + "supervisor": "untouched", "worker": "untouched"} + outcome = self._terminate(handle, timeout=timeout, grace=grace) + finally: + if handle is not None: + handle.close() + worker = self._stop_worker(record, timeout=timeout, grace=grace) + if outcome == "exited" and record is not None: + self.write_record(dict(record, pid=None, workerPid=None, + stoppedAt=_now(), stoppedBy=actor)) + ok = outcome == "exited" and worker in ("exited", "gone") + return {"ok": ok, "reason": None if ok else "did_not_exit", + "detail": None, "supervisor": outcome, "worker": worker} + + def _terminate(self, handle, *, timeout, grace) -> str: + if not handle.send(signal.SIGTERM): + return "signal_refused" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if not handle.alive(): + return "exited" + time.sleep(grace) + handle.send(signal.SIGKILL) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if not handle.alive(): + return "exited" + time.sleep(grace) + return "still_running" + + def _stop_worker(self, record, *, timeout, grace) -> str: + """Reach the worker directly when the supervisor is already gone.""" + pid = (record or {}).get("workerPid") + if not pid: + return "gone" + handle = ProcessHandle(pid) + try: + if handle.already_gone or not handle.usable: + return "gone" if handle.already_gone else "unverifiable" + ticks = (record or {}).get("workerStartTicks") + if ticks is not None and start_ticks(pid) != ticks: + return "gone" + return self._terminate(handle, timeout=timeout, grace=grace) + finally: + handle.close() + + def restart(self, *, allow_isolated: bool = False, launcher=None, **kw) -> dict: + """Preserves the recorded intent. A disabled service is refused, never switched on.""" + intent = self.intent.read() + if not intent["enabled"]: + return {"ok": False, "reason": "service_disabled", "intent": intent, + "detail": "restart never enables a service the owner turned off"} + stopped = self.stop(**{k: v for k, v in kw.items() if k in ("actor", "timeout")}) + if not stopped["ok"] and stopped["reason"] not in ("not_running",): + return {"ok": False, "reason": stopped["reason"], "stop": stopped} + # Re-read AFTER stopping: an owner who disabled the service while it was being + # stopped must not get a replacement launched from the value we cached above. + if not self.intent.read()["enabled"]: + return {"ok": False, "reason": "service_disabled", "stop": stopped, + "intent": self.intent.read(), + "detail": "intent changed to disabled while the service was stopping"} + started = self.start(allow_isolated=allow_isolated, launcher=launcher, **kw) + return {"ok": started["ok"], "reason": started["reason"], "stop": stopped, + "start": started} + + def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cli", + max_ticks=None, deadline=None, timeout: float = 20.0, poll: float = 0.05) -> dict: + """Preconditions here; ownership in the child. + + The daemon lock and the scope claim are taken by the process that will HOLD them, not + by this one: a parent that claimed and then exited would hand the child an unowned + scope. So start checks what it can cheaply, launches, and then reports what the child + actually managed to do. + """ + gate = self.authority_check(allow_isolated=allow_isolated) + if not gate["ok"]: + return gate + intent = self.intent.read() + if not intent["enabled"]: + return {"ok": False, "reason": "service_disabled", "intent": intent, + "detail": "start never enables a service; enable it explicitly first"} + if self.lock_is_held(): + return {"ok": False, "reason": "already_running", "status": self.status()} + live = [conflict for conflict in self.conflicts() if conflict["live"]] + if live: + return {"ok": False, "reason": "scope_owned_by_other_store", "conflicts": live} + + before = (self.record() or {}).get("pid") + child = (launcher or self.default_launcher)( + self, allow_isolated=allow_isolated, max_ticks=max_ticks, deadline=deadline, + ) + deadline_at = time.monotonic() + timeout + while time.monotonic() < deadline_at: + record = self.record() or {} + if record.get("pid") and record.get("pid") != before and self.lock_is_held(): + return {"ok": True, "reason": None, "pid": record["pid"], + "scopeAuthority": self.scope.authority, + "scopeRoot": str(self.scope.root), "status": self.status()} + if getattr(child, "poll", lambda: None)() is not None: + return {"ok": False, "reason": "child_exited", + "exitCode": child.returncode, "log": self._log_tail()} + time.sleep(poll) + return {"ok": False, "reason": "did_not_report", "log": self._log_tail()} + + def _log_tail(self, lines: int = 20) -> str: + try: + return "".join( + (self.selection.path / DAEMON_LOG).read_text(encoding="utf-8").splitlines(True) + [-lines:] + ) + except OSError: + return "" + + def default_launcher(self, service, *, allow_isolated, max_ticks, deadline): + import subprocess + import sys + + argv = [sys.executable, "-m", "codex_session_relay.cli", + "--state", str(self.selection.path)] + if self.socket_path: + argv += ["--socket", str(self.socket_path)] + argv += ["service", "run"] + if allow_isolated: + argv.append("--allow-isolated-scope") + if max_ticks is not None: + argv += ["--max-ticks", str(max_ticks)] + if deadline is not None: + argv += ["--deadline", str(deadline)] + environment = dict(os.environ) + if self.scope.authority == ISOLATED: + # The ALREADY RESOLVED absolute root, so a relative override cannot resolve + # differently in the child, and the child cannot land in another registry. + environment[SCOPE_ENV] = str(self.scope.root) + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: + return subprocess.Popen( + argv, stdout=log, stderr=log, stdin=subprocess.DEVNULL, + start_new_session=True, env=environment, + ) + + +@contextmanager +def owned_service(service: RelayService, *, allow_isolated: bool = False, token=None, + require_intent: bool = True): + """Hold the daemon lock and the scope claim for as long as this process serves. + + Both are released together on the way out, and the scope RECORD is deliberately kept: a + stopped registration on a different store is still a duplicate registration, and a scan + that only looked at live records could not see it. + """ + from .daemon import SingleInstance + + gate = service.authority_check(allow_isolated=allow_isolated) + if not gate["ok"]: + raise ServiceRefused(gate["reason"], gate["detail"]) + if require_intent and not service.intent.read()["enabled"]: + raise ServiceRefused( + "service_disabled", + "this service is not enabled; running it would ignore the owner's intent", + ) + with SingleInstance(service.selection.path): + claim = {"ok": True, "reason": None} + if service.socket_path: + claim = service.scope.claim(service.socket_path, service.new_record( + pid=os.getpid(), token=token, + )) + if not claim["ok"]: + raise ServiceRefused(claim["reason"], json.dumps(claim.get("held_by"))) + record = service.write_record(service.new_record(pid=os.getpid(), token=token)) + try: + yield record + finally: + if service.socket_path: + service.scope.release(service.socket_path) + current = service.record() or record + service.write_record(dict(current, pid=None, workerPid=None, stoppedAt=_now())) + + +class ServiceRefused(Exception): + def __init__(self, reason, detail=None): + super().__init__(detail or reason) + self.reason = reason + self.detail = detail diff --git a/packages/codex-session-relay/tests/test_daemon_cadence.py b/packages/codex-session-relay/tests/test_daemon_cadence.py index 648aece..6da997c 100644 --- a/packages/codex-session-relay/tests/test_daemon_cadence.py +++ b/packages/codex-session-relay/tests/test_daemon_cadence.py @@ -5,6 +5,8 @@ deadline-only run busy-spun for its whole duration. These tests pin both halves. """ +import os +import shutil import tempfile import time import unittest @@ -21,6 +23,10 @@ def __init__(self, state, socket=None, max_ticks=None, deadline=None): self.socket = socket self.max_ticks = max_ticks self.deadline = deadline + # A bounded daemon run now takes the same scope claim a managed service does, so a + # test has to say which registry it is claiming in. Without this it would write an + # ownership record under the real home. + self.allow_isolated_scope = True class SchedulerWait(unittest.TestCase): @@ -50,6 +56,16 @@ class CliSuppliesTheCadence(unittest.TestCase): def _services(self, **kwargs): directory = tempfile.mkdtemp(prefix="relay-cadence-") + self.addCleanup(shutil.rmtree, directory, ignore_errors=True) + scopes = tempfile.mkdtemp(prefix="relay-cadence-scopes-") + self.addCleanup(shutil.rmtree, scopes, ignore_errors=True) + previous = os.environ.get("CODEX_SESSION_RELAY_SCOPE_DIR") + os.environ["CODEX_SESSION_RELAY_SCOPE_DIR"] = scopes + self.addCleanup( + lambda: os.environ.__setitem__("CODEX_SESSION_RELAY_SCOPE_DIR", previous) + if previous is not None + else os.environ.pop("CODEX_SESSION_RELAY_SCOPE_DIR", None) + ) args = _Args(directory, socket="/nonexistent-for-this-test", **kwargs) services = Services(args) # The lazy adapter returns whatever is already set, so no bridge is built and no socket diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py new file mode 100644 index 0000000..9059aed --- /dev/null +++ b/packages/codex-session-relay/tests/test_service.py @@ -0,0 +1,265 @@ +"""Who owns the daemon, and who is allowed to stop it. + +These tests use real child processes on purpose. A mocked flock proves that the code called +flock; only a second process trying to start proves that the first one is actually excluded. +""" + +import json +import os +import shutil +import subprocess +import sys +import tempfile +import time +import unittest +from pathlib import Path +from unittest import mock + +from codex_session_relay import service as service_module +from codex_session_relay.service import ( + ISOLATED, PRODUCTION, ProcessHandle, RelayService, ScopeRegistry, ServiceRefused, + installation_id, owned_service, production_scope_root, resolve_scope_root, +) +from codex_session_relay.store import Store, resolve_state_dir + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +SOCKET = "/nonexistent-app-server.sock" + +# A child that takes the claim and holds it until told to stop, so exclusion is observed +# rather than asserted against a double. +HOLDER = """ +import os, sys, time +sys.path.insert(0, {src!r}) +from codex_session_relay.service import RelayService, ScopeRegistry, owned_service +from codex_session_relay.store import resolve_state_dir +service = RelayService( + resolve_state_dir({state!r}), socket_path={socket!r}, + scope=ScopeRegistry(__import__("pathlib").Path({scopes!r}), "isolated"), + store_id={store_id!r}, +) +with owned_service(service, allow_isolated=True, require_intent=False) as record: + # The record file IS the handshake. A pipe would add buffering and lifetime questions + # that have nothing to do with what this test is about. + while True: + time.sleep(0.05) +""" + + +class ServiceTestCase(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-service-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.scopes = os.path.join(self.tmp, "scopes") + self.children = [] + self.addCleanup(self._reap) + + def _reap(self): + for child in self.children: + if child.poll() is None: + child.kill() + try: + child.wait(timeout=10) + except subprocess.TimeoutExpired: # pragma: no cover - a hung child is a failure + self.fail(f"child {child.pid} did not exit") + + def service(self, name="a", *, socket=SOCKET, scopes=None): + state = os.path.join(self.tmp, name) + os.makedirs(state, exist_ok=True) + store = Store(Path(state) / "relay.sqlite3") + store_id = store.identity + store.close() + return RelayService( + resolve_state_dir(state), socket_path=socket, + scope=ScopeRegistry(Path(scopes or self.scopes), ISOLATED), store_id=store_id, + ) + + def holder(self, service): + """Start a child that really holds the lock and the claim, and wait until it does.""" + program = HOLDER.format( + src=os.path.join(REPO, "src"), state=str(service.selection.path), + socket=service.socket_path, scopes=str(service.scope.root), + store_id=service.store_id, + ) + child = subprocess.Popen( + [sys.executable, "-c", program], stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, text=True, + ) + self.children.append(child) + deadline = time.monotonic() + 20.0 + while time.monotonic() < deadline: + record = service.record() + if record and record.get("pid") and service.lock_is_held(): + return child, record["pid"] + if child.poll() is not None: + self.fail(f"holder exited {child.returncode}: {child.stderr.read()}") + time.sleep(0.02) + self.fail("the holder never took the claim") + + +class Ownership(ServiceTestCase): + def test_a_second_start_is_refused_and_the_first_is_untouched(self): + service = self.service("a") + child, pid = self.holder(service) + service.intent.write(enabled=True, actor="test") + refused = service.start(allow_isolated=True, launcher=self._never_launch) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "already_running") + self.assertIsNone(child.poll(), "the running service must not be disturbed") + + def _never_launch(self, *_args, **_kwargs): # pragma: no cover - reaching it is the failure + self.fail("start launched a second process while one was already running") + + def test_a_different_state_directory_cannot_serve_the_same_socket(self): + """The hole a per-state-directory lock cannot close.""" + first = self.service("a") + self.holder(first) + second = self.service("b") + self.assertNotEqual(second.store_id, first.store_id) + second.intent.write(enabled=True, actor="test") + refused = second.start(allow_isolated=True, launcher=self._never_launch) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "scope_owned_by_other_store") + self.assertEqual(refused["conflicts"][0]["storeId"], first.store_id) + + def test_a_stopped_registration_on_another_store_is_still_a_conflict(self): + """Live-only scanning would miss this one entirely.""" + first = self.service("a") + child, _pid = self.holder(first) + child.terminate() + child.wait(timeout=10) + second = self.service("b") + conflicts = second.conflicts() + self.assertEqual([c["storeId"] for c in conflicts], [first.store_id]) + self.assertFalse(conflicts[0]["live"]) + + def test_stop_refuses_a_process_another_installation_owns(self): + service = self.service("a") + child, pid = self.holder(service) + record = service.record() + service.write_record(dict(record, installationId="someone-else")) + refused = service.stop() + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("another installation", refused["detail"]) + self.assertIsNone(child.poll(), "a foreign process must not be signalled") + + def test_stop_refuses_a_recycled_pid(self): + service = self.service("a") + child, pid = self.holder(service) + record = service.record() + service.write_record(dict(record, startTicks=(record["startTicks"] or 0) + 1)) + refused = service.stop() + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("reused", refused["detail"]) + self.assertIsNone(child.poll()) + + def test_stop_refuses_when_the_process_cannot_be_aimed_at(self): + service = self.service("a") + child, pid = self.holder(service) + with mock.patch.object(os, "pidfd_open", side_effect=OSError("no pidfd here")): + refused = service.stop() + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIsNone(child.poll(), "refusing is the point; never fall back to os.kill") + + def test_stop_terminates_a_process_this_installation_owns(self): + service = self.service("a") + child, pid = self.holder(service) + stopped = service.stop() + self.assertTrue(stopped["ok"], stopped) + self.assertEqual(stopped["supervisor"], "exited") + child.wait(timeout=10) + self.assertIsNone(service.record()["pid"]) + self.assertFalse(service.lock_is_held()) + + def test_a_stale_record_does_not_block_a_fresh_start(self): + service = self.service("a") + child, pid = self.holder(service) + child.terminate() + child.wait(timeout=10) + status = service.status() + self.assertFalse(status["running"]) + self.assertEqual(status["ownership"], "none") + self.assertFalse(service.lock_is_held()) + + +class Intent(ServiceTestCase): + def test_start_never_enables_a_service_that_was_never_configured(self): + service = self.service("a") + refused = service.start(allow_isolated=True, launcher=self._fail) + self.assertEqual(refused["reason"], "service_disabled") + self.assertFalse(service.intent.read()["enabled"]) + self.assertFalse(service.intent.read()["configured"]) + self.assertFalse((service.selection.path / "service.json").exists()) + + def _fail(self, *_a, **_k): # pragma: no cover + self.fail("a disabled service must not be launched") + + def test_restart_refuses_a_disabled_service_and_leaves_it_disabled(self): + service = self.service("a") + service.enable(actor="test") + service.disable(actor="test") + refused = service.restart(allow_isolated=True, launcher=self._fail) + self.assertEqual(refused["reason"], "service_disabled") + self.assertFalse(service.intent.read()["enabled"]) + + def test_a_disable_during_restart_stops_the_replacement(self): + """The window between restart stopping the old process and launching the new one.""" + service = self.service("a") + service.enable(actor="test") + original = service.stop + + def stop_then_disable(**kwargs): + outcome = original(**kwargs) + service.intent.write(enabled=False, actor="owner") + return outcome + + with mock.patch.object(service, "stop", stop_then_disable): + refused = service.restart(allow_isolated=True, launcher=self._fail) + self.assertEqual(refused["reason"], "service_disabled") + self.assertIn("while the service was stopping", refused["detail"]) + self.assertFalse(service.intent.read()["enabled"]) + + +class ScopeAuthority(ServiceTestCase): + def test_the_production_root_does_not_move_with_the_environment(self): + fake = tempfile.mkdtemp(prefix="relay-passwd-home-") + self.addCleanup(shutil.rmtree, fake, ignore_errors=True) + entry = mock.Mock(pw_dir=fake) + with mock.patch.object(service_module.pwd, "getpwuid", return_value=entry): + with mock.patch.dict(os.environ, {"HOME": "/somewhere/else", + "XDG_STATE_HOME": "/elsewhere"}, clear=False): + first = production_scope_root() + with mock.patch.dict(os.environ, {"HOME": "/a/third/place"}, clear=False): + os.environ.pop("XDG_STATE_HOME", None) + second = production_scope_root() + self.assertEqual(first, second) + self.assertTrue(str(first).startswith(fake)) + + def test_an_override_is_isolated_and_refused_without_an_explicit_opt_in(self): + """Two launches with DIFFERENT overrides would otherwise both own the same socket.""" + for name, root in (("a", "scopes-a"), ("b", "scopes-b")): + service = self.service(name, scopes=os.path.join(self.tmp, root)) + service.intent.write(enabled=True, actor="test") + refused = service.start(allow_isolated=False, launcher=self._fail) + self.assertEqual(refused["reason"], "isolated_scope_not_allowed") + self.assertIn("CODEX_SESSION_RELAY_SCOPE_DIR", refused["detail"]) + self.assertEqual(service.status()["scopeAuthority"], ISOLATED) + + def _fail(self, *_a, **_k): # pragma: no cover + self.fail("an isolated authority must not start without an explicit opt-in") + + def test_resolve_reports_production_when_nothing_is_overridden(self): + with mock.patch.dict(os.environ, {}, clear=False): + os.environ.pop("CODEX_SESSION_RELAY_SCOPE_DIR", None) + _root, authority = resolve_scope_root() + self.assertEqual(authority, PRODUCTION) + + def test_an_installation_is_this_checkout_and_this_state_directory(self): + self.assertNotEqual( + installation_id(os.path.join(self.tmp, "a")), + installation_id(os.path.join(self.tmp, "b")), + ) + + +if __name__ == "__main__": + unittest.main() From 59197a84eaa2ce837ee4d95de67f8564cfc1e6a6 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 14:01:44 +0900 Subject: [PATCH 05/29] Carry an assignment past the lifetime of the process serving it RelayDaemon.run is bounded by construction and should stay that way, so continuation cannot be a longer loop inside one process. It is a supervisor over successive bounded workers. The supervisor acquires daemon.lock and the scope lock once and inherits them to every worker, because flock belongs to the open file description: the two share one, so the lock holds while either lives and a replacement cannot start beside an orphaned worker. That is also why a shared holder releases by closing and never by LOCK_UN, which would release it for both at once. A worker adopts those descriptors instead of taking a second lock, which the naive arrangement would have deadlocked on immediately. A worker is authenticated before it serves: the token recorded in daemon.json, a device/inode check on each inherited descriptor, and a PR_SET_PDEATHSIG armed in its own bootstrap followed at once by a getppid check, since the signal is not delivered retrospectively and a parent that died first leaves an orphan. An incomplete or unmatched supervised invocation is refused rather than falling back to an unlocked run. Recovery runs before the first worker and sends nothing. An expired lease returns its delivery to held_uncertain for the reconciler to judge, never to queued, because a queued row is eligible to send again on no evidence at all. Two defects in the ownership code landed earlier are fixed here. stop() returned before looking at the worker when the supervisor record was already gone, which is exactly the orphan case it needed to reach; and it cleared workerPid before confirming the exit, losing the only handle a later stop had. It now records the stop request before signalling and keeps each identity until that process is confirmed gone. Separately, a delivery now has to be addressed to its own assignment's endpoint. Membership in allowed_recipients is not sufficient: two assignments on one host may legitimately authorize the same recipient, so a completion belonging to A addressed to B's parent passed the membership check and was still a cross delivery. The direction is checked at enqueue and again inside attempt, before any transport call. --- .../codex-session-relay/docs/operations.md | 24 +- .../src/codex_session_relay/cli.py | 108 +++++++- .../src/codex_session_relay/daemon.py | 33 ++- .../src/codex_session_relay/delivery.py | 35 ++- .../src/codex_session_relay/policy.py | 16 ++ .../src/codex_session_relay/registry.py | 16 ++ .../src/codex_session_relay/scope.py | 41 +++ .../src/codex_session_relay/service.py | 254 ++++++++++++++++-- .../tests/test_delivery.py | 170 +++++++++++- .../codex-session-relay/tests/test_service.py | 179 ++++++++++++ 10 files changed, 828 insertions(+), 48 deletions(-) diff --git a/packages/codex-session-relay/docs/operations.md b/packages/codex-session-relay/docs/operations.md index 652236c..d2f92ec 100644 --- a/packages/codex-session-relay/docs/operations.md +++ b/packages/codex-session-relay/docs/operations.md @@ -73,9 +73,10 @@ An update tool should read `service status`, act on `stop` or `restart`, and rel guarantee: **a service that was disabled is never enabled as a side effect**. Intent lives in `service.json` and absence means never configured, which is not enabled. -Status: the commands above are implemented. `service run` currently serves ONE bounded -segment and exits; supervision across successive worker processes, which is what carries an -assignment past a single process lifetime, is still planned in PR-A. +Status: implemented. `service run` is the supervisor: it holds the locks once and replaces +bounded workers, so an assignment continues on the same store and generation past any single +process lifetime. A clean segment waits the restart interval; repeated failure backs off +exponentially to a cap and is reported as `degraded` rather than retried silently. ### Ownership @@ -94,9 +95,11 @@ Termination is bound to a process handle rather than a pid, and identity include id, the installation and the store, so a reused pid is never signalled by mistake. Where a stable handle is unavailable, ownership reports `unverifiable` and stop refuses. -Status: implemented, except that the inherited-descriptor arrangement only matters once the -supervisor launches workers, which is still planned. Today the single service process holds -both locks itself. +Status: implemented. The supervisor acquires both locks, marks them inheritable and passes +them to each worker, which adopts them rather than taking a second lock. A worker is +authenticated by a token recorded in `daemon.json` plus a device/inode check on each +descriptor, arms `PR_SET_PDEATHSIG` in its own bootstrap and immediately re-checks its +parent, and refuses to serve if the supervisor has already gone. ## One service, several projects @@ -111,8 +114,8 @@ Cancelling, pausing or archiving one assignment removes it from the loop. It doe the shared service: only the supervisor's own bound, an explicit stop, or disabled intent does that. -Status: the per-assignment refusal is partly implemented (I-13, I-74); the direction check, -project grouping and shared-service guarantees are planned in PR-A. +Status: implemented. The direction check runs at both `enqueue` and `attempt`, before any +transport call. ## Reading a stuck delivery @@ -147,5 +150,6 @@ doing anything else, and reconciliation establishes what happened without sendin attempt whose response was lost stays uncertain until evidence resolves it, and is never resent on the strength of elapsed time. -Status: the durable state and reconciliation exist; supervised restart and lease recovery -are planned in PR-A. +Status: implemented. The supervisor runs recovery before its first worker, and an expired +lease returns its delivery to `held_uncertain` for the reconciler to judge, never to the +send queue, because a queued row would be eligible to send again on no evidence. diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 4b8613b..45717e0 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -803,9 +803,11 @@ def _run_bounded(services, service, args, *, require_intent: bool) -> dict: deadline = services.clock.now() + args.deadline if args.deadline else None allow_isolated = getattr(args, "allow_isolated_scope", False) + adopted = _adopt_supervised(service, args) try: with owned_service( service, allow_isolated=allow_isolated, require_intent=require_intent, + adopt_lock_fd=adopted.get("lockFd"), adopt_scope_fd=adopted.get("scopeFd"), ) as record: daemon = RelayDaemon( services.store, services.registry, services.intake, services.delivery, @@ -823,6 +825,54 @@ def _run_bounded(services, service, args, *, require_intent: bool) -> dict: "ticks": [report.as_dict() for report in reports]} +def _adopt_supervised(service, args) -> dict: + """Validate a supervised invocation, or refuse it. Never fall back to an unlocked run. + + An fstat match proves the descriptor points at the right FILE, not that it shares the + supervisor's open file description - an independently opened descriptor for the same path + passes it. The token recorded in daemon.json and the recorded-parent check are what + actually establish that this process was launched by that supervisor. + """ + from .service import DAEMON_LOCK, arm_parent_death_signal + + import os + + token = getattr(args, "supervised_token", None) + lock_fd = getattr(args, "supervised_lock_fd", None) + scope_fd = getattr(args, "supervised_scope_fd", None) + supplied = [value for value in (token, lock_fd, scope_fd) if value is not None] + if not supplied: + return {} + + def refuse(reason, detail): + raise PayloadExit( + {"ok": False, "reason": reason, "detail": detail}, EXIT_REFUSED, + ) + + if token is None or lock_fd is None or scope_fd is None: + refuse("supervised_invocation_incomplete", + "a supervised worker needs the token and both descriptors together") + record = service.record() or {} + if not record.get("token") or record["token"] != token: + refuse("supervised_token_mismatch", "the token does not match this state directory") + if record.get("stateDir") not in (None, str(service.selection.path)): + refuse("supervised_state_mismatch", "the record names a different state directory") + if record.get("socketPath") not in (None, service.socket_path): + refuse("supervised_socket_mismatch", "the record names a different operating scope") + try: + want = os.stat(service.selection.path / DAEMON_LOCK) + got = os.fstat(lock_fd) + except OSError as error: + refuse("supervised_fd_unreadable", f"{type(error).__name__}: {error}") + if (got.st_dev, got.st_ino) != (want.st_dev, want.st_ino): + refuse("supervised_fd_mismatch", "the inherited descriptor is not this daemon lock") + death = arm_parent_death_signal(record.get("pid") or -1) + if death["orphaned"]: + refuse("supervisor_already_gone", + f"parent is {death['parent']}, not the recorded supervisor {record.get('pid')}") + return {"lockFd": lock_fd, "scopeFd": scope_fd if scope_fd >= 0 else None} + + def cmd_service(services, args) -> dict: service = _service_for(services) action = args.service_command @@ -838,15 +888,54 @@ def cmd_service(services, args) -> dict: _require_adapter(services) call = service.start if action == "start" else service.restart return _refuse_unless_ok(call( - allow_isolated=args.allow_isolated_scope, max_ticks=args.max_ticks, - deadline=args.deadline, actor=args.actor or "cli", + allow_isolated=args.allow_isolated_scope, deadline=args.deadline, + segment_seconds=args.segment_seconds, max_segments=args.max_segments, + actor=args.actor or "cli", )) if action == "run": _require_adapter(services) - return _run_bounded(services, service, args, require_intent=True) + return _supervise(services, service, args) raise SystemExit2(f"unknown service action {action!r}", EXIT_USAGE) +def _supervise(services, service, args) -> dict: + """The supervisor: it holds the locks and replaces bounded workers.""" + from .service import ServiceRefused + + def recover(): + # Establish what happened to anything in flight BEFORE a worker can send. Recovery + # itself sends nothing; it only decides what the evidence supports. + services.reconciler.recover_on_start(services.adapter) + _release_expired_leases(services) + + try: + return service.supervise( + allow_isolated=args.allow_isolated_scope, segment_seconds=args.segment_seconds, + max_segments=args.max_segments, deadline=args.deadline, on_start=recover, + ) + except ServiceRefused as refusal: + raise PayloadExit( + {"ok": False, "reason": refusal.reason, "detail": refusal.detail}, EXIT_REFUSED, + ) from refusal + + +def _release_expired_leases(services) -> None: + """An expired lease returns the attempt to reconciliation, never to the send queue. + + A sending row whose lease ran out may already have reached the recipient, so putting it + back to queued would make it eligible to send again on no evidence at all. held_uncertain + is where the reconciler can judge it (I-78). + """ + now = services.clock.now() + with services.store.transaction() as db: + db.execute( + "UPDATE deliveries SET state = 'held_uncertain', lease_owner = NULL," + " lease_until = NULL, updated_at = ?" + " WHERE state = 'sending' AND lease_until IS NOT NULL AND lease_until <= ?", + (services.clock.iso(), now), + ) + + def _reachability(services, report) -> dict: """What THIS process can actually do here, measured rather than assumed. @@ -1143,6 +1232,9 @@ def build_parser() -> argparse.ArgumentParser: daemon.add_argument("--max-ticks", type=int) daemon.add_argument("--deadline", type=float) daemon.add_argument("--allow-isolated-scope", action="store_true") + daemon.add_argument("--supervised-token") + daemon.add_argument("--supervised-lock-fd", type=int) + daemon.add_argument("--supervised-scope-fd", type=int) daemon.set_defaults(handler=cmd_daemon) service = subparsers.add_parser("service") @@ -1154,10 +1246,12 @@ def build_parser() -> argparse.ArgumentParser: hosted = actions.add_parser(name) hosted.add_argument("--actor") hosted.add_argument("--allow-isolated-scope", action="store_true") - hosted.add_argument("--max-ticks", type=int) - # One bounded segment by default. Supervision ACROSS segments is a separate change; - # an unbounded run is not constructible here and must not become one by omission. - hosted.add_argument("--deadline", type=float, default=3600.0) + # The WORKER's bound. The supervisor replaces workers; it is not itself bounded by + # this, or the service would end after a single segment. + hosted.add_argument("--segment-seconds", type=float) + # The supervisor's own optional bounds, for a test or a deliberately finite run. + hosted.add_argument("--max-segments", type=int) + hosted.add_argument("--deadline", type=float) service.set_defaults(handler=cmd_service) doctor = subparsers.add_parser("doctor") diff --git a/packages/codex-session-relay/src/codex_session_relay/daemon.py b/packages/codex-session-relay/src/codex_session_relay/daemon.py index 27f10f0..4a45cb7 100644 --- a/packages/codex-session-relay/src/codex_session_relay/daemon.py +++ b/packages/codex-session-relay/src/codex_session_relay/daemon.py @@ -43,28 +43,53 @@ def as_dict(self) -> dict: class SingleInstance: - """One daemon per state directory. A second one exits rather than racing the first.""" + """One daemon per state directory. A second one exits rather than racing the first. - def __init__(self, directory): + A supervised worker ADOPTS the descriptor its supervisor already holds rather than taking a + second lock. flock belongs to the open file description, so the supervisor and its worker + share one and the lock stays held while either of them lives - which is what stops a + replacement from starting beside an orphaned worker. + + That sharing is also why a shared holder releases by CLOSING and never by LOCK_UN: + unlocking through any duplicate descriptor releases it for every holder at once. + """ + + def __init__(self, directory, *, shared: bool = False, adopt_fd=None): self.path = Path(directory) / "daemon.lock" + self.shared = bool(shared or adopt_fd is not None) + self.adopted = adopt_fd is not None + self._adopt_fd = adopt_fd self._handle = None def __enter__(self): + if self._adopt_fd is not None: + # Already locked by the supervisor through this same description. Re-acquiring + # would be a second lock on a file we already hold. + self._handle = os.fdopen(self._adopt_fd, "r+", closefd=True) + return self self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - self._handle = open(self.path, "w") + self._handle = open(self.path, "a+") try: fcntl.flock(self._handle, fcntl.LOCK_EX | fcntl.LOCK_NB) except OSError as error: self._handle.close() self._handle = None raise RuntimeError(f"another relay daemon already holds {self.path}") from error + self._handle.seek(0) + self._handle.truncate() self._handle.write(str(os.getpid())) self._handle.flush() + if self.shared: + os.set_inheritable(self._handle.fileno(), True) return self + def fileno(self): + return self._handle.fileno() if self._handle is not None else None + def __exit__(self, *_exc): if self._handle is not None: - fcntl.flock(self._handle, fcntl.LOCK_UN) + if not self.shared: + fcntl.flock(self._handle, fcntl.LOCK_UN) self._handle.close() self._handle = None diff --git a/packages/codex-session-relay/src/codex_session_relay/delivery.py b/packages/codex-session-relay/src/codex_session_relay/delivery.py index 9961764..ced5ba5 100644 --- a/packages/codex-session-relay/src/codex_session_relay/delivery.py +++ b/packages/codex-session-relay/src/codex_session_relay/delivery.py @@ -14,7 +14,7 @@ from .identity import request_id as derive_request_id from .lifecycle import UNKNOWN as LIFECYCLE_UNKNOWN, hold_reason_for, observe, record as record_lifecycle from .policy import RetryPolicy -from .scope import check_recipient +from .scope import assert_assignment_delivery, check_recipient from .transport import ( DEFERRED_BUSY, DISPATCHED, @@ -78,8 +78,14 @@ def enqueue(self, event_id: str, *, kind: str = COMPLETION, recipient_task_id=No relationship["child"]["taskId"] if kind == REVISION else relationship["parent"]["taskId"] ) - # Refused before any transport call, and re-checked inside attempt(). - check_recipient(recipient_task_id, relationship["authorizedScope"]["allowedRecipients"]) + # Refused before any transport call, and re-checked inside attempt(). Checked BEFORE + # the idempotent early return, so re-enqueueing cannot smuggle a cross delivery past + # a row that already exists. + assert_assignment_delivery( + relationship, kind=kind, recipient_task_id=recipient_task_id, + event_relationship_id=event["relationship_id"], + manifest_paths=_manifest_paths(event), + ) existing = self.find(event_id) if existing is not None: return dict(existing) @@ -330,7 +336,12 @@ def attempt(self, event_id: str, adapter, *, now=None, owner: str = "relay"): return None if row["next_eligible_at"] is not None and row["next_eligible_at"] > now: return None - check_recipient(recipient, relationship["authorizedScope"]["allowedRecipients"]) + assert_assignment_delivery( + relationship, kind=row["kind"], recipient_task_id=recipient, + recipient_thread_id=row["recipient_thread_id"], + event_relationship_id=row["relationship_id"], + manifest_paths=_manifest_paths(self.intake.row(event_id)), + ) if self._rate_limited(recipient, now): self._reschedule( event_id, row["state"], now + self.policy.min_send_interval_seconds, @@ -728,3 +739,19 @@ def _status_for_record(observation) -> str: class _NotClaimable(Exception): pass + + +def _manifest_paths(event_row): + """The declared paths a receipt carries, or none. The manifest lives inside the receipt + JSON rather than in a column of its own.""" + if event_row is None: + return () + try: + receipt = json.loads(event_row["receipt"]) + except (TypeError, ValueError, IndexError, KeyError): + return () + entries = receipt.get("manifest") or () + return tuple( + entry["path"] for entry in entries + if isinstance(entry, dict) and isinstance(entry.get("path"), str) + ) diff --git a/packages/codex-session-relay/src/codex_session_relay/policy.py b/packages/codex-session-relay/src/codex_session_relay/policy.py index e617625..e3d6142 100644 --- a/packages/codex-session-relay/src/codex_session_relay/policy.py +++ b/packages/codex-session-relay/src/codex_session_relay/policy.py @@ -29,6 +29,22 @@ class RetryPolicy: poll_interval_seconds: float = 20.0 max_sends_per_tick: int = 4 max_reconciles_per_tick: int = 8 + # Supervision cadence. A worker is bounded by segment_seconds; the supervisor replaces it, + # which is what carries an assignment past any single process lifetime. + segment_seconds: float = 3600.0 + restart_interval_seconds: float = 2.0 + restart_base_seconds: float = 2.0 + restart_backoff_max_seconds: float = 300.0 + repeat_failure_threshold: int = 3 + + def restart_delay_for(self, consecutive_failures: int) -> float: + """A clean segment waits the plain interval; repeated failure backs off and caps.""" + if consecutive_failures <= 0: + return self.restart_interval_seconds + return min( + self.restart_backoff_max_seconds, + self.restart_base_seconds * (2 ** (consecutive_failures - 1)), + ) def delay_for(self, attempt_no: int, reason: str) -> float: """Capped exponential backoff, so repeated failure slows down instead of hammering.""" diff --git a/packages/codex-session-relay/src/codex_session_relay/registry.py b/packages/codex-session-relay/src/codex_session_relay/registry.py index 9b7874b..43a003f 100644 --- a/packages/codex-session-relay/src/codex_session_relay/registry.py +++ b/packages/codex-session-relay/src/codex_session_relay/registry.py @@ -483,6 +483,22 @@ def contract_record(record: dict) -> dict: return clean +def project_key(record: dict) -> str: + """Which project an assignment belongs to, for grouping a shared service's work. + + Grouping information, never authorization: the cross-delivery refusal is decided from the + relationship's own endpoints, not from this. A parent with no recorded cwd falls back to + its host, so every assignment lands under some key. + """ + parent = record.get("parent") or {} + cwd = parent.get("cwd") + if cwd: + import posixpath + + return posixpath.normpath(cwd) + return f"host:{parent.get('hostId')}" + + def record_settings(store, clock, task_id: str, settings: dict, *, source: str) -> dict: """Record the execution settings a task was actually created with. diff --git a/packages/codex-session-relay/src/codex_session_relay/scope.py b/packages/codex-session-relay/src/codex_session_relay/scope.py index f842546..cc273fa 100644 --- a/packages/codex-session-relay/src/codex_session_relay/scope.py +++ b/packages/codex-session-relay/src/codex_session_relay/scope.py @@ -108,6 +108,47 @@ def check_recipient(task_id: str, allowed) -> None: ) +def assert_assignment_delivery(relationship, *, kind, recipient_task_id, + recipient_thread_id=None, event_relationship_id=None, + manifest_paths=()): + """A delivery belongs to ONE assignment, and goes to that assignment's own endpoint. + + Membership in allowed_recipients is not sufficient by itself. Two assignments on one host + may legitimately authorize the same recipient, so a completion belonging to assignment A + addressed to assignment B's parent passes a membership check and is still a cross + delivery. The DIRECTION is what ties a message to its own assignment: a completion travels + to this relationship's parent, a revision to this relationship's child, and nothing else + is an authorized destination. + """ + rid = relationship["relationshipId"] + if event_relationship_id is not None and event_relationship_id != rid: + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"event belongs to relationship {event_relationship_id!r}, not {rid!r}", + ) + expected = ( + relationship["child"]["taskId"] if kind == "revision_request" + else relationship["parent"]["taskId"] + ) + if recipient_task_id != expected: + direction = "child" if kind == "revision_request" else "parent" + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"a {kind} for {rid!r} goes to its own {direction} {expected!r}, not to " + f"{recipient_task_id!r}", + ) + if recipient_thread_id is not None and recipient_thread_id != recipient_task_id: + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"the native thread {recipient_thread_id!r} is not the recipient task " + f"{recipient_task_id!r}", + ) + check_recipient(recipient_task_id, relationship["authorizedScope"]["allowedRecipients"]) + roots = relationship["authorizedScope"]["artifactRoots"] + for path in manifest_paths: + assert_within(path, roots) + + @dataclass(frozen=True) class PathBinding: declared: str diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 49cae0a..360bed0 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -22,6 +22,7 @@ import pwd import signal import time +import uuid from contextlib import contextmanager from dataclasses import dataclass, field from pathlib import Path @@ -33,6 +34,7 @@ DAEMON_RECORD = "daemon.json" SERVICE_INTENT = "service.json" DAEMON_LOG = "daemon.log" +STOP_REQUEST = "stop.request" OURS, FOREIGN, UNVERIFIABLE, NONE = "ours", "foreign", "unverifiable", "none" @@ -148,7 +150,7 @@ def read(self, socket_path): except (OSError, ValueError): return None - def claim(self, socket_path, record: dict): + def claim(self, socket_path, record: dict, *, shared: bool = False): """Take the scope lock and write the record, or report who already holds it. Liveness is the LOCK, never a recorded pid. A supervisor that dies leaving a worker @@ -167,6 +169,8 @@ def claim(self, socket_path, record: dict): return {"ok": False, "reason": "scope_owned_by_other_store", "held_by": self.read(socket_path), "scopeKey": self.key(socket_path)} self._handle = handle + if shared: + os.set_inheritable(handle.fileno(), True) payload = dict(record, scopeKey=self.key(socket_path), scopeAuthority=self.authority, socketPath=str(socket_path), registeredAt=_now()) self.record_path(socket_path).write_text(json.dumps(payload, indent=2), encoding="utf-8") @@ -247,6 +251,34 @@ def installation_id(state_dir) -> str: return hashlib.sha256(material.encode()).hexdigest()[:16] +PR_SET_PDEATHSIG = 1 + + +def arm_parent_death_signal(expected_parent: int) -> dict: + """Ask the kernel to signal this process when its parent dies, then check it is not late. + + PR_SET_PDEATHSIG is not delivered retrospectively, so a parent that died before the child + armed it leaves an orphan. The getppid check immediately afterwards is what closes that + window, and it can only be done here, in the child. + + Armed in the child's own bootstrap rather than through preexec_fn, which is unsafe once + the supervisor has started the adapter's transport thread. + """ + armed, detail = False, None + try: + import ctypes + + libc = ctypes.CDLL("libc.so.6", use_errno=True) + armed = libc.prctl(PR_SET_PDEATHSIG, signal.SIGTERM, 0, 0, 0) == 0 + if not armed: + detail = f"prctl failed: errno {ctypes.get_errno()}" + except (OSError, AttributeError, ValueError) as error: + detail = f"{type(error).__name__}: {error}" + actual = os.getppid() + orphaned = actual != expected_parent + return {"armed": armed, "detail": detail, "parent": actual, "orphaned": orphaned} + + class ProcessHandle: """A handle to a specific process, not to a number. @@ -484,11 +516,11 @@ def conflicts(self) -> list: def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) -> dict: """Signal only a process this installation owns, through a handle to that process.""" record = self.record() + # Recorded before any signal, so a supervisor that is about to die knows this was a + # graceful stop and does not launch a replacement on its way out. + self.request_stop() owner, handle, detail = self.ownership(record) try: - if owner == NONE: - return {"ok": False, "reason": "not_running", "detail": detail, - "supervisor": "gone", "worker": "gone"} if owner == FOREIGN: return {"ok": False, "reason": "not_ours", "detail": detail, "supervisor": "untouched", "worker": "untouched"} @@ -497,15 +529,32 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # process gets killed after its number is reused. return {"ok": False, "reason": "ownership_unverifiable", "detail": detail, "supervisor": "untouched", "worker": "untouched"} - outcome = self._terminate(handle, timeout=timeout, grace=grace) + # NONE still falls through to the worker: a supervisor can die leaving its worker + # alive, and that orphan is exactly what a stop has to reach. + outcome = ("gone" if owner == NONE + else self._terminate(handle, timeout=timeout, grace=grace)) finally: if handle is not None: handle.close() worker = self._stop_worker(record, timeout=timeout, grace=grace) - if outcome == "exited" and record is not None: - self.write_record(dict(record, pid=None, workerPid=None, - stoppedAt=_now(), stoppedBy=actor)) - ok = outcome == "exited" and worker in ("exited", "gone") + supervisor_done = outcome in ("exited", "gone") + worker_done = worker in ("exited", "gone") + if record is not None: + # Identity is kept until termination is CONFIRMED. Clearing a pid we have not + # seen exit would lose the only handle a later stop has to reach it. + cleared = dict(record, stoppedBy=actor) + if supervisor_done: + cleared["pid"] = None + if worker_done: + cleared["workerPid"] = None + cleared["workerStartTicks"] = None + if supervisor_done and worker_done: + cleared["stoppedAt"] = _now() + self.write_record(cleared) + if owner == NONE and worker == "gone": + return {"ok": False, "reason": "not_running", "detail": detail, + "supervisor": "gone", "worker": "gone"} + ok = supervisor_done and worker_done return {"ok": ok, "reason": None if ok else "did_not_exit", "detail": None, "supervisor": outcome, "worker": worker} @@ -561,7 +610,8 @@ def restart(self, *, allow_isolated: bool = False, launcher=None, **kw) -> dict: "start": started} def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cli", - max_ticks=None, deadline=None, timeout: float = 20.0, poll: float = 0.05) -> dict: + max_ticks=None, deadline=None, segment_seconds=None, max_segments=None, + timeout: float = 20.0, poll: float = 0.05) -> dict: """Preconditions here; ownership in the child. The daemon lock and the scope claim are taken by the process that will HOLD them, not @@ -585,6 +635,7 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl before = (self.record() or {}).get("pid") child = (launcher or self.default_launcher)( self, allow_isolated=allow_isolated, max_ticks=max_ticks, deadline=deadline, + segment_seconds=segment_seconds, max_segments=max_segments, ) deadline_at = time.monotonic() + timeout while time.monotonic() < deadline_at: @@ -608,7 +659,8 @@ def _log_tail(self, lines: int = 20) -> str: except OSError: return "" - def default_launcher(self, service, *, allow_isolated, max_ticks, deadline): + def default_launcher(self, service, *, allow_isolated, max_ticks=None, deadline=None, + segment_seconds=None, max_segments=None): import subprocess import sys @@ -619,8 +671,10 @@ def default_launcher(self, service, *, allow_isolated, max_ticks, deadline): argv += ["service", "run"] if allow_isolated: argv.append("--allow-isolated-scope") - if max_ticks is not None: - argv += ["--max-ticks", str(max_ticks)] + if segment_seconds is not None: + argv += ["--segment-seconds", str(segment_seconds)] + if max_segments is not None: + argv += ["--max-segments", str(max_segments)] if deadline is not None: argv += ["--deadline", str(deadline)] environment = dict(os.environ) @@ -635,10 +689,152 @@ def default_launcher(self, service, *, allow_isolated, max_ticks, deadline): start_new_session=True, env=environment, ) + # ------------------------------------------------------------- supervision + + @property + def stop_request_path(self) -> Path: + return self.selection.path / STOP_REQUEST + + def request_stop(self) -> None: + """Recorded BEFORE any signal, so a graceful stop is never read as a crash.""" + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + self.stop_request_path.write_text(_now(), encoding="utf-8") + + def clear_stop_request(self) -> None: + try: + self.stop_request_path.unlink() + except OSError: + pass + + def stop_requested(self) -> bool: + return self.stop_request_path.exists() + + def spawn_worker(self, *, lock_fd, scope_fd, token, segment_seconds, allow_isolated): + """One bounded worker, sharing the descriptors this supervisor already holds.""" + import subprocess + import sys + + argv = [sys.executable, "-m", "codex_session_relay.cli", + "--state", str(self.selection.path)] + if self.socket_path: + argv += ["--socket", str(self.socket_path)] + argv += ["daemon", "--deadline", str(segment_seconds), + "--supervised-token", token, + "--supervised-lock-fd", str(lock_fd), + "--supervised-scope-fd", str(scope_fd if scope_fd is not None else -1)] + if allow_isolated: + argv.append("--allow-isolated-scope") + environment = dict(os.environ) + if self.scope.authority == ISOLATED: + environment[SCOPE_ENV] = str(self.scope.root) + pass_fds = tuple(fd for fd in (lock_fd, scope_fd) if fd is not None) + with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: + return subprocess.Popen( + argv, stdout=log, stderr=log, stdin=subprocess.DEVNULL, + pass_fds=pass_fds, env=environment, + ) + + def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments=None, + deadline=None, spawn=None, policy=None, sleeper=None, on_start=None) -> dict: + """Replace bounded workers for as long as the owner wants this service running. + + RelayDaemon.run stays bounded by construction; continuation is a supervisor OVER + successive bounded workers, not a longer loop inside one. The locks are acquired once + here and inherited by every worker, so the store, the generations and the scope claim + are untouched across a worker boundary - which is what carries an assignment past any + single process lifetime without asking the parent model anything. + """ + from .daemon import SingleInstance + from .policy import RetryPolicy + + policy = policy or RetryPolicy() + sleeper = sleeper or time.sleep + spawn = spawn or self.spawn_worker + segment_seconds = segment_seconds or policy.segment_seconds + + gate = self.authority_check(allow_isolated=allow_isolated) + if not gate["ok"]: + raise ServiceRefused(gate["reason"], gate["detail"]) + if not self.intent.read()["enabled"]: + raise ServiceRefused( + "service_disabled", + "this service is not enabled; supervising it would ignore the owner's intent", + ) + + self.clear_stop_request() + token = uuid.uuid4().hex + segments, failures, degraded = [], 0, None + with SingleInstance(self.selection.path, shared=True) as lock: + scope_fd = None + if self.socket_path: + claim = self.scope.claim( + self.socket_path, self.new_record(pid=os.getpid(), token=token), + shared=True, + ) + if not claim["ok"]: + raise ServiceRefused(claim["reason"], json.dumps(claim.get("held_by"))) + scope_fd = claim["lockFd"] + self.write_record(self.new_record(pid=os.getpid(), token=token)) + try: + if on_start is not None: + on_start() + started = time.monotonic() + while True: + if max_segments is not None and len(segments) >= max_segments: + break + if deadline is not None and time.monotonic() - started >= deadline: + break + if self.stop_requested(): + break + # Re-read every cycle: an owner who disables the service while a worker + # was running gets no replacement, and the supervisor never writes intent. + if not self.intent.read()["enabled"]: + break + child = spawn( + lock_fd=lock.fileno(), scope_fd=scope_fd, token=token, + segment_seconds=segment_seconds, allow_isolated=allow_isolated, + ) + self._note(workerPid=child.pid, workerStartTicks=start_ticks(child.pid)) + code = child.wait() + self._note(workerPid=None, workerStartTicks=None, lastExit=code, + restarts=len(segments) + 1) + segments.append(code) + failures = failures + 1 if code != 0 else 0 + if failures >= policy.repeat_failure_threshold: + degraded = (f"{failures} consecutive worker failures, last exit {code}") + self.store_journal_note(degraded) + self._note(consecutiveFailures=failures, degraded=degraded) + if self.stop_requested() or not self.intent.read()["enabled"]: + break + wait = policy.restart_delay_for(failures) + self._note(nextRestartAt=time.time() + wait) + sleeper(wait) + finally: + if self.socket_path: + self.scope.release(self.socket_path) + current = self.record() or {} + self.write_record(dict(current, pid=None, workerPid=None, stoppedAt=_now())) + return {"ok": True, "reason": None, "segments": segments, + "consecutiveFailures": failures, "degraded": degraded} + + def _note(self, **fields) -> None: + record = self.record() + if record is not None: + self.write_record(dict(record, **fields)) + + def store_journal_note(self, detail: str) -> None: + """Repeated failure is reported, not silently retried forever.""" + path = self.selection.path / DAEMON_LOG + try: + with open(path, "a", encoding="utf-8") as log: + log.write(f"{_now()} service_degraded: {detail}\n") + except OSError: + pass + @contextmanager def owned_service(service: RelayService, *, allow_isolated: bool = False, token=None, - require_intent: bool = True): + require_intent: bool = True, adopt_lock_fd=None, adopt_scope_fd=None): """Hold the daemon lock and the scope claim for as long as this process serves. Both are released together on the way out, and the scope RECORD is deliberately kept: a @@ -655,22 +851,38 @@ def owned_service(service: RelayService, *, allow_isolated: bool = False, token= "service_disabled", "this service is not enabled; running it would ignore the owner's intent", ) - with SingleInstance(service.selection.path): + supervised = adopt_lock_fd is not None + with SingleInstance(service.selection.path, adopt_fd=adopt_lock_fd): claim = {"ok": True, "reason": None} - if service.socket_path: + if service.socket_path and not supervised: claim = service.scope.claim(service.socket_path, service.new_record( pid=os.getpid(), token=token, )) if not claim["ok"]: raise ServiceRefused(claim["reason"], json.dumps(claim.get("held_by"))) - record = service.write_record(service.new_record(pid=os.getpid(), token=token)) + if supervised: + # The supervisor owns both the claim and the record. A worker that rewrote them + # would erase the identity a stop needs to reach the supervisor. + record = service.record() or service.new_record(pid=os.getpid(), token=token) + else: + record = service.write_record(service.new_record(pid=os.getpid(), token=token)) try: yield record finally: - if service.socket_path: - service.scope.release(service.socket_path) - current = service.record() or record - service.write_record(dict(current, pid=None, workerPid=None, stoppedAt=_now())) + if not supervised: + if service.socket_path: + service.scope.release(service.socket_path) + current = service.record() or record + service.write_record( + dict(current, pid=None, workerPid=None, stoppedAt=_now()), + ) + elif adopt_scope_fd is not None and adopt_scope_fd >= 0: + # Close our copy only. The supervisor still holds the description, so the + # scope stays claimed; unlocking would release it for both of us. + try: + os.close(adopt_scope_fd) + except OSError: + pass class ServiceRefused(Exception): diff --git a/packages/codex-session-relay/tests/test_delivery.py b/packages/codex-session-relay/tests/test_delivery.py index 69024da..7e7ff6d 100644 --- a/packages/codex-session-relay/tests/test_delivery.py +++ b/packages/codex-session-relay/tests/test_delivery.py @@ -1,8 +1,9 @@ """JUN-91 delivery: busy handling, dispatch, bounds, and honest reporting.""" +import os import unittest -from codex_session_relay.delivery import COMPLETION, REVISION +from codex_session_relay.delivery import COMPLETION, REVISION, DeliveryService from codex_session_relay.errors import RefusalReason from codex_session_relay.lifecycle import ARCHIVED, BUDGET_LIMITED, CANNOT_ACCEPT, PAUSED, UNKNOWN from codex_session_relay.transport import ( @@ -14,7 +15,7 @@ WITHHELD_PRE_SEND, ) -from .support import CHILD, PARENT, DeliveryTestCase +from .support import CHILD, HOST, PARENT, DeliveryTestCase class Queueing(DeliveryTestCase): @@ -583,3 +584,168 @@ def test_the_instruction_each_side_is_given_is_the_one_that_actually_works(self) # What the message DOES tell the child to do is available: emit under generation 2. relationship = self.registry.get(self._rid) self.assertEqual(relationship["executionGeneration"], 2) + + +class CrossAssignmentDelivery(DeliveryTestCase): + """One shared service carries several assignments; none may answer for another.""" + + def other_assignment(self): + """A second project whose parent is ALSO an authorized recipient of the first.""" + from codex_session_relay.models import Endpoint + + other_root = os.path.join(self.tmp, "other-project") + os.makedirs(other_root, exist_ok=True) + return self.registry.register( + parent=Endpoint("01other-parent", HOST, cwd="/other", cxc_session="cxc-other"), + child=Endpoint("01other-child", HOST, cwd=other_root, cxc_session="cxc-other-c"), + issue_key="REL-2", + artifact_roots=[other_root], + allowed_recipients=["01other-parent"], + dispatch_request_id="dispatch-2", + dispatch_turn_id="turn-dispatch-2", + ) + + def test_a_completion_may_not_be_addressed_to_another_assignments_parent(self): + other = self.other_assignment() + # The first assignment authorizes the other project's parent as a recipient, which is + # legitimate. Membership alone must still not make it a valid destination. + relationship, event_id = self.ready_event( + recipients=[PARENT, other["parent"]["taskId"]], + ) + refused = self.assertRefused( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + self.delivery.enqueue, event_id, + recipient_task_id=other["parent"]["taskId"], + ) + self.assertIn("its own parent", refused.detail) + self.assertIsNone(self.delivery.find(event_id)) + + def test_a_tampered_delivery_row_is_refused_before_any_transport_call(self): + other = self.other_assignment() + relationship, event_id = self.queued_event( + recipients=[PARENT, other["parent"]["taskId"]], + ) + self.adapter.add_thread(other["parent"]["taskId"]) + with self.store.transaction() as db: + db.execute( + "UPDATE deliveries SET recipient_task_id = ?, recipient_thread_id = ?" + " WHERE event_id = ?", + (other["parent"]["taskId"], other["parent"]["taskId"], event_id), + ) + self.assertRefused( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, self.attempt, event_id, + ) + self.assertEqual(self.adapter.sends, [], "nothing may reach the host") + + def test_the_ordinary_completion_still_goes_to_its_own_parent(self): + self.other_assignment() + relationship, event_id = self.queued_event() + record = self.attempt(event_id) + self.assertEqual(record["deliveryState"], "dispatched") + self.assertEqual(self.delivery.get(event_id)["recipient_task_id"], PARENT) + + def test_projects_are_distinguishable_for_a_shared_service(self): + from codex_session_relay.registry import project_key + + other = self.other_assignment() + mine = self.register(issue_key="REL-3", dispatch_request_id="dispatch-3") + self.assertNotEqual(project_key(mine), project_key(other)) + self.assertEqual(project_key(other), "/other") + + +class RestartPreservation(DeliveryTestCase): + """Everything durable survives a process boundary, and nothing is sent twice for it.""" + + def reopen(self): + """Close the store and open it again: the process boundary, minus the process.""" + from codex_session_relay.ack import AckService + from codex_session_relay.receipts import ReceiptIntake + from codex_session_relay.reconcile import Reconciler + from codex_session_relay.registry import Registry + from codex_session_relay.store import Store + from codex_session_relay.sync import SyncOutbox + + path = self.store.path + self.store.close() + self.store = Store(path) + self.addCleanup(self.store.close) + self.registry = Registry(self.store, self.clock) + self.intake = ReceiptIntake(self.store, self.registry, self.clock) + self.delivery = DeliveryService(self.store, self.registry, self.intake, self.clock) + self.ack = AckService( + self.store, self.registry, self.intake, self.delivery, self.clock, + ) + self.reconciler = Reconciler(self.store, self.registry, self.delivery, self.clock) + self.sync = SyncOutbox(self.store, self.clock) + + def snapshot(self, table, columns): + return [ + tuple(row[column] for column in columns) + for row in self.store.all(f"SELECT * FROM {table} ORDER BY rowid") + ] + + def test_a_restart_keeps_every_durable_record_and_resends_nothing(self): + relationship, dispatched_event = self.queued_event() + rid = relationship["relationshipId"] + record = self.attempt(dispatched_event) + self.assertEqual(record["deliveryState"], DISPATCHED) + + # A second event left genuinely unresolved: a settled attempt never enters + # open_attempts, so without this the recovery assertion would prove nothing. + path = self.artifact("second.txt", "still in flight") + payload = self.ready_payload(relationship, [path], attempt=2) + self.accept(payload) + in_flight = payload["eventId"] + self.delivery.enqueue(in_flight) + self.adapter.script("transport_unknown") + # Past the minimum send interval, or the claim never happens and there is no + # unresolved attempt for recovery to find. + later = self.clock.now() + 3600 + self.attempt(in_flight, now=later) + unresolved = self.store.one( + "SELECT request_id FROM attempts WHERE event_id = ? AND state = ?", + (in_flight, HELD_UNCERTAIN), + )["request_id"] + + from codex_session_relay.sync import SyncOutbox + + SyncOutbox(self.store, self.clock).set_target(rid, "coordination_document", "DOC-1") + before = { + "relationships": self.snapshot("relationships", ("relationship_id", "status", + "execution_generation")), + "generations": self.snapshot("generations", ("relationship_id", + "execution_generation", + "anchor_state", "dispatch_turn_id")), + "events": self.snapshot("events", ("event_id", "stage", "revision_hash")), + "deliveries": self.snapshot("deliveries", ("event_id", "state", "attempt_count")), + "sync_targets": self.snapshot("sync_targets", ("relationship_id", "target_ref")), + } + sends_before = len(self.adapter.sends) + + self.reopen() + + for table, columns in ( + ("relationships", ("relationship_id", "status", "execution_generation")), + ("generations", ("relationship_id", "execution_generation", "anchor_state", + "dispatch_turn_id")), + ("events", ("event_id", "stage", "revision_hash")), + ("deliveries", ("event_id", "state", "attempt_count")), + ("sync_targets", ("relationship_id", "target_ref")), + ): + self.assertEqual(self.snapshot(table, columns), before[table], table) + + # Recovery establishes what happened. It sends nothing. + outcome = self.reconciler.recover_on_start(self.adapter) + self.assertIn(unresolved, [entry["requestId"] for entry in outcome["reconciled"]]) + self.assertEqual(outcome["resent"], []) + self.assertIn(dispatched_event, outcome["awaitingAck"]) + self.assertEqual(len(self.adapter.sends), sends_before, "recovery must not send") + + # And a replay of the already dispatched event is refused by the claim itself, + # rather than merely being left out of the schedule. + attempts_before = self.delivery.get(dispatched_event)["attempt_count"] + self.assertIsNone(self.attempt(dispatched_event, now=later + 3600)) + self.assertEqual( + self.delivery.get(dispatched_event)["attempt_count"], attempts_before, + ) + self.assertEqual(len(self.adapter.sends), sends_before) diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 9059aed..659d53e 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -263,3 +263,182 @@ def test_an_installation_is_this_checkout_and_this_state_directory(self): if __name__ == "__main__": unittest.main() + + +class FakeWorker: + """Stands in for a bounded worker process, so cadence is tested without real time.""" + + def __init__(self, code=0, pid=None): + self.returncode = code + self.pid = pid or os.getpid() + + def wait(self): + return self.returncode + + +class Supervision(ServiceTestCase): + def supervised(self, service, *, codes, **kwargs): + """Run the supervisor over a scripted sequence of worker exits.""" + launches, slept = [], [] + queue = list(codes) + + def spawn(**call): + launches.append(call) + return FakeWorker(queue.pop(0) if queue else 0) + + outcome = service.supervise( + allow_isolated=True, spawn=spawn, sleeper=slept.append, + max_segments=len(codes), **kwargs, + ) + return outcome, launches, slept + + def test_a_worker_that_exits_is_replaced_on_the_same_store(self): + service = self.service("a") + service.enable(actor="test") + before = service.store_id + outcome, launches, _slept = self.supervised(service, codes=[0, 0, 0]) + self.assertEqual(outcome["segments"], [0, 0, 0]) + self.assertEqual(len(launches), 3, "each bounded worker is replaced by a new one") + # One lock, one claim, one store across every replacement. + self.assertEqual(service.store_id, before) + self.assertEqual(service.record()["restarts"], 3) + self.assertFalse(service.lock_is_held(), "the supervisor released on the way out") + + def test_every_worker_inherits_the_descriptors_the_supervisor_holds(self): + service = self.service("a") + service.enable(actor="test") + _outcome, launches, _slept = self.supervised(service, codes=[0, 0]) + self.assertEqual(len({call["lock_fd"] for call in launches}), 1) + self.assertEqual(len({call["scope_fd"] for call in launches}), 1) + self.assertEqual(len({call["token"] for call in launches}), 1) + self.assertIsNotNone(launches[0]["lock_fd"]) + self.assertIsNotNone(launches[0]["scope_fd"]) + + def test_repeated_failure_backs_off_and_is_reported(self): + service = self.service("a") + service.enable(actor="test") + outcome, _launches, slept = self.supervised(service, codes=[1, 1, 1]) + self.assertEqual(outcome["consecutiveFailures"], 3) + self.assertIsNotNone(outcome["degraded"], "repeated failure is reported, not hidden") + self.assertIn("3 consecutive worker failures", outcome["degraded"]) + # Capped exponential, not a fixed interval and not a tight loop. + self.assertEqual(slept[:2], [2.0, 4.0]) + log = (service.selection.path / "daemon.log").read_text(encoding="utf-8") + self.assertIn("service_degraded", log) + + def test_a_clean_segment_resets_the_failure_count(self): + service = self.service("a") + service.enable(actor="test") + outcome, _launches, slept = self.supervised(service, codes=[1, 1, 0]) + self.assertEqual(outcome["consecutiveFailures"], 0) + self.assertEqual(slept[:3], [2.0, 4.0, 2.0]) + + def test_disabling_the_service_ends_supervision_at_the_boundary(self): + service = self.service("a") + service.enable(actor="test") + launches = [] + + def spawn(**call): + launches.append(call) + service.intent.write(enabled=False, actor="owner") + return FakeWorker(0) + + outcome = service.supervise( + allow_isolated=True, spawn=spawn, sleeper=lambda _s: None, max_segments=5, + ) + self.assertEqual(len(launches), 1, "no replacement after the owner disabled it") + self.assertEqual(outcome["segments"], [0]) + self.assertFalse(service.intent.read()["enabled"]) + + def test_a_stop_request_ends_supervision_without_a_replacement(self): + service = self.service("a") + service.enable(actor="test") + launches = [] + + def spawn(**call): + launches.append(call) + service.request_stop() + return FakeWorker(0) + + service.supervise( + allow_isolated=True, spawn=spawn, sleeper=lambda _s: None, max_segments=5, + ) + self.assertEqual(len(launches), 1) + + def test_supervising_a_disabled_service_is_refused(self): + service = self.service("a") + with self.assertRaises(ServiceRefused) as caught: + service.supervise(allow_isolated=True, spawn=lambda **_k: FakeWorker(0)) + self.assertEqual(caught.exception.reason, "service_disabled") + + +class SupervisedWorker(ServiceTestCase): + """A worker adopts what its supervisor holds, or it is refused outright.""" + + def adopt(self, service, **overrides): + from codex_session_relay.cli import _adopt_supervised + + class Args: + pass + + args = Args() + args.supervised_token = overrides.get("token", "tok") + args.supervised_lock_fd = overrides.get("lock_fd", 0) + args.supervised_scope_fd = overrides.get("scope_fd", -1) + return _adopt_supervised(service, args) + + def prepared(self, **record): + service = self.service("a") + service.selection.path.mkdir(parents=True, exist_ok=True) + (service.selection.path / "daemon.lock").write_text("", encoding="utf-8") + base = service.new_record(pid=os.getppid(), token="tok") + service.write_record(dict(base, **record)) + return service + + def test_an_unsupervised_run_is_untouched(self): + from codex_session_relay.cli import _adopt_supervised + + class Args: + supervised_token = None + supervised_lock_fd = None + supervised_scope_fd = None + + self.assertEqual(_adopt_supervised(self.service("a"), Args()), {}) + + def test_a_partial_supervised_invocation_is_refused(self): + from codex_session_relay.cli import PayloadExit + + service = self.prepared() + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, scope_fd=None) + self.assertEqual(caught.exception.payload["reason"], "supervised_invocation_incomplete") + + def test_a_wrong_token_is_refused(self): + from codex_session_relay.cli import PayloadExit + + service = self.prepared() + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, token="not-the-token") + self.assertEqual(caught.exception.payload["reason"], "supervised_token_mismatch") + + def test_a_descriptor_for_another_file_is_refused(self): + from codex_session_relay.cli import PayloadExit + + service = self.prepared() + stranger = os.open(os.path.join(self.tmp, "not-a-lock"), os.O_CREAT | os.O_RDWR, 0o600) + self.addCleanup(os.close, stranger) + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, lock_fd=stranger) + self.assertEqual(caught.exception.payload["reason"], "supervised_fd_mismatch") + + def test_a_worker_whose_supervisor_is_already_gone_refuses_to_serve(self): + from codex_session_relay.cli import PayloadExit + + # The record names a supervisor that is not this process's parent, which is exactly + # what an orphan looks like when PDEATHSIG arrived too late to help. + service = self.prepared(pid=999999) + lock = os.open(service.selection.path / "daemon.lock", os.O_RDWR) + self.addCleanup(os.close, lock) + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, lock_fd=lock) + self.assertEqual(caught.exception.payload["reason"], "supervisor_already_gone") From d48c8e470f948bd55002886bcbccd678a7e86f66 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 15:39:11 +0900 Subject: [PATCH 06/29] Treat two names for one socket as one operating scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ScopeRegistry.key hashed the socket path as supplied, only made absolute. A socket reached through a symlink or a relative alias is the same App Server, but the two spellings hashed differently, so both launches took a scope lock and both served it — which is the precise thing this registry exists to prevent. Resolving before hashing closes it. Found by review on #8. --- .../src/codex_session_relay/service.py | 6 +++++- packages/codex-session-relay/tests/test_service.py | 12 ++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 360bed0..3128984 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -119,7 +119,11 @@ class ScopeRegistry: _handle: object = field(default=None, repr=False) def key(self, socket_path) -> str: - canonical = str(Path(socket_path).expanduser().absolute()) + # Resolved, not merely absolute. A socket reachable through a symlink or a relative + # alias is the SAME operating scope, and hashing the supplied spelling would give the + # two launches different keys - so both would take a lock and both would serve one + # App Server, which is the exact thing this registry exists to prevent. + canonical = str(Path(socket_path).expanduser().absolute().resolve()) digest = hashlib.sha256(canonical.encode()).hexdigest()[:16] if self.authority == PRODUCTION: return digest diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 659d53e..9a41f1c 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -260,6 +260,18 @@ def test_an_installation_is_this_checkout_and_this_state_directory(self): installation_id(os.path.join(self.tmp, "b")), ) + def test_two_names_for_one_socket_are_one_operating_scope(self): + """A symlinked socket is the same App Server, so it must be the same lock.""" + real = os.path.join(self.tmp, "real.sock") + open(real, "w", encoding="utf-8").close() + alias = os.path.join(self.tmp, "alias.sock") + os.symlink(real, alias) + registry = ScopeRegistry(Path(self.scopes), ISOLATED) + self.assertEqual( + registry.key(real), registry.key(alias), + "two spellings of one socket must not give two owners", + ) + if __name__ == "__main__": unittest.main() From 1610302f73a73854f33625d79184a0643dade672 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 15:47:02 +0900 Subject: [PATCH 07/29] Answer the ownership findings review raised on this branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit stop() wrote the stop request before it validated ownership, so a refusal still had an effect: a caller told not_ours or ownership_unverifiable had already left a request that would halt another installation's supervisor at its next boundary. The request is now written only after ownership is established. A record with no recorded start time was treated as ours. Without a start time a pid is just a number and whatever holds it now would pass the check, which is the pid-reuse hazard the check exists for. It now reports unverifiable, and stop refuses on that. The supervisor deadline was checked only before spawning, so a worker started just before it outlived the supervisor by a whole segment — up to an hour with the default. The worker's bound is now clamped to whatever remains of the supervisor's own. --- .../src/codex_session_relay/service.py | 24 +++++++++++++++---- .../codex-session-relay/tests/test_service.py | 16 +++++++++++++ 2 files changed, 35 insertions(+), 5 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 3128984..0c51e96 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -424,7 +424,13 @@ def ownership(self, record=None): if record.get("bootId") not in (None, boot_id()): mismatches.append("recorded before a different boot") ticks = start_ticks(record["pid"]) - if record.get("startTicks") is not None and ticks != record["startTicks"]: + if record.get("startTicks") is None or ticks is None: + # Without a start time a pid is just a number, and whatever holds it now would + # pass. Unverifiable is the honest answer, and stop refuses on it. + return UNVERIFIABLE, handle, ( + "no start time is available for this pid, so identity cannot be established" + ) + if ticks != record["startTicks"]: mismatches.append("the pid was reused by a different process") if record.get("installationId") != self.installation_id: mismatches.append("another installation owns it") @@ -520,9 +526,6 @@ def conflicts(self) -> list: def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) -> dict: """Signal only a process this installation owns, through a handle to that process.""" record = self.record() - # Recorded before any signal, so a supervisor that is about to die knows this was a - # graceful stop and does not launch a replacement on its way out. - self.request_stop() owner, handle, detail = self.ownership(record) try: if owner == FOREIGN: @@ -533,6 +536,10 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # process gets killed after its number is reused. return {"ok": False, "reason": "ownership_unverifiable", "detail": detail, "supervisor": "untouched", "worker": "untouched"} + # Only now. Writing the stop request before validating ownership left a refused + # stop able to halt another installation's supervisor at its next boundary, which + # is a refusal that still had an effect. + self.request_stop() # NONE still falls through to the worker: a supervisor can die leaving its worker # alive, and that orphan is exactly what a stop has to reach. outcome = ("gone" if owner == NONE @@ -796,7 +803,14 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= break child = spawn( lock_fd=lock.fileno(), scope_fd=scope_fd, token=token, - segment_seconds=segment_seconds, allow_isolated=allow_isolated, + # Clamped to what remains of the supervisor's own bound, or a worker + # started just before the deadline outlives it by a whole segment. + segment_seconds=( + segment_seconds if deadline is None else + max(0.1, min(segment_seconds, + deadline - (time.monotonic() - started))) + ), + allow_isolated=allow_isolated, ) self._note(workerPid=child.pid, workerStartTicks=start_ticks(child.pid)) code = child.wait() diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 9a41f1c..d501434 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -160,6 +160,22 @@ def test_stop_refuses_when_the_process_cannot_be_aimed_at(self): refused = service.stop() self.assertEqual(refused["reason"], "ownership_unverifiable") self.assertIsNone(child.poll(), "refusing is the point; never fall back to os.kill") + self.assertFalse( + service.stop_request_path.exists(), + "a refused stop must leave no request behind; it would halt someone else", + ) + + def test_a_record_without_a_start_time_is_unverifiable_not_ours(self): + """A pid with no start time is just a number, and anything holding it would pass.""" + service = self.service("a") + child, _pid = self.holder(service) + record = service.record() + service.write_record(dict(record, startTicks=None)) + refused = service.stop() + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIn("start time", refused["detail"]) + self.assertIsNone(child.poll()) + self.assertFalse(service.stop_request_path.exists()) def test_stop_terminates_a_process_this_installation_owns(self): service = self.service("a") From 75273844a06706927007ff87198f898fb28dc2dc Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 15:52:29 +0900 Subject: [PATCH 08/29] Answer the second review round on ownership and identity Five findings, one of them caused by my own previous fix. That one first: returning unverifiable as soon as a start time was missing discarded definitive proof. Installation, store and boot each establish a record is foreign on their own, so they are now evaluated first and unverifiable is reserved for the case where nothing proves foreign and process identity still cannot be established. A stopped registration naming a different store was being overwritten. The record deliberately outlives the process so a duplicate stays visible, and then the next claim erased it. claim now refuses a scope already registered to another store and returns that registration, so replacing one has to be deliberate. An unreadable nonce was graded a mismatch, which told an operator two participants use different stores when the truth was that this one could not look. Unreadable is unproven; mismatch is reserved for a readable store where the nonce is absent. start waited for the recorded pid to change, but after a crash the OS can hand the replacement the very pid the stale record names, so a healthy service would have timed out as did_not_report. It now matches a per-launch id the child writes. The supervisor released the scope without shared=True on the exceptional path, which issues LOCK_UN on the descriptor every worker inherited and would let another state directory claim the socket beside a still-running orphan. And a managed launch forwarded --state without the environment variable, so it inherited the store/ledger split the contract warns about; both are now set together. --- .../src/codex_session_relay/cli.py | 3 ++ .../src/codex_session_relay/service.py | 49 +++++++++++++++---- .../src/codex_session_relay/store.py | 7 ++- .../codex-session-relay/tests/test_service.py | 27 ++++++++++ .../codex-session-relay/tests/test_store.py | 12 +++++ 5 files changed, 88 insertions(+), 10 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 45717e0..d981ce8 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -902,6 +902,8 @@ def _supervise(services, service, args) -> dict: """The supervisor: it holds the locks and replaces bounded workers.""" from .service import ServiceRefused + service.launch_id = getattr(args, "launch_id", None) + def recover(): # Establish what happened to anything in flight BEFORE a worker can send. Recovery # itself sends nothing; it only decides what the evidence supports. @@ -1252,6 +1254,7 @@ def build_parser() -> argparse.ArgumentParser: # The supervisor's own optional bounds, for a test or a deliberately finite run. hosted.add_argument("--max-segments", type=int) hosted.add_argument("--deadline", type=float) + hosted.add_argument("--launch-id") service.set_defaults(handler=cmd_service) doctor = subparsers.add_parser("doctor") diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 0c51e96..0b5c59a 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -162,6 +162,16 @@ def claim(self, socket_path, record: dict, *, shared: bool = False): pid in the record is not permission to take it. """ self.prepare() + # A stopped registration naming a DIFFERENT store is still that store's registration. + # Taking the free lock and overwriting the record would erase the only evidence that + # two stores have served this socket, which is the duplicate this registry exists to + # surface. Replacing it has to be deliberate. + existing = self.read(socket_path) + if (existing and record.get("storeId") and existing.get("storeId") + and existing["storeId"] != record["storeId"] + and not record.get("takeover")): + return {"ok": False, "reason": "scope_registered_to_other_store", + "held_by": existing, "scopeKey": self.key(socket_path)} path = self.root / f"{self.key(socket_path)}.lock" handle = open(path, "a+") try: @@ -377,6 +387,7 @@ def __init__(self, selection, *, socket_path=None, scope=None, store_id=None): self.selection = selection self.socket_path = socket_path self.store_id = store_id + self.launch_id = None if scope is None: root, authority = resolve_scope_root() scope = ScopeRegistry(root, authority) @@ -403,6 +414,7 @@ def new_record(self, *, pid, token=None, worker_pid=None) -> dict: return { "pid": pid, "startTicks": start_ticks(pid), "bootId": boot_id(), "workerPid": worker_pid, "token": token, + "launchId": self.launch_id, "storeId": self.store_id, "installationId": self.installation_id, "stateDir": str(self.selection.path), "socketPath": self.socket_path, "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), @@ -423,19 +435,24 @@ def ownership(self, record=None): mismatches = [] if record.get("bootId") not in (None, boot_id()): mismatches.append("recorded before a different boot") + if record.get("installationId") != self.installation_id: + mismatches.append("another installation owns it") + if self.store_id is not None and record.get("storeId") not in (None, self.store_id): + mismatches.append("it is using a different store") ticks = start_ticks(record["pid"]) + if mismatches: + # Decided BEFORE the start-time question. Installation, store and boot each prove + # the record is foreign on their own, and answering unverifiable would discard a + # definite answer in favour of an uncertain one. + return FOREIGN, handle, "; ".join(mismatches) if record.get("startTicks") is None or ticks is None: - # Without a start time a pid is just a number, and whatever holds it now would - # pass. Unverifiable is the honest answer, and stop refuses on it. + # Nothing proves it foreign, and without a start time a pid is just a number that + # anything could be holding now. Unverifiable is the honest answer; stop refuses. return UNVERIFIABLE, handle, ( "no start time is available for this pid, so identity cannot be established" ) if ticks != record["startTicks"]: mismatches.append("the pid was reused by a different process") - if record.get("installationId") != self.installation_id: - mismatches.append("another installation owns it") - if self.store_id is not None and record.get("storeId") not in (None, self.store_id): - mismatches.append("it is using a different store") if mismatches: return FOREIGN, handle, "; ".join(mismatches) return OURS, handle, None @@ -643,7 +660,8 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl if live: return {"ok": False, "reason": "scope_owned_by_other_store", "conflicts": live} - before = (self.record() or {}).get("pid") + launch = uuid.uuid4().hex + self.launch_id = launch child = (launcher or self.default_launcher)( self, allow_isolated=allow_isolated, max_ticks=max_ticks, deadline=deadline, segment_seconds=segment_seconds, max_segments=max_segments, @@ -651,7 +669,10 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl deadline_at = time.monotonic() + timeout while time.monotonic() < deadline_at: record = self.record() or {} - if record.get("pid") and record.get("pid") != before and self.lock_is_held(): + # Matched on the launch id, not on the pid changing. After a crash the OS can + # hand the replacement the very pid the stale record already names, and waiting + # for a different number would time out on a service that is running fine. + if record.get("launchId") == launch and self.lock_is_held(): return {"ok": True, "reason": None, "pid": record["pid"], "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), "status": self.status()} @@ -693,6 +714,11 @@ def default_launcher(self, service, *, allow_isolated, max_ticks=None, deadline= # The ALREADY RESOLVED absolute root, so a relative override cannot resolve # differently in the child, and the child cannot land in another registry. environment[SCOPE_ENV] = str(self.scope.root) + # The transport ledger resolves from the environment, not from --state, so a managed + # launch forwarding only the flag would inherit the store/ledger split. + environment["CODEX_SESSION_RELAY_STATE"] = str(self.selection.path) + if self.launch_id: + argv += ["--launch-id", self.launch_id] self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: return subprocess.Popen( @@ -738,6 +764,7 @@ def spawn_worker(self, *, lock_fd, scope_fd, token, segment_seconds, allow_isola environment = dict(os.environ) if self.scope.authority == ISOLATED: environment[SCOPE_ENV] = str(self.scope.root) + environment["CODEX_SESSION_RELAY_STATE"] = str(self.selection.path) pass_fds = tuple(fd for fd in (lock_fd, scope_fd) if fd is not None) with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: return subprocess.Popen( @@ -829,7 +856,11 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= sleeper(wait) finally: if self.socket_path: - self.scope.release(self.socket_path) + # shared=True: this descriptor is the one every worker inherited, and + # LOCK_UN through it would release the scope for all of them. After an + # exception a worker may still be alive, and unlocking would let another + # state directory claim this socket beside the orphan. + self.scope.release(self.socket_path, shared=True) current = self.record() or {} self.write_record(dict(current, pid=None, workerPid=None, stoppedAt=_now())) return {"ok": True, "reason": None, "segments": segments, diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index c80a60f..0dc18b5 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -755,7 +755,12 @@ def compare_store(store: dict, *, expect_store=None, expect_inode=None, nonce=No else: reasons.append((PROVEN, "same device and inode")) if nonce is not None: - if nonce.get("found"): + if nonce.get("readable") is False: + # Not being able to read is not the same as the nonce being absent. Calling it a + # mismatch would tell an operator two participants use different stores when the + # truth is that this one merely could not look. + reasons.append((UNPROVEN, f"the nonce could not be read here: {nonce.get('detail')}")) + elif nonce.get("found"): reasons.append((PROVEN, "a nonce written by another participant is readable here")) else: reasons.append((MISMATCH, "a nonce written by another participant is not here")) diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index d501434..8012848 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -132,6 +132,33 @@ def test_a_stopped_registration_on_another_store_is_still_a_conflict(self): self.assertEqual([c["storeId"] for c in conflicts], [first.store_id]) self.assertFalse(conflicts[0]["live"]) + def test_a_stopped_registration_is_not_silently_overwritten(self): + """Overwriting it would erase the only evidence two stores served one socket.""" + first = self.service("a") + child, _pid = self.holder(first) + child.terminate() + child.wait(timeout=10) + second = self.service("b") + refused = second.scope.claim( + second.socket_path, second.new_record(pid=os.getpid()), + ) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "scope_registered_to_other_store") + self.assertEqual(refused["held_by"]["storeId"], first.store_id) + + def test_a_definite_mismatch_beats_a_missing_start_time(self): + """Installation identity already proves foreign; unverifiable would discard that.""" + service = self.service("a") + child, _pid = self.holder(service) + record = service.record() + service.write_record( + dict(record, startTicks=None, installationId="someone-else"), + ) + refused = service.stop() + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("another installation", refused["detail"]) + self.assertIsNone(child.poll()) + def test_stop_refuses_a_process_another_installation_owns(self): service = self.service("a") child, pid = self.holder(service) diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index b460d8c..6628e87 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -254,6 +254,18 @@ def test_a_store_with_no_identity_is_never_proven_equal(self): compare_store({"storeId": None}, expect_store="whatever")["sameStore"], "unproven", ) + def test_an_unreadable_nonce_is_unproven_rather_than_a_mismatch(self): + """Not being able to look is not the same as looking and finding nothing.""" + unreadable = {"nonce": "abc", "found": False, "readable": False, + "detail": "OperationalError: unable to open database file"} + graded = compare_store({"storeId": "x"}, nonce=unreadable) + self.assertEqual(graded["sameStore"], "unproven") + self.assertIn("could not be read", graded["detail"]) + absent = {"nonce": "abc", "found": False, "readable": True, "detail": None} + self.assertEqual( + compare_store({"storeId": "x"}, nonce=absent)["sameStore"], "mismatch", + ) + class Probe(unittest.TestCase): def setUp(self): From 5d0924ffb18100ce06b2e25730657cee8e01b0dc Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 16:04:34 +0900 Subject: [PATCH 09/29] Require a live pid before start calls a launch successful start matched the child's record on the launch id and the daemon lock, and both of those stay true through supervision's cleanup: the supervisor clears the pid from the record while still inside the lock context, and only releases the lock afterwards. A launch that finished during startup, which a small --max-segments or an already expired deadline produces, therefore spent a moment looking exactly like a healthy one. A caller polling in that moment got ok with pid null and nothing running behind it. The launch-id match now also requires a recorded pid. A finished launch falls through to the ordinary paths instead, so the caller learns the child exited or that the launch never reported, which is what happened. The regression test parks a real child in that exact window: it takes the daemon lock and writes the record once, already cleared, so the parent cannot pass for the wrong reason by catching a live pid first. Against the previous condition it fails with ok true and pid null. A second test drives the same path with a pid in the record so the new requirement is not refusing every launch. --- .../src/codex_session_relay/service.py | 5 +- .../codex-session-relay/tests/test_service.py | 83 ++++++++++++++++++- 2 files changed, 85 insertions(+), 3 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 0b5c59a..4d7609b 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -672,7 +672,10 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl # Matched on the launch id, not on the pid changing. After a crash the OS can # hand the replacement the very pid the stale record already names, and waiting # for a different number would time out on a service that is running fine. - if record.get("launchId") == launch and self.lock_is_held(): + # The pid must still be there: supervision clears it on the way out while the + # daemon lock is not yet released, so a launch that already finished spends a + # moment matching the id and holding the lock with nothing left running. + if record.get("pid") and record.get("launchId") == launch and self.lock_is_held(): return {"ok": True, "reason": None, "pid": record["pid"], "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), "status": self.status()} diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 8012848..173514f 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -37,6 +37,7 @@ scope=ScopeRegistry(__import__("pathlib").Path({scopes!r}), "isolated"), store_id={store_id!r}, ) +service.launch_id = {launch!r} with owned_service(service, allow_isolated=True, require_intent=False) as record: # The record file IS the handshake. A pipe would add buffering and lifetime questions # that have nothing to do with what this test is about. @@ -44,6 +45,31 @@ time.sleep(0.05) """ +# A child parked in the window supervision leaves behind on its way out: the launch is +# recorded, the pid has been cleared, and the daemon lock is not released until the context +# exits. The record is written once, already cleared, so the parent cannot observe a live +# pid first and pass for the wrong reason. +FINISHING = """ +import os, sys, time +sys.path.insert(0, {src!r}) +from codex_session_relay.daemon import SingleInstance +from codex_session_relay.service import RelayService, ScopeRegistry +from codex_session_relay.store import resolve_state_dir +service = RelayService( + resolve_state_dir({state!r}), socket_path={socket!r}, + scope=ScopeRegistry(__import__("pathlib").Path({scopes!r}), "isolated"), + store_id={store_id!r}, +) +service.launch_id = {launch!r} +with SingleInstance(service.selection.path): + service.write_record(dict( + service.new_record(pid=os.getpid()), + pid=None, workerPid=None, stoppedAt="cleanup", + )) + while True: + time.sleep(0.05) +""" + class ServiceTestCase(unittest.TestCase): def setUp(self): @@ -73,12 +99,12 @@ def service(self, name="a", *, socket=SOCKET, scopes=None): scope=ScopeRegistry(Path(scopes or self.scopes), ISOLATED), store_id=store_id, ) - def holder(self, service): + def holder(self, service, *, launch=None): """Start a child that really holds the lock and the claim, and wait until it does.""" program = HOLDER.format( src=os.path.join(REPO, "src"), state=str(service.selection.path), socket=service.socket_path, scopes=str(service.scope.root), - store_id=service.store_id, + store_id=service.store_id, launch=launch, ) child = subprocess.Popen( [sys.executable, "-c", program], stdout=subprocess.DEVNULL, @@ -225,6 +251,59 @@ def test_a_stale_record_does_not_block_a_fresh_start(self): self.assertFalse(service.lock_is_held()) +class LaunchReporting(ServiceTestCase): + """start reports what the child managed to do, and a finished launch did nothing.""" + + def finishing_launcher(self): + """A launcher whose child reaches supervision's cleanup window and stays there.""" + def launcher(service, **_kw): + program = FINISHING.format( + src=os.path.join(REPO, "src"), state=str(service.selection.path), + socket=service.socket_path, scopes=str(service.scope.root), + store_id=service.store_id, launch=service.launch_id, + ) + child = subprocess.Popen( + [sys.executable, "-c", program], stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, text=True, + ) + self.children.append(child) + return child + return launcher + + def test_a_launch_that_already_finished_is_not_reported_as_running(self): + """A bound small enough to finish during startup - or an expired deadline. + + The launch id still matches and the daemon lock is still held, so matching on + those two alone hands the caller success for a service that is on its way out. + """ + service = self.service("a") + service.enable(actor="test") + + outcome = service.start( + allow_isolated=True, launcher=self.finishing_launcher(), timeout=2.0, + ) + + self.assertFalse(outcome["ok"], f"a cleared pid is not a running service: {outcome}") + self.assertIsNone(outcome.get("pid")) + self.assertEqual(outcome["reason"], "did_not_report") + + def test_a_live_launch_is_still_reported_as_running(self): + """The same path with a pid in the record, so the guard is not refusing everything.""" + service = self.service("b") + service.enable(actor="test") + started = {} + + def launcher(svc, **_kw): + child, pid = self.holder(svc, launch=svc.launch_id) + started["pid"] = pid + return child + + outcome = service.start(allow_isolated=True, launcher=launcher, timeout=10.0) + + self.assertTrue(outcome["ok"], outcome) + self.assertEqual(outcome["pid"], started["pid"]) + + class Intent(ServiceTestCase): def test_start_never_enables_a_service_that_was_never_configured(self): service = self.service("a") From 37118cd0d8f5fd55f184d7e1739a1f89ea648c3d Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 16:22:38 +0900 Subject: [PATCH 10/29] Answer the third review round on ownership, side effects and readiness Seven findings, all reachable, all reproduced by a test that fails against the code as it stood. ownership() returned none as soon as the recorded supervisor was gone, before reading the markers that prove a record foreign. stop() deliberately falls through to the worker in that case, because an orphaned worker is exactly what a stop has to reach - so a foreign installation whose supervisor had died handed us its live worker to signal. The installation, store and boot markers come from the record and need no live process, so they are read first and win. _stop_worker accepted a missing workerStartTicks and terminated whatever held the recorded pid. That is the rule the supervisor already has, and the worker was exempt from it: the regression test shows the old code returning "exited" for a worker it could not identify. Missing or unreadable start time is now unverifiable, and unverifiable is refused. daemon.json was rewritten in place at every worker boundary, so a concurrent stop could read the truncated middle, call a running supervisor absent, return not_running and leave its worker to finish the segment. The same window could hide the foreign markers and let a stop request be written for another installation. It is now written to a temporary file and renamed over the old one. start reported success from a record supervise() writes before on_start has run. Recovery happens in on_start and can outlast the poll interval or fail outright on the App Server connection, and every signal start matched on - pid, launch id, daemon lock - was already true. A readyAt written only after on_start returns is now required, so an initialisation that is still running, or that failed, is not a started service. doctor promised to construct no Store and then did, indirectly: the contents block counted rows through services.store, and Store.__init__ opens O_RDWR, enables WAL and runs the whole schema script. Pointing the diagnosis at an empty, legacy or unrelated readable relay.sqlite3 silently turned it into a relay database. Counts now go through a read-only connection, and the test asserts the file is still zero bytes with no sidecars afterwards. service status advertised project grouping in the operations contract and returned none; project_key had no caller outside its own unit test. It now groups the live relationships by project - assignments, active count, parents, issues - read through that same read-only path, so an offline status still creates nothing. The default state directory hashed the socket path as spelled while ScopeRegistry.key canonicalised it. A relative path or a symlink alias for one socket therefore chose a different store, and the registry then refused it as a foreign owner instead of letting it join the service already on that socket. Both canonicalise now. Relay suite 526 passed, 30 skipped. validate.py, contracts.py, check_operations_contract.py and git diff --check all exit 0. --- .../src/codex_session_relay/cli.py | 24 ++- .../src/codex_session_relay/service.py | 97 +++++++++-- .../src/codex_session_relay/store.py | 32 +++- .../codex-session-relay/tests/test_cli.py | 24 +++ .../codex-session-relay/tests/test_service.py | 164 +++++++++++++++++- .../codex-session-relay/tests/test_store.py | 28 +++ 6 files changed, 352 insertions(+), 17 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index d981ce8..9604b34 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -729,15 +729,29 @@ def _ledger_location(services) -> dict: def _contents(services, report) -> dict: """Counts, but only when the store can actually be opened for them.""" + from .store import read_only_rows + if not report["access"]["dbReadable"]: return {"available": False, "relationships": None, "openAttempts": None, "detail": "the database is not readable from this process"} - try: - relationships = services.store.one("SELECT COUNT(*) AS c FROM relationships")["c"] - open_attempts = len(services.reconciler.open_attempts()) - except Exception as error: # noqa: BLE001 - doctor reports, it does not fail + # Read through the probe's own read-only connection. services.store would construct a + # Store, and Store.__init__ opens O_RDWR, switches on WAL and runs the whole schema + # script - so asking doctor to COUNT rows in an empty, legacy or unrelated readable + # relay.sqlite3 quietly turned it into a relay database. Diagnosis writes nothing. + counted = read_only_rows( + services.selection, + "SELECT (SELECT COUNT(*) FROM relationships) AS relationships," + " (SELECT COUNT(*) FROM attempts a" + " JOIN deliveries d ON d.event_id = a.event_id" + " WHERE a.internal_state = 'in_flight'" + " OR (a.state = 'held_uncertain'" + " AND d.state IN ('held_uncertain','sending'))) AS open_attempts", + ) + if not counted["readable"] or counted["detail"] or not counted["rows"]: return {"available": False, "relationships": None, "openAttempts": None, - "detail": f"{type(error).__name__}: {error}"} + "detail": counted["detail"] or "the store could not be read"} + relationships = counted["rows"][0]["relationships"] + open_attempts = counted["rows"][0]["open_attempts"] return {"available": True, "relationships": relationships, "openAttempts": open_attempts, "detail": None} diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 4d7609b..b110a65 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -407,7 +407,17 @@ def record(self): def write_record(self, payload: dict) -> dict: self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) - self.record_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + # Replaced atomically. write_text truncates first, and _note rewrites this file on + # every worker boundary, so a concurrent stop could read the empty or half-written + # middle, call a running supervisor absent and return not_running without signalling + # its worker - or miss the foreign markers and write a stop request for someone else. + temporary = self.record_path.with_name(f".{self.record_path.name}.{os.getpid()}") + try: + temporary.write_text(json.dumps(payload, indent=2), encoding="utf-8") + os.replace(temporary, self.record_path) + except OSError: + temporary.unlink(missing_ok=True) + raise return payload def new_record(self, *, pid, token=None, worker_pid=None) -> dict: @@ -415,6 +425,8 @@ def new_record(self, *, pid, token=None, worker_pid=None) -> dict: "pid": pid, "startTicks": start_ticks(pid), "bootId": boot_id(), "workerPid": worker_pid, "token": token, "launchId": self.launch_id, + # Written by supervise once on_start has returned. Absent means "not serving yet". + "readyAt": None, "storeId": self.store_id, "installationId": self.installation_id, "stateDir": str(self.selection.path), "socketPath": self.socket_path, "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), @@ -427,18 +439,20 @@ def ownership(self, record=None): record = self.record() if record is None else record if not record or not record.get("pid"): return NONE, None, "no daemon record" + # Read from the record itself, so they survive the supervisor's death. A foreign + # installation whose supervisor is gone can still have a live worker recorded, and + # answering none there sent stop() down the orphan path and straight at it. + foreign = self._foreign_markers(record) handle = ProcessHandle(record["pid"]) if handle.already_gone: + if foreign: + return FOREIGN, handle, "; ".join(foreign) return NONE, handle, "the recorded process is gone" if not handle.usable: + if foreign: + return FOREIGN, handle, "; ".join(foreign) return UNVERIFIABLE, handle, handle.detail - mismatches = [] - if record.get("bootId") not in (None, boot_id()): - mismatches.append("recorded before a different boot") - if record.get("installationId") != self.installation_id: - mismatches.append("another installation owns it") - if self.store_id is not None and record.get("storeId") not in (None, self.store_id): - mismatches.append("it is using a different store") + mismatches = list(foreign) ticks = start_ticks(record["pid"]) if mismatches: # Decided BEFORE the start-time question. Installation, store and boot each prove @@ -457,6 +471,17 @@ def ownership(self, record=None): return FOREIGN, handle, "; ".join(mismatches) return OURS, handle, None + def _foreign_markers(self, record) -> list: + """What the record alone proves about who owns it, with no live process required.""" + markers = [] + if record.get("bootId") not in (None, boot_id()): + markers.append("recorded before a different boot") + if record.get("installationId") != self.installation_id: + markers.append("another installation owns it") + if self.store_id is not None and record.get("storeId") not in (None, self.store_id): + markers.append("it is using a different store") + return markers + def lock_is_held(self) -> bool: """Probed by trying to take it: the lock is the only honest liveness signal.""" path = self.selection.path / DAEMON_LOCK @@ -528,8 +553,47 @@ def status(self) -> dict: "lastExit": (record or {}).get("lastExit"), "nextRestartAt": (record or {}).get("nextRestartAt"), "conflicts": self.conflicts(), + "projects": self.projects(), } + def projects(self) -> dict: + """Which projects this one service is carrying, read without opening a Store. + + Grouping only. The cross-delivery refusal is decided per assignment from its own + endpoints; this exists so an operator can see that a single supervisor is serving + several repositories rather than having to infer it. + """ + from .registry import project_key + from .store import read_only_rows + + answer = read_only_rows( + self.selection, + "SELECT relationship_id, issue_key, status, parent_task_id, parent_host_id," + " parent_cwd" + " FROM relationships" + " WHERE superseded_by IS NULL", + ) + if not answer["readable"]: + return {"available": False, "detail": answer["detail"], "projects": []} + grouped = {} + for row in answer["rows"]: + key = project_key({"parent": {"cwd": row["parent_cwd"], + "hostId": row["parent_host_id"]}}) + entry = grouped.setdefault( + key, {"project": key, "assignments": 0, "active": 0, "parents": set(), + "issues": set()}, + ) + entry["assignments"] += 1 + entry["active"] += 1 if row["status"] == "active" else 0 + entry["parents"].add(row["parent_task_id"]) + entry["issues"].add(row["issue_key"]) + projects = [ + {"project": e["project"], "assignments": e["assignments"], "active": e["active"], + "parents": sorted(e["parents"]), "issues": sorted(e["issues"])} + for e in sorted(grouped.values(), key=lambda e: e["project"]) + ] + return {"available": True, "detail": answer["detail"], "projects": projects} + def conflicts(self) -> list: if not self.socket_path: return [] @@ -612,7 +676,13 @@ def _stop_worker(self, record, *, timeout, grace) -> str: if handle.already_gone or not handle.usable: return "gone" if handle.already_gone else "unverifiable" ticks = (record or {}).get("workerStartTicks") - if ticks is not None and start_ticks(pid) != ticks: + current = start_ticks(pid) + if ticks is None or current is None: + # The same rule the supervisor gets. Without a recorded start time the worker + # pid is just a number, and a crashed worker whose number was reused would + # put SIGTERM into an unrelated process. + return "unverifiable" + if current != ticks: return "gone" return self._terminate(handle, timeout=timeout, grace=grace) finally: @@ -675,7 +745,10 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl # The pid must still be there: supervision clears it on the way out while the # daemon lock is not yet released, so a launch that already finished spends a # moment matching the id and holding the lock with nothing left running. - if record.get("pid") and record.get("launchId") == launch and self.lock_is_held(): + # readyAt is written only after on_start returns, so recovery that is still + # running - or about to fail on an App Server connection - is not success either. + if (record.get("pid") and record.get("readyAt") + and record.get("launchId") == launch and self.lock_is_held()): return {"ok": True, "reason": None, "pid": record["pid"], "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), "status": self.status()} @@ -819,6 +892,10 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= try: if on_start is not None: on_start() + # Only now is this service serving. Recovery runs before any worker can send, + # and a caller told "started" while it was still in flight would go on to use + # a service that might yet fail to initialise at all. + self._note(readyAt=_now()) started = time.monotonic() while True: if max_segments is not None and len(segments) >= max_segments: diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 0dc18b5..f042f71 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -441,7 +441,13 @@ def _now_iso() -> str: def socket_scope(socket_path) -> str: if not socket_path: return "default" - return hashlib.sha256(str(Path(socket_path).expanduser()).encode()).hexdigest()[:16] + # Canonicalised the same way ScopeRegistry.key does. Hashing the spelling as supplied + # gave a relative path and a symlink alias for one socket two different default state + # directories, so the second invocation opened a different store and was then refused + # by the scope registry as a foreign owner instead of joining the service already there. + return hashlib.sha256( + str(Path(socket_path).expanduser().absolute().resolve()).encode() + ).hexdigest()[:16] @dataclass(frozen=True) @@ -703,6 +709,30 @@ def probe(selection: StateSelection) -> dict: return {"stateSelection": selection.to_record(), "store": store, "access": access} +def read_only_rows(selection: StateSelection, sql: str, params=()) -> dict: + """Answer a question about the store without creating or migrating one. + + Store.__init__ opens the file O_RDWR, switches on WAL and runs the whole schema script, + so any command that reaches for it to READ leaves a fully formed relay database behind. + For diagnosis that is a side effect the command promised not to have: pointing it at an + empty, legacy or unrelated file would silently adopt it. Every error becomes a field. + """ + db_path = selection.db_path + try: + connection = sqlite3.connect(f"{db_path.as_uri()}?mode=ro", uri=True, timeout=5) + except (OSError, sqlite3.Error, ValueError) as error: + return {"readable": False, "rows": [], + "detail": f"{type(error).__name__}: {error}"} + try: + connection.row_factory = sqlite3.Row + rows = [dict(row) for row in connection.execute(sql, params).fetchall()] + except sqlite3.Error as error: + connection.close() + return {"readable": True, "rows": [], "detail": f"{type(error).__name__}: {error}"} + connection.close() + return {"readable": True, "rows": rows, "detail": None} + + def nonce_lookup(selection: StateSelection, nonce: str) -> dict: """Look for a challenge nonce read-only, so a comparison never writes to the store.""" db_path = selection.db_path diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index ae91823..40db7e4 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -275,6 +275,30 @@ def test_doctor_answers_for_a_state_directory_that_does_not_exist_yet(self): # The old doctor built a Store first, which created the directory it was asked about. self.assertFalse(os.path.exists(absent)) + def test_doctor_does_not_turn_an_unrelated_file_into_a_relay_database(self): + """The directory existing was not the whole side effect; counting rows was too. + + A readable relay.sqlite3 sent the contents block through a real Store, and + Store.__init__ opens O_RDWR, switches on WAL and runs the entire schema script. An + empty, legacy or unrelated file was quietly adopted by the command that promised to + do nothing but look. + """ + state = os.path.join(self.tmp, "borrowed") + os.makedirs(state) + target = os.path.join(state, "relay.sqlite3") + open(target, "w").close() + + report = self.cli("doctor", state=state) + + self.assertEqual(os.path.getsize(target), 0, "doctor wrote a schema into it") + self.assertEqual( + sorted(os.listdir(state)), ["relay.sqlite3"], "no WAL or shm sidecar either", + ) + self.assertTrue(report["access"]["dbExists"]) + self.assertFalse(report["contents"]["available"], + "and it says so rather than inventing counts") + self.assertIsNotNone(report["contents"]["detail"]) + def test_doctor_names_the_rule_that_chose_the_directory(self): chosen = os.path.join(self.tmp, "chosen") by_env = self.cli("doctor", env={"CODEX_SESSION_RELAY_STATE": chosen}) diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 173514f..edb96c2 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -39,6 +39,8 @@ ) service.launch_id = {launch!r} with owned_service(service, allow_isolated=True, require_intent=False) as record: + # What supervise() writes once on_start has returned: this child is serving. + service._note(readyAt="2026-01-01T00:00:00Z") # The record file IS the handshake. A pipe would add buffering and lifetime questions # that have nothing to do with what this test is about. while True: @@ -64,7 +66,7 @@ with SingleInstance(service.selection.path): service.write_record(dict( service.new_record(pid=os.getpid()), - pid=None, workerPid=None, stoppedAt="cleanup", + readyAt="2026-01-01T00:00:00Z", pid=None, workerPid=None, stoppedAt="cleanup", )) while True: time.sleep(0.05) @@ -304,6 +306,81 @@ def launcher(svc, **_kw): self.assertEqual(outcome["pid"], started["pid"]) +class ForeignWorkers(ServiceTestCase): + """A dead supervisor does not make its installation's worker ours to signal.""" + + def test_a_foreign_record_whose_supervisor_died_still_protects_its_worker(self): + """The gone check answered before the foreign markers were ever read. + + stop() deliberately falls through to the worker when the supervisor is gone, because + that orphan is exactly what a stop has to reach. Classifying a foreign record as + none first aimed that fall-through at another installation's worker. + """ + service = self.service("a") + child, worker_pid = self.holder(service) + record = service.record() + service.write_record(dict( + record, installationId="someone-else", workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + real = os.pidfd_open + + def supervisor_is_gone(pid, *args, **kwargs): + if pid == record["pid"]: + raise ProcessLookupError(f"supervisor {pid} is gone") + return real(pid, *args, **kwargs) + + with mock.patch.object(os, "pidfd_open", supervisor_is_gone): + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("another installation", refused["detail"]) + self.assertEqual(refused["worker"], "untouched") + self.assertIsNone(child.poll(), "a foreign worker must not be signalled") + self.assertFalse( + service.stop_request_path.exists(), + "a refused stop must not leave a request that halts another installation", + ) + + def test_a_worker_with_no_recorded_start_time_is_unverifiable_not_killable(self): + """The rule the supervisor already had. A reused worker pid is an unrelated process.""" + service = self.service("a") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), workerPid=worker_pid, workerStartTicks=None, + )) + + self.assertEqual( + service._stop_worker(service.record(), timeout=1.0, grace=0.05), "unverifiable", + ) + self.assertIsNone(child.poll(), "an unverifiable worker must not be signalled") + + +class RecordDurability(ServiceTestCase): + def test_a_record_is_replaced_atomically_rather_than_truncated(self): + """_note rewrites this file at every worker boundary; a reader must never see half. + + A concurrent stop reading the truncated middle would call a running supervisor + absent and return not_running without ever signalling its worker. + """ + service = self.service("a") + service.write_record(service.new_record(pid=os.getpid())) + before = service.record_path.stat() + + service._note(restarts=7) + + self.assertEqual(service.record()["restarts"], 7) + self.assertNotEqual( + service.record_path.stat().st_ino, before.st_ino, + "an in-place rewrite is the truncation window; the file must be replaced", + ) + self.assertEqual( + [p.name for p in service.selection.path.glob(".daemon.json.*")], [], + "no temporary file is left behind", + ) + + class Intent(ServiceTestCase): def test_start_never_enables_a_service_that_was_never_configured(self): service = self.service("a") @@ -505,6 +582,91 @@ def test_supervising_a_disabled_service_is_refused(self): service.supervise(allow_isolated=True, spawn=lambda **_k: FakeWorker(0)) self.assertEqual(caught.exception.reason, "service_disabled") + def test_a_launch_is_not_ready_until_initialisation_has_returned(self): + """start matched a record written before on_start ran. + + Recovery happens in on_start, and it can take longer than the poll interval or fail + outright on the App Server connection. Everything start matched on - the pid, the + launch id, the daemon lock - is already true while that is still in flight. + """ + service = self.service("a") + service.enable(actor="test") + service.launch_id = "launch-under-test" + seen = {} + + def on_start(): + seen["record"] = service.record() + + outcome, _launches, _slept = self.supervised(service, codes=[0], on_start=on_start) + + self.assertTrue(outcome["ok"], outcome) + self.assertEqual(seen["record"]["launchId"], "launch-under-test") + self.assertIsNotNone(seen["record"]["pid"], "the record start polls was already there") + self.assertIsNone( + seen["record"]["readyAt"], + "initialisation had not returned, so this is not a started service", + ) + self.assertIsNotNone(service.record()["readyAt"], "and it is ready afterwards") + + def test_a_failed_initialisation_never_becomes_ready(self): + service = self.service("a") + service.enable(actor="test") + + def on_start(): + raise RuntimeError("the App Server connection failed") + + with self.assertRaises(RuntimeError): + self.supervised(service, codes=[0], on_start=on_start) + self.assertIsNone(service.record()["readyAt"]) + + +class Projects(ServiceTestCase): + """The operations contract says status groups by project; it has to actually do it.""" + + def assignment(self, service, name, *, cwd, issue, status="active"): + from codex_session_relay.store import Store + + store = Store(service.selection.db_path) + try: + store.db.execute( + "INSERT INTO relationships (relationship_id, issue_key, status," + " parent_task_id, parent_host_id, parent_cwd, child_task_id, child_host_id," + " execution_generation, artifact_roots, allowed_recipients, created_at," + " updated_at) VALUES (?,?,?,?,?,?,?,?,1,'[]','[]','now','now')", + (f"rel-{name}", issue, status, f"01parent-{name}", "host", + cwd, f"01child-{name}", "host"), + ) + finally: + store.close() + + def test_status_groups_a_shared_service_by_project(self): + service = self.service("a") + self.assignment(service, "one", cwd="/code/alpha", issue="ALPHA-1") + self.assignment(service, "two", cwd="/code/alpha", issue="ALPHA-2", status="paused") + self.assignment(service, "three", cwd="/code/beta", issue="BETA-1") + + projects = service.status()["projects"] + + self.assertTrue(projects["available"], projects) + self.assertEqual([p["project"] for p in projects["projects"]], + ["/code/alpha", "/code/beta"]) + alpha = projects["projects"][0] + self.assertEqual(alpha["assignments"], 2) + self.assertEqual(alpha["active"], 1, "a paused assignment is carried but not active") + self.assertEqual(alpha["issues"], ["ALPHA-1", "ALPHA-2"]) + self.assertEqual(len(alpha["parents"]), 2) + + def test_a_store_that_does_not_exist_is_reported_rather_than_created(self): + """status is an offline command; it must not bring a store into being to answer.""" + service = self.service("a") + service.selection.db_path.unlink() + + projects = service.status()["projects"] + + self.assertFalse(projects["available"]) + self.assertEqual(projects["projects"], []) + self.assertFalse(service.selection.db_path.exists(), "asking must not create it") + class SupervisedWorker(ServiceTestCase): """A worker adopts what its supervisor holds, or it is refused outright.""" diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 6628e87..ab3c346 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -162,6 +162,34 @@ def test_a_relative_flag_resolves_to_the_same_store_as_its_absolute_form(self): self.assertEqual(relative["store"]["storeId"], absolute["store"]["storeId"]) self.assertEqual(relative["store"]["inode"], absolute["store"]["inode"]) + def test_two_spellings_of_one_socket_choose_the_same_default_store(self): + """The scope registry canonicalises the socket; this hash used to take it verbatim. + + With no --state and no environment override, an alias for the socket produced a + different default state directory and therefore a different store. The registry then + refused the second invocation as a foreign owner rather than letting it join the + service already running on that socket. + """ + real = os.path.join(self.tmp, "run", "app-server.sock") + os.makedirs(os.path.dirname(real)) + open(real, "w").close() + alias_dir = os.path.join(self.tmp, "alias") + os.symlink(os.path.join(self.tmp, "run"), alias_dir) + + canonical = resolve_state_dir(None, real) + through_symlink = resolve_state_dir(None, os.path.join(alias_dir, "app-server.sock")) + + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = resolve_state_dir(None, os.path.join("run", "app-server.sock")) + finally: + os.chdir(here) + + self.assertEqual(through_symlink.socket_scope, canonical.socket_scope) + self.assertEqual(relative.socket_scope, canonical.socket_scope) + self.assertEqual(through_symlink.path, canonical.path) + self.assertEqual(relative.path, canonical.path) class Identity(unittest.TestCase): def setUp(self): From ca4f3f6635dcee49785efa7a9d82cbb9eaca3eeb Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 16:47:08 +0900 Subject: [PATCH 11/29] Answer the fourth review round on abandoned children and unreadable answers Six findings, three of them about a failure being reported as an absence. start returned did_not_report and walked away from the child it had launched. A child that is merely slow comes up after the caller was told the start failed, and it holds the daemon lock and the scope claim while it does - against the retry the caller is about to make. The child this call started is now terminated and reaped, and the outcome says which. service.json was rewritten in place. read() treats an unreadable document as not configured, which reads as disabled, and the supervisor re-reads intent at every worker boundary: a reader landing in the truncated middle would stop a service its owner had left enabled. Replaced atomically, like the daemon record. supervise cleared workerPid unconditionally on the way out, including after an exception between spawn and wait. The worker still holds the locks it inherited, so erasing the only identity a stop can aim at left it delivering for the rest of its segment while status reported not_running. The identity is kept unless the child was actually waited on. The restart delay was not clamped to the supervisor's own bound the way a worker segment already is. The regression test shows a 0.01 second deadline waiting 240 seconds. projects treated a query that failed as an inventory that is empty. The file opening is not the question; whether the query answered is. An inventory that could not be read now reports unavailable with its detail. nonce_lookup did the same thing one layer down, and worse: a locked, malformed or momentarily unavailable database opened and then failed the query, and calling that readable turned "we could not look" into "it is not there", which compare_store grades as a definite store mismatch. doctor --expect-nonce would have claimed two participants use different stores on a transient read failure. Relay suite 533 passed, 30 skipped. --- .../src/codex_session_relay/service.py | 55 ++++++++- .../src/codex_session_relay/store.py | 5 +- .../codex-session-relay/tests/test_service.py | 112 ++++++++++++++++++ .../codex-session-relay/tests/test_store.py | 23 ++++ 4 files changed, 191 insertions(+), 4 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index b110a65..06f0268 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -376,7 +376,17 @@ def read(self) -> dict: def write(self, *, enabled: bool, actor: str) -> dict: self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) payload = {"enabled": bool(enabled), "changedAt": _now(), "changedBy": actor} - self.path.write_text(json.dumps(payload, indent=2), encoding="utf-8") + # Replaced atomically, for the same reason the daemon record is. read() treats an + # unreadable document as not configured and therefore disabled, and the supervisor + # re-reads intent at every worker boundary - so a reader landing in the truncated + # middle of this write would stop a service its owner had left enabled. + temporary = self.path.with_name(f".{self.path.name}.{os.getpid()}") + try: + temporary.write_text(json.dumps(payload, indent=2), encoding="utf-8") + os.replace(temporary, self.path) + except OSError: + temporary.unlink(missing_ok=True) + raise return dict(payload, configured=True) @@ -575,6 +585,10 @@ def projects(self) -> dict: ) if not answer["readable"]: return {"available": False, "detail": answer["detail"], "projects": []} + if answer["detail"]: + # The file opened and the query did not. An inventory we could not read is not an + # empty inventory, and reporting available with no projects says it is. + return {"available": False, "detail": answer["detail"], "projects": []} grouped = {} for row in answer["rows"]: key = project_key({"parent": {"cwd": row["parent_cwd"], @@ -756,7 +770,26 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl return {"ok": False, "reason": "child_exited", "exitCode": child.returncode, "log": self._log_tail()} time.sleep(poll) - return {"ok": False, "reason": "did_not_report", "log": self._log_tail()} + # A launch that never reported is not a launch that can be left alone. It may still + # be initialising and would come up AFTER the caller was told it failed, holding the + # locks against the retry the caller is about to make. + return {"ok": False, "reason": "did_not_report", "log": self._log_tail(), + "child": self._abandon(child, timeout=timeout)} + + def _abandon(self, child, *, timeout: float) -> str: + """Stop a child this call started and could not confirm, and reap it.""" + if getattr(child, "poll", lambda: None)() is not None: + return "exited" + for step in (getattr(child, "terminate", None), getattr(child, "kill", None)): + if step is None: + continue + try: + step() + child.wait(timeout=max(1.0, timeout / 2)) + return "terminated" + except Exception: # noqa: BLE001 - a child we cannot reach is reported, not raised + continue + return "still_running" def _log_tail(self, lines: int = 20) -> str: try: @@ -878,6 +911,7 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= self.clear_stop_request() token = uuid.uuid4().hex segments, failures, degraded = [], 0, None + outstanding = None with SingleInstance(self.selection.path, shared=True) as lock: scope_fd = None if self.socket_path: @@ -920,7 +954,9 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= allow_isolated=allow_isolated, ) self._note(workerPid=child.pid, workerStartTicks=start_ticks(child.pid)) + outstanding = child code = child.wait() + outstanding = None self._note(workerPid=None, workerStartTicks=None, lastExit=code, restarts=len(segments) + 1) segments.append(code) @@ -932,6 +968,11 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= if self.stop_requested() or not self.intent.read()["enabled"]: break wait = policy.restart_delay_for(failures) + if deadline is not None: + # Clamped the same way a worker segment is. An unclamped delay - up to + # five minutes after repeated failures - outlives the supervisor's own + # bound, so a short deadline took minutes to return. + wait = max(0.0, min(wait, deadline - (time.monotonic() - started))) self._note(nextRestartAt=time.time() + wait) sleeper(wait) finally: @@ -942,7 +983,15 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= # state directory claim this socket beside the orphan. self.scope.release(self.socket_path, shared=True) current = self.record() or {} - self.write_record(dict(current, pid=None, workerPid=None, stoppedAt=_now())) + cleared = dict(current, pid=None, stoppedAt=_now()) + if outstanding is None: + cleared["workerPid"] = None + cleared["workerStartTicks"] = None + # Otherwise the worker was never waited on - an exception between spawn and + # wait - and it still holds the inherited locks. Erasing its pid and start + # time would leave a stop with nothing to aim at, so the orphan would keep + # delivering for the rest of its segment while status reported not_running. + self.write_record(cleared) return {"ok": True, "reason": None, "segments": segments, "consecutiveFailures": failures, "degraded": degraded} diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index f042f71..f6f3fb6 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -747,7 +747,10 @@ def nonce_lookup(selection: StateSelection, nonce: str) -> dict: "SELECT written_by, written_at FROM store_challenge WHERE nonce = ?", (nonce,) ).fetchone() except sqlite3.Error as error: - return {"nonce": nonce, "found": False, "readable": True, + # NOT readable. A locked, malformed or momentarily unavailable database answers no + # question, and calling it readable turns "we could not look" into "it is not there", + # which compare_store then grades as a definite store mismatch. + return {"nonce": nonce, "found": False, "readable": False, "detail": f"{type(error).__name__}: {error}"} finally: connection.close() diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index edb96c2..f63314d 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -16,6 +16,7 @@ from unittest import mock from codex_session_relay import service as service_module +from codex_session_relay.policy import RetryPolicy from codex_session_relay.service import ( ISOLATED, PRODUCTION, ProcessHandle, RelayService, ScopeRegistry, ServiceRefused, installation_id, owned_service, production_scope_root, resolve_scope_root, @@ -305,6 +306,30 @@ def launcher(svc, **_kw): self.assertTrue(outcome["ok"], outcome) self.assertEqual(outcome["pid"], started["pid"]) + def test_a_launch_that_never_reports_is_not_left_running(self): + """Reporting failure and walking away leaves the locks held against the retry. + + A child that is merely slow can come up after the caller was told the start failed, + and it holds the daemon lock and the scope claim while it does. + """ + service = self.service("c") + service.enable(actor="test") + program = "import time\nwhile True:\n time.sleep(0.05)\n" + + def launcher(_svc, **_kw): + child = subprocess.Popen( + [sys.executable, "-c", program], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + self.children.append(child) + return child + + outcome = service.start(allow_isolated=True, launcher=launcher, timeout=0.5) + + self.assertFalse(outcome["ok"], outcome) + self.assertEqual(outcome["reason"], "did_not_report") + self.assertEqual(outcome["child"], "terminated") + self.assertIsNotNone(self.children[-1].poll(), "the child this call started is gone") class ForeignWorkers(ServiceTestCase): """A dead supervisor does not make its installation's worker ours to signal.""" @@ -380,6 +405,76 @@ def test_a_record_is_replaced_atomically_rather_than_truncated(self): "no temporary file is left behind", ) + def test_the_intent_file_is_replaced_atomically_too(self): + """read() calls an unreadable document not configured, which reads as disabled. + + The supervisor re-reads intent at every worker boundary, so a reader landing in the + truncated middle of this write stops a service its owner left enabled. + """ + service = self.service("a") + service.enable(actor="test") + before = service.intent.path.stat() + + service.intent.write(enabled=True, actor="owner") + + self.assertTrue(service.intent.read()["enabled"]) + self.assertNotEqual(service.intent.path.stat().st_ino, before.st_ino) + self.assertEqual( + [p.name for p in service.selection.path.glob(".service.json.*")], [], + ) + + +class SupervisorCleanup(ServiceTestCase): + """What the supervisor leaves behind when it does not get to finish.""" + + def test_a_worker_that_was_never_waited_on_keeps_its_identity(self): + """The cleanup cleared workerPid unconditionally, including after an exception. + + The worker still holds the daemon lock and the scope claim it inherited, so erasing + the only identity a stop can aim at leaves it delivering for the rest of its segment + while status reports not_running. + """ + service = self.service("a") + service.enable(actor="test") + worker = FakeWorker(0, pid=os.getpid()) + worker.wait = _explode + + with self.assertRaises(RuntimeError): + service.supervise(allow_isolated=True, spawn=lambda **_k: worker, + sleeper=lambda _s: None, max_segments=1) + + record = service.record() + self.assertIsNone(record["pid"], "the supervisor itself is gone") + self.assertEqual(record["workerPid"], worker.pid, + "but the orphan it left behind is still reachable") + self.assertIsNotNone(record["workerStartTicks"]) + + def test_a_clean_run_still_clears_the_worker(self): + service = self.service("b") + service.enable(actor="test") + service.supervise(allow_isolated=True, spawn=lambda **_k: FakeWorker(0), + sleeper=lambda _s: None, max_segments=1) + record = service.record() + self.assertIsNone(record["workerPid"]) + self.assertIsNone(record["workerStartTicks"]) + + def test_the_restart_delay_never_outlives_the_supervisors_own_bound(self): + """An unclamped delay made a 0.01-second deadline take seconds.""" + service = self.service("c") + service.enable(actor="test") + slept = [] + service.supervise( + allow_isolated=True, spawn=lambda **_k: FakeWorker(1), + sleeper=slept.append, max_segments=4, deadline=0.01, + policy=RetryPolicy(restart_base_seconds=30.0, restart_backoff_max_seconds=300.0), + ) + self.assertTrue(slept, "a failed segment does wait before its replacement") + self.assertLess( + max(slept), 30.0, + "the unclamped delay is the policy interval, which outlives the whole bound", + ) + self.assertTrue(all(value <= 0.01 for value in slept), slept) + class Intent(ServiceTestCase): def test_start_never_enables_a_service_that_was_never_configured(self): @@ -487,6 +582,12 @@ def wait(self): return self.returncode +def _explode(): + """A worker the supervisor never gets an exit code from.""" + raise RuntimeError("the supervisor died between spawn and wait") + + + class Supervision(ServiceTestCase): def supervised(self, service, *, codes, **kwargs): """Run the supervisor over a scripted sequence of worker exits.""" @@ -667,6 +768,17 @@ def test_a_store_that_does_not_exist_is_reported_rather_than_created(self): self.assertEqual(projects["projects"], []) self.assertFalse(service.selection.db_path.exists(), "asking must not create it") + def test_an_inventory_that_cannot_be_queried_is_not_an_empty_one(self): + """The file opened and the query did not. Reporting available says there are none.""" + service = self.service("a") + service.selection.db_path.write_bytes(b"") + + projects = service.status()["projects"] + + self.assertFalse(projects["available"]) + self.assertIsNotNone(projects["detail"]) + self.assertEqual(projects["projects"], []) + class SupervisedWorker(ServiceTestCase): """A worker adopts what its supervisor holds, or it is refused outright.""" diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index ab3c346..c06e0b1 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -262,6 +262,29 @@ def test_a_nonce_written_after_the_copy_separates_them(self): "mismatch", ) + def test_a_nonce_query_that_fails_is_unreadable_rather_than_absent(self): + """Could not look is not is not there, and compare_store grades the difference. + + A locked, malformed or momentarily unavailable database opens and then fails the + query. Calling that readable turned it into a definite store mismatch, so + doctor --expect-nonce would claim two participants use different stores on the + strength of a transient read failure. + """ + broken = os.path.join(self.tmp, "broken") + os.makedirs(broken) + with open(os.path.join(broken, "relay.sqlite3"), "wb"): + pass + + answer = nonce_lookup(resolve_state_dir(broken), "any-nonce") + + self.assertFalse(answer["found"]) + self.assertFalse(answer["readable"], "the query failed; nothing was learned") + self.assertIsNotNone(answer["detail"]) + self.assertNotEqual( + compare_store(probe(resolve_state_dir(broken))["store"], nonce=answer)["sameStore"], + "mismatch", + ) + def test_a_symlinked_directory_is_the_same_store(self): alias = os.path.join(self.tmp, "alias") os.symlink(self.a, alias) From f522993b503dca45601e193c383ded24a38304bd Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 17:09:16 +0900 Subject: [PATCH 12/29] Answer the fifth review round on shared intent, stale worker reads and a fresh store service disable wrote enabled=false and only then discovered that the recorded supervisor belongs to another installation. service.json is shared by everything pointed at this state directory and a supervisor re-reads intent at every worker boundary, so the refused disable still shut that supervisor down at its next boundary. A refusal that has an effect is not a refusal. Ownership is established first now, and a foreign or unverifiable owner leaves the intent exactly as it was. stop read the daemon record once and used that worker pid after terminating the supervisor. A supervisor replacing its worker inside that window meant the pid named a worker that had already exited, so stop reported success while the replacement - holding the inherited locks - kept delivering. The record is re-read after the supervisor is gone. restart_delay_for had the same unbounded exponent the pre-send backoff did: a supervisor whose worker fails on every segment keeps counting, and past about a thousand failures the product is an integer too large to convert to a float, so the supervisor died of its own backoff instead of retrying at the cap. On a fresh state directory the probe that builds the service answers from a file that does not exist yet and reports no store id, so the daemon record and the durable scope registration were both written with a null id. ScopeRegistry's mismatch guard needs both ids present, so that registration could later be overwritten by a different store - losing the evidence that two stores served one socket, which is the thing the registry exists to keep. The supervisor now takes the real identity from the store it is about to use anyway. A worker whose PR_SET_PDEATHSIG could not be armed is recorded rather than refused. The getppid check beside it already closes the window that matters here, a supervisor that is ALREADY gone, and refusing outright would make the relay unusable on any host without prctl. What is genuinely lost is the later case: a supervisor crashing mid-segment will not have its worker signalled, so that worker runs to the end of its bounded segment holding the locks. Bounded, but real, and now visible in the record instead of assumed. Relay suite 533 passed, 30 skipped. --- .../src/codex_session_relay/cli.py | 21 ++++- .../src/codex_session_relay/policy.py | 6 ++ .../src/codex_session_relay/service.py | 23 ++++- .../codex-session-relay/tests/test_service.py | 87 +++++++++++++++++++ 4 files changed, 135 insertions(+), 2 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 9604b34..9cac26e 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -884,7 +884,19 @@ def refuse(reason, detail): if death["orphaned"]: refuse("supervisor_already_gone", f"parent is {death['parent']}, not the recorded supervisor {record.get('pid')}") - return {"lockFd": lock_fd, "scopeFd": scope_fd if scope_fd >= 0 else None} + if not death["armed"]: + # Recorded rather than refused. The getppid check above closes the window that + # matters here - a supervisor that is ALREADY gone - and refusing outright would make + # the relay unusable on any host without prctl. What is lost is the later case: if + # the supervisor crashes mid-segment the kernel will not signal this worker, so it + # runs to the end of its bounded segment holding the inherited locks. Bounded, but + # real, and an operator can see it in the record instead of assuming it is armed. + service.write_record(dict( + service.record() or record, + workerDeathSignal="unarmed", workerDeathSignalDetail=death.get("detail"), + )) + return {"lockFd": lock_fd, "scopeFd": scope_fd if scope_fd >= 0 else None, + "parentDeathSignal": "armed" if death["armed"] else "unarmed"} def cmd_service(services, args) -> dict: @@ -917,6 +929,13 @@ def _supervise(services, service, args) -> dict: from .service import ServiceRefused service.launch_id = getattr(args, "launch_id", None) + # The probe that built this service answers from a file that may not exist yet, so on a + # fresh state directory it reports no store id at all. Opening the store HERE is not the + # side effect doctor and status refuse: a supervisor is about to use it either way. It + # matters because the scope registration is written next, and ScopeRegistry's mismatch + # guard needs both ids to be present - a registration recorded with a null id could be + # overwritten later by a different store, losing the evidence two stores served one socket. + service.store_id = services.store.identity def recover(): # Establish what happened to anything in flight BEFORE a worker can send. Recovery diff --git a/packages/codex-session-relay/src/codex_session_relay/policy.py b/packages/codex-session-relay/src/codex_session_relay/policy.py index e3d6142..f215480 100644 --- a/packages/codex-session-relay/src/codex_session_relay/policy.py +++ b/packages/codex-session-relay/src/codex_session_relay/policy.py @@ -41,6 +41,12 @@ def restart_delay_for(self, consecutive_failures: int) -> float: """A clean segment waits the plain interval; repeated failure backs off and caps.""" if consecutive_failures <= 0: return self.restart_interval_seconds + # Bounded before the exponent is evaluated. A supervisor whose worker fails on every + # segment keeps counting, and past about a thousand failures base * 2 ** (n - 1) is an + # integer too large to convert to a float - so the supervisor would die of its own + # backoff instead of continuing to retry at the cap. + if consecutive_failures > 64: + return self.restart_backoff_max_seconds return min( self.restart_backoff_max_seconds, self.restart_base_seconds * (2 ** (consecutive_failures - 1)), diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 06f0268..f243aee 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -529,6 +529,22 @@ def enable(self, *, actor: str = "cli") -> dict: "intent": self.intent.write(enabled=True, actor=actor)} def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: + # Ownership BEFORE the intent write. service.json is shared by everything pointed at + # this state directory, and a foreign supervisor re-reads it at every worker + # boundary: writing enabled=false and only then discovering the supervisor is not + # ours refused the stop while still shutting that supervisor down at its next + # boundary. A refusal that still has an effect is not a refusal. + owner, handle, detail = self.ownership() + if handle is not None: + handle.close() + if owner in (FOREIGN, UNVERIFIABLE): + return {"ok": False, + "reason": "not_ours" if owner == FOREIGN else "ownership_unverifiable", + "detail": detail, "intent": self.intent.read(), + "stop": {"ok": False, "reason": "refused", "supervisor": "untouched", + "worker": "untouched"}, + "note": "intent is shared with the owner of this state directory and was" + " left unchanged"} written = self.intent.write(enabled=False, actor=actor) stopped = self.stop(actor=actor, timeout=timeout) # A stop that had nothing to stop is not a failure; the intent is what disable owns. @@ -642,7 +658,12 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) finally: if handle is not None: handle.close() - worker = self._stop_worker(record, timeout=timeout, grace=grace) + # Re-read AFTER the supervisor is gone. A supervisor replacing a worker between our + # first read and now means the pid we started with names a worker that has already + # exited, and stopping that one would report success while the replacement, which + # holds the inherited locks, is still delivering. + worker = self._stop_worker(self.record() or record, timeout=timeout, grace=grace) + record = self.record() or record supervisor_done = outcome in ("exited", "gone") worker_done = worker in ("exited", "gone") if record is not None: diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index f63314d..7912621 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -199,6 +199,39 @@ def test_stop_refuses_a_process_another_installation_owns(self): self.assertIn("another installation", refused["detail"]) self.assertIsNone(child.poll(), "a foreign process must not be signalled") + def test_disable_refuses_a_foreign_service_without_touching_shared_intent(self): + """service.json is shared by everything pointed at this state directory. + + Writing enabled=false and only then discovering the supervisor is foreign refused + the stop while still shutting that supervisor down at its next worker boundary, + because it re-reads intent there. A refusal that still has an effect is not one. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), installationId="someone-else")) + + refused = service.disable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertTrue( + service.intent.read()["enabled"], + "the owner's intent must survive a refused disable", + ) + self.assertIsNone(child.poll()) + self.assertFalse(service.stop_request_path.exists()) + + def test_disable_still_works_on_a_service_this_installation_owns(self): + service = self.service("b") + service.enable(actor="owner") + child, _pid = self.holder(service) + + disabled = service.disable(actor="owner") + + self.assertTrue(disabled["ok"], disabled) + self.assertFalse(service.intent.read()["enabled"]) + child.wait(timeout=10) def test_stop_refuses_a_recycled_pid(self): service = self.service("a") child, pid = self.holder(service) @@ -243,6 +276,47 @@ def test_stop_terminates_a_process_this_installation_owns(self): self.assertIsNone(service.record()["pid"]) self.assertFalse(service.lock_is_held()) + def test_stop_reaches_the_worker_the_record_names_now(self): + """A supervisor replacing a worker while stop runs left the first read stale. + + Stopping the worker that already exited reports success while its replacement, + which holds the inherited locks, is still delivering. + """ + service = self.service("a") + child, _pid = self.holder(service) + first = subprocess.Popen( + [sys.executable, "-c", "import time\nwhile True:\n time.sleep(0.05)\n"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(first) + first.terminate() + first.wait(timeout=10) + service.write_record(dict( + service.record(), workerPid=first.pid, workerStartTicks=1, + )) + replacement = subprocess.Popen( + [sys.executable, "-c", "import time\nwhile True:\n time.sleep(0.05)\n"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(replacement) + original = service._terminate + + def replace_then_terminate(handle, **kwargs): + # The supervisor swapping workers in the window stop() used to read across. + service.write_record(dict( + service.record(), workerPid=replacement.pid, + workerStartTicks=service_module.start_ticks(replacement.pid), + )) + return original(handle, **kwargs) + + with mock.patch.object(service, "_terminate", replace_then_terminate): + stopped = service.stop() + + child.wait(timeout=10) + self.assertIsNotNone( + replacement.poll(), "the worker the record names NOW is the one stop must reach", + ) + self.assertEqual(stopped["worker"], "exited") def test_a_stale_record_does_not_block_a_fresh_start(self): service = self.service("a") child, pid = self.holder(service) @@ -475,6 +549,19 @@ def test_the_restart_delay_never_outlives_the_supervisors_own_bound(self): ) self.assertTrue(all(value <= 0.01 for value in slept), slept) + def test_a_long_failure_streak_keeps_retrying_at_the_cap(self): + """The backoff built the product and clamped after, so it overflowed and killed the + supervisor it was meant to pace. + """ + policy = RetryPolicy() + ceiling = policy.restart_backoff_max_seconds + for failures in (1, 2, 10, 1024, 1025, 10 ** 6): + delay = policy.restart_delay_for(failures) + self.assertIsInstance(delay, (int, float)) + self.assertLessEqual(delay, ceiling) + self.assertEqual(policy.restart_delay_for(10 ** 6), ceiling) + self.assertLess(policy.restart_delay_for(1), policy.restart_delay_for(4)) + class Intent(ServiceTestCase): def test_start_never_enables_a_service_that_was_never_configured(self): From c6fc0d11f4b26c0aac954e6b93c9ac78d785b2e4 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 17:28:38 +0900 Subject: [PATCH 13/29] Answer the sixth review round on the record the worker must not write Three findings, two of them defects the last round introduced. An unarmed worker wrote its own note into daemon.json. That record belongs to the supervisor, which rewrites the whole document at every worker boundary, so a whole-document write from the worker races it and can erase the very workerPid a stop needs to reach an orphan - defeating the purpose of recording the unarmed state. The note goes to the daemon log now, which is append-only. ownership() returned none as soon as the recorded supervisor pid was gone, and the previous round made that reachable in a new way: supervise now deliberately keeps the worker identity when it could not confirm the child exited. That is exactly the orphan record stop() is supposed to reach, so a foreign one pointed the orphan path at another installation's worker. A record with no supervisor but a live worker identity is classified before none is returned. restart_delay_for's overflow bound was a fixed step count, which is wrong for any policy whose base is small enough to need more doublings than that to reach its ceiling: such a supervisor jumped straight to the cap while its own backoff still had room. The bound is derived from the policy's own ratio now. Relay suite 537 passed, 30 skipped. --- .../src/codex_session_relay/cli.py | 12 +++-- .../src/codex_session_relay/policy.py | 19 ++++--- .../src/codex_session_relay/service.py | 8 +++ .../codex-session-relay/tests/test_service.py | 51 +++++++++++++++++++ 4 files changed, 80 insertions(+), 10 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 9cac26e..0a85fb7 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -891,10 +891,14 @@ def refuse(reason, detail): # the supervisor crashes mid-segment the kernel will not signal this worker, so it # runs to the end of its bounded segment holding the inherited locks. Bounded, but # real, and an operator can see it in the record instead of assuming it is armed. - service.write_record(dict( - service.record() or record, - workerDeathSignal="unarmed", workerDeathSignalDetail=death.get("detail"), - )) + # Appended to the log rather than written into daemon.json: the supervisor owns that + # record and rewrites it at every worker boundary, so a whole-document write from the + # worker would race it and could erase the workerPid a stop needs. + service.store_journal_note( + f"worker {os.getpid()} could not arm PR_SET_PDEATHSIG" + f" ({death.get('detail') or 'no detail'}); if the supervisor crashes this worker" + " runs to the end of its segment holding the inherited locks" + ) return {"lockFd": lock_fd, "scopeFd": scope_fd if scope_fd >= 0 else None, "parentDeathSignal": "armed" if death["armed"] else "unarmed"} diff --git a/packages/codex-session-relay/src/codex_session_relay/policy.py b/packages/codex-session-relay/src/codex_session_relay/policy.py index f215480..329caaf 100644 --- a/packages/codex-session-relay/src/codex_session_relay/policy.py +++ b/packages/codex-session-relay/src/codex_session_relay/policy.py @@ -39,14 +39,21 @@ class RetryPolicy: def restart_delay_for(self, consecutive_failures: int) -> float: """A clean segment waits the plain interval; repeated failure backs off and caps.""" + import math + if consecutive_failures <= 0: return self.restart_interval_seconds - # Bounded before the exponent is evaluated. A supervisor whose worker fails on every - # segment keeps counting, and past about a thousand failures base * 2 ** (n - 1) is an - # integer too large to convert to a float - so the supervisor would die of its own - # backoff instead of continuing to retry at the cap. - if consecutive_failures > 64: - return self.restart_backoff_max_seconds + base, ceiling = self.restart_base_seconds, self.restart_backoff_max_seconds + # Bounded before the exponent is evaluated, because a supervisor whose worker fails on + # every segment keeps counting and base * 2 ** (n - 1) eventually builds an integer too + # large to convert to a float - so the supervisor would die of its own backoff instead + # of retrying at the cap. The bound is DERIVED from this policy rather than fixed: a + # small base needs more doublings to reach its ceiling, and a constant would have sent + # a customized supervisor straight to the cap while its own backoff still had room. + if base <= 0 or ceiling <= base: + return ceiling + if consecutive_failures - 1 >= math.ceil(math.log2(ceiling / base)): + return ceiling return min( self.restart_backoff_max_seconds, self.restart_base_seconds * (2 ** (consecutive_failures - 1)), diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index f243aee..78058cb 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -448,6 +448,14 @@ def ownership(self, record=None): """ours | foreign | unverifiable | none, decided through a real process handle.""" record = self.record() if record is None else record if not record or not record.get("pid"): + if record and record.get("workerPid"): + # A supervisor that died leaving its worker alive keeps the worker identity + # on purpose, so this record still names something stop() will reach. It has + # to be classified first: answering none here sent the orphan path straight + # at another installation's worker. + foreign = self._foreign_markers(record) + if foreign: + return FOREIGN, None, "; ".join(foreign) return NONE, None, "no daemon record" # Read from the record itself, so they survive the supervisor's death. A foreign # installation whose supervisor is gone can still have a live worker recorded, and diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 7912621..b244736 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -442,6 +442,43 @@ def supervisor_is_gone(pid, *args, **kwargs): "a refused stop must not leave a request that halts another installation", ) + def test_an_orphaned_foreign_worker_is_still_not_ours_to_signal(self): + """The record supervise() now leaves behind: pid cleared, worker identity kept. + + Keeping the worker identity is what lets a stop reach an orphan, and the ownership + check returned none as soon as the supervisor pid was gone - so that orphan path + pointed straight at another installation's worker. + """ + service = self.service("a") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, installationId="someone-else", + workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertEqual(refused["worker"], "untouched") + self.assertIsNone(child.poll(), "a foreign orphan must not be signalled") + self.assertFalse(service.stop_request_path.exists()) + + def test_an_orphaned_worker_of_our_own_is_still_reachable(self): + """The guard must not refuse the case it was added to support.""" + service = self.service("b") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + stopped = service.stop() + + child.wait(timeout=10) + self.assertEqual(stopped["worker"], "exited") + def test_a_worker_with_no_recorded_start_time_is_unverifiable_not_killable(self): """The rule the supervisor already had. A reused worker pid is an unrelated process.""" service = self.service("a") @@ -562,6 +599,20 @@ def test_a_long_failure_streak_keeps_retrying_at_the_cap(self): self.assertEqual(policy.restart_delay_for(10 ** 6), ceiling) self.assertLess(policy.restart_delay_for(1), policy.restart_delay_for(4)) + def test_the_bound_follows_the_policy_rather_than_a_fixed_step_count(self): + """A small base needs more doublings, and a constant bound truncated its backoff.""" + # Deliberately extreme: this ratio needs about 70 doublings, so a fixed 64-step bound + # returns the ceiling while the policy's own backoff still has room. + fine = RetryPolicy(restart_base_seconds=1e-12, restart_backoff_max_seconds=1e9) + self.assertLess( + fine.restart_delay_for(65), fine.restart_backoff_max_seconds, + "this policy has not reached its own ceiling yet", + ) + self.assertLess(fine.restart_delay_for(65), fine.restart_delay_for(80)) + self.assertEqual(fine.restart_delay_for(10 ** 6), fine.restart_backoff_max_seconds) + for failures in (1, 65, 1025, 10 ** 6): + self.assertIsInstance(fine.restart_delay_for(failures), (int, float)) + class Intent(ServiceTestCase): def test_start_never_enables_a_service_that_was_never_configured(self): From 0752a69aefac9b4c0d0feab0cbbb1316e56b8f19 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 17:45:33 +0900 Subject: [PATCH 14/29] Answer the seventh review round on shared intent and an absent boot A record with no boot id was treated as compatible with every boot. A reboot resets the pid space and the /proc start-tick counter together, so without a recorded boot the two agreeing proves nothing: an unrelated process holding the old number passes both checks and would have been classified as ours and signalled. It is unverifiable now, and stop refuses. service enable overwrote the shared service.json without the ownership check disable already has. An owner who has just disabled a service whose supervisor is still exiting could have that reversed by another installation, leaving the service eligible to restart against its owner's stated intent. The check is scoped to a LIVE foreign owner: a stopped registration belonging to someone else is not a reason to make a state directory permanently unconfigurable, and a test covers that case too. Relay suite 540 passed, 30 skipped. --- .../src/codex_session_relay/service.py | 23 +++++++++ .../codex-session-relay/tests/test_service.py | 51 +++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 78058cb..2775568 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -477,6 +477,13 @@ def ownership(self, record=None): # the record is foreign on their own, and answering unverifiable would discard a # definite answer in favour of an uncertain one. return FOREIGN, handle, "; ".join(mismatches) + if record.get("bootId") is None: + # A reboot resets both the pid space and the start-tick counter, so without a + # recorded boot the two together prove nothing: an unrelated process can hold the + # old number with a matching tick count and would be classified as ours. + return UNVERIFIABLE, handle, ( + "no boot id is recorded, so a pid from before a reboot cannot be ruled out" + ) if record.get("startTicks") is None or ticks is None: # Nothing proves it foreign, and without a start time a pid is just a number that # anything could be holding now. Unverifiable is the honest answer; stop refuses. @@ -533,6 +540,22 @@ def authority_check(self, *, allow_isolated: bool): return {"ok": True, "reason": None} def enable(self, *, actor: str = "cli") -> dict: + # The same ownership question disable asks, for the same reason: service.json is + # shared by everything pointed at this state directory. An owner who has just + # disabled a service whose supervisor is still exiting must not have that reversed + # by another installation, which would leave it eligible to restart. + owner, handle, detail = self.ownership() + if handle is not None: + handle.close() + if owner in (FOREIGN, UNVERIFIABLE) and self.lock_is_held(): + # Only while something is actually RUNNING here. A stopped registration belonging + # to someone else is not a reason to refuse an owner configuring their own + # installation, and refusing then would make a state directory unusable forever. + return {"ok": False, + "reason": "not_ours" if owner == FOREIGN else "ownership_unverifiable", + "detail": detail, "intent": self.intent.read(), + "note": "intent is shared with the owner of this state directory and was" + " left unchanged"} return {"ok": True, "reason": None, "intent": self.intent.write(enabled=True, actor=actor)} diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index b244736..1e55c2f 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -222,6 +222,57 @@ def test_disable_refuses_a_foreign_service_without_touching_shared_intent(self): self.assertIsNone(child.poll()) self.assertFalse(service.stop_request_path.exists()) + def test_a_record_with_no_boot_id_is_unverifiable(self): + """A reboot resets the pid space AND the start-tick counter together. + + Without a recorded boot, a matching pid with a matching tick count proves nothing: + an unrelated process holding the old number passes both checks and would be signalled. + """ + service = self.service("a") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), bootId=None)) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIn("boot id", refused["detail"]) + self.assertIsNone(child.poll(), "refusing is the point") + + def test_enable_refuses_to_reverse_a_foreign_owners_intent(self): + """service.json is shared, and disable already asks this question. + + An owner who has just disabled a service whose supervisor is still exiting must not + have that reversed by another installation, leaving it eligible to restart. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.intent.write(enabled=False, actor="owner") + service.write_record(dict(service.record(), installationId="someone-else")) + + refused = service.enable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertFalse( + service.intent.read()["enabled"], + "the owner's disable must not be reversed by another installation", + ) + self.assertIsNone(child.poll()) + + def test_enable_still_works_when_nothing_is_running_here(self): + """A stopped registration is not a reason to make a state directory unusable.""" + service = self.service("b") + service.write_record(dict( + service.new_record(pid=os.getpid()), installationId="someone-else", + )) + + enabled = service.enable(actor="owner") + + self.assertTrue(enabled["ok"], enabled) + self.assertTrue(service.intent.read()["enabled"]) + def test_disable_still_works_on_a_service_this_installation_owns(self): service = self.service("b") service.enable(actor="owner") From e3899083edf16a2fb7c23c80d34822bbf8600245 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 17:59:22 +0900 Subject: [PATCH 15/29] Answer the eighth review round on enable, a missing boot and a blocked socket Five findings, three of them consequences of the previous round's enable fix. A supervisor clears its pids before releasing the lock, and ownership answers none once both are absent - which is exactly the interval in which a foreign shutdown could have its owner's disable reversed. enable reads the record's own foreign markers as well as the classification now, so the window the last round left is closed. The boot-id refusal was unconditional, which broke the case it was not aimed at: where no boot id is available at all, every record lacks one - including the one this installation just wrote - so a blanket refusal left a service unstoppable by its own owner. A narrow risk traded for a certain failure. The refusal applies only where the host CAN produce a boot id and the record still has none, which is the pre-reboot record it was written for. A refused enable returned its payload and exited zero, so automation would read a refusal that deliberately changed nothing as a success. It goes through the same refusal path as every other mutating service command. The direct daemon run claimed the scope before the store was ever constructed, so on a fresh state directory it registered with a null store id - the same defect fixed for the managed launch two rounds ago, in the path that was not fixed. A registration naming a store that no longer exists blocked its socket with no way out but finding and deleting a registry file by hand. ScopeRegistry.claim has always anticipated a takeover marker and nothing ever set it; service start --takeover-scope does. It is refused while anything is live on the scope, so it can only ever replace a registration nothing is running behind. Relay suite 550 passed, 30 skipped. --- .../src/codex_session_relay/cli.py | 18 ++++- .../src/codex_session_relay/service.py | 35 +++++--- .../codex-session-relay/tests/test_cli.py | 43 ++++++++++ .../codex-session-relay/tests/test_service.py | 80 +++++++++++++++++++ 4 files changed, 165 insertions(+), 11 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 0a85fb7..6ec500e 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -817,6 +817,10 @@ def _run_bounded(services, service, args, *, require_intent: bool) -> dict: deadline = services.clock.now() + args.deadline if args.deadline else None allow_isolated = getattr(args, "allow_isolated_scope", False) + # Before the claim, for the same reason _supervise does it: the probe that built this + # service answers from a file that may not exist yet, and a scope registration recorded + # with a null store id can later be overwritten by a different store. + service.store_id = services.store.identity adopted = _adopt_supervised(service, args) try: with owned_service( @@ -909,7 +913,10 @@ def cmd_service(services, args) -> dict: if action == "status": return service.status() if action == "enable": - return service.enable(actor=args.actor or "cli") + # Through the same refusal path as every other mutating service command: returning + # the payload directly exits zero, and automation would read a refused enable that + # deliberately changed nothing as a success. + return _refuse_unless_ok(service.enable(actor=args.actor or "cli")) if action == "disable": return _refuse_unless_ok(service.disable(actor=args.actor or "cli")) if action == "stop": @@ -920,7 +927,7 @@ def cmd_service(services, args) -> dict: return _refuse_unless_ok(call( allow_isolated=args.allow_isolated_scope, deadline=args.deadline, segment_seconds=args.segment_seconds, max_segments=args.max_segments, - actor=args.actor or "cli", + actor=args.actor or "cli", takeover=getattr(args, "takeover_scope", False), )) if action == "run": _require_adapter(services) @@ -1292,6 +1299,13 @@ def build_parser() -> argparse.ArgumentParser: hosted.add_argument("--max-segments", type=int) hosted.add_argument("--deadline", type=float) hosted.add_argument("--launch-id") + # For a registration whose store no longer exists - deleted, lost or deliberately + # replaced. Refused while anything is live on the scope, so this can only ever + # replace a registration nothing is running behind. + hosted.add_argument( + "--takeover-scope", action="store_true", + help="replace a stopped registration that names a store this one is not", + ) service.set_defaults(handler=cmd_service) doctor = subparsers.add_parser("doctor") diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 2775568..4caa3a3 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -398,6 +398,8 @@ def __init__(self, selection, *, socket_path=None, scope=None, store_id=None): self.socket_path = socket_path self.store_id = store_id self.launch_id = None + # Set only by an explicit --takeover, and only ever read by ScopeRegistry.claim. + self.takeover = False if scope is None: root, authority = resolve_scope_root() scope = ScopeRegistry(root, authority) @@ -435,6 +437,7 @@ def new_record(self, *, pid, token=None, worker_pid=None) -> dict: "pid": pid, "startTicks": start_ticks(pid), "bootId": boot_id(), "workerPid": worker_pid, "token": token, "launchId": self.launch_id, + "takeover": self.takeover or None, # Written by supervise once on_start has returned. Absent means "not serving yet". "readyAt": None, "storeId": self.store_id, "installationId": self.installation_id, @@ -477,10 +480,13 @@ def ownership(self, record=None): # the record is foreign on their own, and answering unverifiable would discard a # definite answer in favour of an uncertain one. return FOREIGN, handle, "; ".join(mismatches) - if record.get("bootId") is None: - # A reboot resets both the pid space and the start-tick counter, so without a - # recorded boot the two together prove nothing: an unrelated process can hold the - # old number with a matching tick count and would be classified as ours. + if record.get("bootId") is None and boot_id() is not None: + # A reboot resets both the pid space and the start-tick counter, so a record with + # no boot written on a host that HAS one cannot be ruled out as pre-reboot: an + # unrelated process can hold the old number with a matching tick count. + # Conditional on the host, because where no boot id is available at all every + # record lacks one, and refusing them all would leave a service unstoppable by + # its own owner - trading a narrow risk for a certain failure. return UNVERIFIABLE, handle, ( "no boot id is recorded, so a pid from before a reboot cannot be ruled out" ) @@ -544,16 +550,21 @@ def enable(self, *, actor: str = "cli") -> dict: # shared by everything pointed at this state directory. An owner who has just # disabled a service whose supervisor is still exiting must not have that reversed # by another installation, which would leave it eligible to restart. - owner, handle, detail = self.ownership() + record = self.record() + owner, handle, detail = self.ownership(record) if handle is not None: handle.close() - if owner in (FOREIGN, UNVERIFIABLE) and self.lock_is_held(): + # Read from the record as well as from the classification. A supervisor on its way out + # clears its pids before releasing the lock, and ownership answers none once both are + # absent - which is exactly the interval in which a foreign shutdown could be reversed. + markers = self._foreign_markers(record) if record else [] + if self.lock_is_held() and (owner in (FOREIGN, UNVERIFIABLE) or markers): # Only while something is actually RUNNING here. A stopped registration belonging # to someone else is not a reason to refuse an owner configuring their own # installation, and refusing then would make a state directory unusable forever. return {"ok": False, - "reason": "not_ours" if owner == FOREIGN else "ownership_unverifiable", - "detail": detail, "intent": self.intent.read(), + "reason": "ownership_unverifiable" if owner == UNVERIFIABLE else "not_ours", + "detail": detail or "; ".join(markers), "intent": self.intent.read(), "note": "intent is shared with the owner of this state directory and was" " left unchanged"} return {"ok": True, "reason": None, @@ -775,7 +786,7 @@ def restart(self, *, allow_isolated: bool = False, launcher=None, **kw) -> dict: def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cli", max_ticks=None, deadline=None, segment_seconds=None, max_segments=None, - timeout: float = 20.0, poll: float = 0.05) -> dict: + timeout: float = 20.0, poll: float = 0.05, takeover: bool = False) -> dict: """Preconditions here; ownership in the child. The daemon lock and the scope claim are taken by the process that will HOLD them, not @@ -795,6 +806,12 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl live = [conflict for conflict in self.conflicts() if conflict["live"]] if live: return {"ok": False, "reason": "scope_owned_by_other_store", "conflicts": live} + # A registration naming a store that no longer exists - a database deleted, lost or + # deliberately replaced - would otherwise block this socket forever, with recovery + # only through finding and removing a registry file by hand. Taking it over is + # explicit, and only ever from a registration nothing is running behind: the live + # check above has already refused if anything holds the scope. + self.takeover = bool(takeover) launch = uuid.uuid4().hex self.launch_id = launch diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index 40db7e4..241faeb 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -180,6 +180,49 @@ def test_a_staged_claim_is_visible_and_not_deliverable(self): self.assertEqual(self.run_cli("status")["deliveries"], []) +class ServiceExitCodes(CliBase): + """A refusal that exits zero is read by automation as a success.""" + + def test_a_refused_enable_does_not_exit_zero(self): + self.run_cli("service", "enable") + state = os.path.join(self.tmp, "daemon.json") + with open(state, "w", encoding="utf-8") as handle: + json.dump({"pid": os.getpid(), "installationId": "someone-else", + "storeId": "another-store", "bootId": None, + "startTicks": None, "workerPid": None}, handle) + # No lock is held here, so this must still succeed: a stopped foreign registration + # is not a reason to make a state directory unconfigurable. + self.assertTrue(self.run_cli("service", "enable")["ok"]) + + def test_a_refused_enable_is_a_refusal_the_shell_can_see(self): + """Returning the payload directly exits zero, and automation reads that as done.""" + from unittest import mock + + from codex_session_relay import cli + + class Refusing: + def enable(self, *, actor): + return {"ok": False, "reason": "not_ours", "intent": {"enabled": False}} + + args = argparse.Namespace(service_command="enable", actor=None) + with mock.patch.object(cli, "_service_for", lambda _services: Refusing()): + with self.assertRaises(cli.PayloadExit) as caught: + cli.cmd_service(object(), args) + self.assertEqual(caught.exception.code, cli.EXIT_REFUSED) + self.assertEqual(caught.exception.payload["reason"], "not_ours") + + def test_a_refused_disable_does_not_exit_zero(self): + self.run_cli("service", "enable") + state = os.path.join(self.tmp, "daemon.json") + with open(state, "w", encoding="utf-8") as handle: + json.dump({"pid": os.getpid(), "installationId": "someone-else", + "storeId": "another-store", "bootId": "irrelevant", + "startTicks": 1, "workerPid": None}, handle) + refused = self.run_cli("service", "disable", expect=2) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "not_ours") + + class SettingsCommands(CliBase): """The registration interface JUN-92 populates from Run's creation result.""" diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 1e55c2f..cb84b20 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -261,6 +261,86 @@ def test_enable_refuses_to_reverse_a_foreign_owners_intent(self): ) self.assertIsNone(child.poll()) + def test_enable_is_refused_while_a_foreign_supervisor_is_shutting_down(self): + """A supervisor clears its pids BEFORE releasing the lock. + + ownership answers none once both are absent, which is exactly the interval in which + a foreign shutdown could have its owner's disable reversed. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.intent.write(enabled=False, actor="owner") + service.write_record(dict( + service.record(), pid=None, workerPid=None, installationId="someone-else", + )) + self.assertTrue(service.lock_is_held(), "the fixture needs the lock still held") + + refused = service.enable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertFalse(service.intent.read()["enabled"]) + self.assertIsNone(child.poll()) + + def test_a_host_with_no_boot_id_can_still_stop_its_own_service(self): + """Refusing every record without one traded a narrow risk for a certain failure. + + Where no boot id is available at all, every record lacks one - including the record + this installation just wrote - so a blanket refusal makes the service unstoppable by + its own owner. + """ + service = self.service("a") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), bootId=None)) + + with mock.patch.object(service_module, "boot_id", lambda: None): + stopped = service.stop() + + self.assertTrue(stopped["ok"], stopped) + child.wait(timeout=10) + + def test_a_stopped_registration_can_be_taken_over_deliberately(self): + """Otherwise a replaced database blocks its socket forever. + + Recovery was finding and deleting an internal registry file by hand, which is not an + operation anyone should have to discover. + """ + first = self.service("a") + child, _pid = self.holder(first) + child.terminate() + child.wait(timeout=10) + second = self.service("b") + self.assertNotEqual(second.store_id, first.store_id) + + refused = second.scope.claim(second.socket_path, second.new_record(pid=os.getpid())) + self.assertEqual(refused["reason"], "scope_registered_to_other_store") + second.scope.release(second.socket_path) + + second.takeover = True + taken = second.scope.claim(second.socket_path, second.new_record(pid=os.getpid())) + + self.assertTrue(taken["ok"], taken) + self.assertEqual(second.scope.read(second.socket_path)["storeId"], second.store_id) + second.scope.release(second.socket_path) + + def test_a_takeover_cannot_displace_a_live_owner(self): + """It only ever replaces a registration nothing is running behind.""" + first = self.service("a") + self.holder(first) + second = self.service("b") + second.intent.write(enabled=True, actor="test") + + refused = second.start( + allow_isolated=True, launcher=self._never_launch_here, takeover=True, + ) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "scope_owned_by_other_store") + + def _never_launch_here(self, *_a, **_k): # pragma: no cover - reaching it is the failure + self.fail("a takeover must not launch past a live owner") + def test_enable_still_works_when_nothing_is_running_here(self): """A stopped registration is not a reason to make a state directory unusable.""" service = self.service("b") From f155c88ebe56f4346c9fba809b57ee2136fa4804 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 18:16:31 +0900 Subject: [PATCH 16/29] Answer the ninth review round on an upgrade that would have lost a store Five findings, four of them defects the last two rounds introduced, and one that would have been the worst outcome in this PR. Canonicalising the socket hash moved the default state directory. I flagged that myself as an install-time consideration and left it; it is worse than that. A relative or symlinked socket that had been running would open a fresh empty database on upgrade while its assignments, generations and pending deliveries sat in the old directory, invisible. resolve_state_dir now keeps the directory a store is already in: the canonical name is used for anything new, and the old one wins only when it actually holds a store and the new one does not. The orphan-worker branch of ownership skipped the boot-id question the live-supervisor path asks. _stop_worker checks start ticks alone, and a reboot resets those with the pid space, so an unrelated process holding the old worker number would have been signalled. disable never got the record-level marker check enable was given last round, so the cleanup window - pids cleared, lock still held - was closed on one command and open on the other. Closed on both. supervise marked a worker outstanding only AFTER the bookkeeping call that can fail. A _note raising after a successful spawn left a running worker the cleanup then treated as never started, clearing the identity a stop needs while that worker still held the inherited locks. Marked before. --takeover-scope was set on the parent object and dropped at the process boundary. The supervisor is the process that claims the scope, so the flag did nothing at all: the feature the previous round added did not work. It is forwarded on the launch argv and honoured in the supervisor. Relay suite 555 passed, 30 skipped. --- .../src/codex_session_relay/cli.py | 3 + .../src/codex_session_relay/service.py | 31 +++++-- .../src/codex_session_relay/store.py | 24 +++++- .../codex-session-relay/tests/test_service.py | 80 +++++++++++++++++++ .../codex-session-relay/tests/test_store.py | 39 +++++++++ 5 files changed, 171 insertions(+), 6 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 6ec500e..a71b89b 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -940,6 +940,9 @@ def _supervise(services, service, args) -> dict: from .service import ServiceRefused service.launch_id = getattr(args, "launch_id", None) + # This process is the one that claims the scope, so the flag has to be honoured here and + # not only in the parent that decided to pass it. + service.takeover = getattr(args, "takeover_scope", False) # The probe that built this service answers from a file that may not exist yet, so on a # fresh state directory it reports no store id at all. Opening the store HERE is not the # side effect doctor and status refuse: a supervisor is about to use it either way. It diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 4caa3a3..2f5d4d0 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -459,6 +459,14 @@ def ownership(self, record=None): foreign = self._foreign_markers(record) if foreign: return FOREIGN, None, "; ".join(foreign) + if record.get("bootId") is None and boot_id() is not None: + # The same rule the live-supervisor path gets. _stop_worker checks only + # start ticks, and a reboot resets those along with the pid space, so an + # unrelated process holding the old worker number would be signalled. + return UNVERIFIABLE, None, ( + "no boot id is recorded, so this worker pid cannot be" + " distinguished from one reused after a reboot" + ) return NONE, None, "no daemon record" # Read from the record itself, so they survive the supervisor's death. A foreign # installation whose supervisor is gone can still have a live worker recorded, and @@ -576,13 +584,17 @@ def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: # boundary: writing enabled=false and only then discovering the supervisor is not # ours refused the stop while still shutting that supervisor down at its next # boundary. A refusal that still has an effect is not a refusal. - owner, handle, detail = self.ownership() + record = self.record() + owner, handle, detail = self.ownership(record) if handle is not None: handle.close() - if owner in (FOREIGN, UNVERIFIABLE): + # The same cleanup window enable guards: a supervisor clears its pids before it + # releases the lock, and ownership answers none once both are absent. + markers = self._foreign_markers(record) if record else [] + if owner in (FOREIGN, UNVERIFIABLE) or (markers and self.lock_is_held()): return {"ok": False, - "reason": "not_ours" if owner == FOREIGN else "ownership_unverifiable", - "detail": detail, "intent": self.intent.read(), + "reason": "ownership_unverifiable" if owner == UNVERIFIABLE else "not_ours", + "detail": detail or "; ".join(markers), "intent": self.intent.read(), "stop": {"ok": False, "reason": "refused", "supervisor": "untouched", "worker": "untouched"}, "note": "intent is shared with the owner of this state directory and was" @@ -897,6 +909,11 @@ def default_launcher(self, service, *, allow_isolated, max_ticks=None, deadline= environment["CODEX_SESSION_RELAY_STATE"] = str(self.selection.path) if self.launch_id: argv += ["--launch-id", self.launch_id] + if self.takeover: + # The supervisor is the process that CLAIMS the scope, so the flag has to reach + # it. Set only on this object, it was dropped at the process boundary and the + # takeover silently did nothing. + argv.append("--takeover-scope") self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: return subprocess.Popen( @@ -1022,8 +1039,12 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= ), allow_isolated=allow_isolated, ) - self._note(workerPid=child.pid, workerStartTicks=start_ticks(child.pid)) + # Marked outstanding BEFORE the bookkeeping that can fail. A _note that + # raises after a successful spawn left a running worker the cleanup then + # treated as never started, clearing the identity a stop needs while the + # worker still holds the inherited locks. outstanding = child + self._note(workerPid=child.pid, workerStartTicks=start_ticks(child.pid)) code = child.wait() outstanding = None self._note(workerPid=None, workerStartTicks=None, lastExit=code, diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index f6f3fb6..021cfb9 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -450,6 +450,13 @@ def socket_scope(socket_path) -> str: ).hexdigest()[:16] +def legacy_socket_scope(socket_path) -> str: + """What socket_scope produced before it canonicalised, for finding an existing store.""" + if not socket_path: + return "default" + return hashlib.sha256(str(Path(socket_path).expanduser()).encode()).hexdigest()[:16] + + @dataclass(frozen=True) class StateSelection: """Which rule chose the state directory, and the exact value that won. @@ -500,7 +507,22 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: base, source = Path.home() / ".local" / "state", "home" detail = f"default under {Path.home() / '.local' / 'state'}" scope = socket_scope(socket_path) - return StateSelection((base / "codex-session-relay" / scope).absolute(), source, detail, scope) + chosen = (base / "codex-session-relay" / scope).absolute() + # An existing store keeps its directory. Canonicalising the socket changed this hash, so + # a relative or symlinked socket that had been running would otherwise point at a fresh + # empty database while its assignments, generations and pending deliveries sat in the + # old one, invisible. The new name is used for anything new; the old one wins only when + # it actually holds a store and the new one does not. + legacy = legacy_socket_scope(socket_path) + if legacy != scope and not chosen.exists(): + previous = (base / "codex-session-relay" / legacy).absolute() + if (previous / "relay.sqlite3").exists(): + return StateSelection( + previous, source, + f"{detail}; kept the directory this socket was already using", + legacy, + ) + return StateSelection(chosen, source, detail, scope) def state_dir(socket_path: str | None = None) -> Path: diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index cb84b20..245f0ed 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -74,6 +74,10 @@ """ +class _Captured(Exception): + """Stops a start before it launches anything.""" + + class ServiceTestCase(unittest.TestCase): def setUp(self): self.tmp = tempfile.mkdtemp(prefix="relay-service-") @@ -300,6 +304,82 @@ def test_a_host_with_no_boot_id_can_still_stop_its_own_service(self): self.assertTrue(stopped["ok"], stopped) child.wait(timeout=10) + def test_an_orphan_worker_with_no_boot_id_is_unverifiable_too(self): + """_stop_worker checks start ticks only, and a reboot resets those with the pids. + + The live-supervisor path already refused this; the orphan branch did not, so an + unrelated process holding the old worker number would have been signalled. + """ + service = self.service("a") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, bootId=None, workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIsNone(child.poll(), "an unidentifiable orphan must not be signalled") + + def test_disable_is_refused_while_a_foreign_supervisor_is_shutting_down(self): + """The same cleanup window enable guards, in the command that writes the same file.""" + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, workerPid=None, installationId="someone-else", + )) + self.assertTrue(service.lock_is_held(), "the fixture needs the lock still held") + + refused = service.disable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertTrue( + service.intent.read()["enabled"], + "a foreign owner's service must not be left disabled by a refused command", + ) + self.assertIsNone(child.poll()) + + def test_the_takeover_flag_reaches_the_process_that_claims_the_scope(self): + """It was set on the parent object and dropped at the process boundary. + + The supervisor is the process that claims the scope, so a flag the parent keeps to + itself leaves the takeover silently doing nothing. + """ + service = self.service("a") + service.enable(actor="test") + + self.assertNotIn("--takeover-scope", self._launch_argv(service)) + + service.takeover = True + + self.assertIn("--takeover-scope", self._launch_argv(service)) + + def _launch_argv(self, service): + """The argv default_launcher would build, without starting anything.""" + import subprocess + from unittest import mock + + captured = {} + + class Fake: + pid = os.getpid() + returncode = None + + def poll(self): + return None + + def popen(argv, **_kwargs): + captured["argv"] = argv + return Fake() + + with mock.patch.object(subprocess, "Popen", popen): + service.default_launcher(service, allow_isolated=True) + return captured["argv"] + def test_a_stopped_registration_can_be_taken_over_deliberately(self): """Otherwise a replaced database blocks its socket forever. diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index c06e0b1..12555b5 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -162,6 +162,45 @@ def test_a_relative_flag_resolves_to_the_same_store_as_its_absolute_form(self): self.assertEqual(relative["store"]["storeId"], absolute["store"]["storeId"]) self.assertEqual(relative["store"]["inode"], absolute["store"]["inode"]) + def test_a_store_already_in_use_keeps_its_directory_after_the_hash_changed(self): + """Canonicalising the socket moved this hash, which is an upgrade hazard. + + A relative or symlinked socket that had been running would otherwise open a fresh + empty database while its assignments, generations and pending deliveries sat in the + old directory, invisible. + """ + from codex_session_relay.store import legacy_socket_scope + + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = os.path.join("run", "app-server.sock") + os.makedirs(os.path.join(self.tmp, "run"), exist_ok=True) + open(os.path.join(self.tmp, "run", "app-server.sock"), "w").close() + legacy = os.path.join( + self.home, ".local", "state", "codex-session-relay", + legacy_socket_scope(relative), + ) + os.makedirs(legacy) + Store(Path(legacy) / "relay.sqlite3").close() + + chosen = resolve_state_dir(None, relative) + + self.assertEqual(str(chosen.path), legacy, + "the store already in use keeps its directory") + self.assertIn("already using", chosen.detail) + finally: + os.chdir(here) + + def test_a_fresh_socket_uses_the_canonical_directory(self): + """The fallback is for an existing store only; nothing new lands in the old name.""" + from codex_session_relay.store import socket_scope + + chosen = resolve_state_dir(None, "/run/brand-new.sock") + + self.assertEqual(chosen.socket_scope, socket_scope("/run/brand-new.sock")) + self.assertTrue(str(chosen.path).endswith(socket_scope("/run/brand-new.sock"))) + def test_two_spellings_of_one_socket_choose_the_same_default_store(self): """The scope registry canonicalises the socket; this hash used to take it verbatim. From c138b80242404e1dd2415df8a126010b85343a5f Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Wed, 16 Sep 2026 18:34:55 +0900 Subject: [PATCH 17/29] Answer the tenth review round on a directory that is not a store The legacy-store fallback tested whether the canonical DIRECTORY exists, which any command that writes beside the database creates - a stop request is enough. One such command would then hide a legacy store holding real assignments behind an empty folder, which is the data-loss path the fallback was added to close. It tests for the canonical database now. stop wrote its request when the daemon lock was held by a process that had not yet published its identity - a supervisor between taking the lock and writing its record. That is not harmless: a supervisor clears pending requests only BEFORE taking the lock, so it consumes a request written after that and exits. The caller was told not_running and stopped the service anyway. Refused as unverifiable, with no request written. A record naming a WORKER is the other kind of none - our own supervisor gone with its orphan holding the inherited lock - and that still reaches the orphan, which a test covers. Relay suite 555 passed, 30 skipped. --- .../src/codex_session_relay/service.py | 12 ++++++++++ .../src/codex_session_relay/store.py | 6 ++++- .../codex-session-relay/tests/test_service.py | 22 +++++++++++++++++++ .../codex-session-relay/tests/test_store.py | 15 ++++++++++++- 4 files changed, 53 insertions(+), 2 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 2f5d4d0..ab7fe3d 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -701,6 +701,18 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # process gets killed after its number is reused. return {"ok": False, "reason": "ownership_unverifiable", "detail": detail, "supervisor": "untouched", "worker": "untouched"} + if (owner == NONE and self.lock_is_held() + and not (record or {}).get("workerPid")): + # Something holds the lock and has not published who it is: a supervisor + # between taking the lock and writing its record. Writing a stop request here + # is not harmless - it clears requests only BEFORE taking the lock, so it + # consumes this one and exits. A refusal that still stops the service. + # A record that names a WORKER is the other none: our own supervisor gone + # with its orphan holding the inherited lock, which is what stop must reach. + return {"ok": False, "reason": "ownership_unverifiable", + "detail": "the daemon lock is held by a process that has not yet" + " recorded its identity", + "supervisor": "untouched", "worker": "untouched"} # Only now. Writing the stop request before validating ownership left a refused # stop able to halt another installation's supervisor at its next boundary, which # is a refusal that still had an effect. diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 021cfb9..3e42187 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -514,7 +514,11 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: # old one, invisible. The new name is used for anything new; the old one wins only when # it actually holds a store and the new one does not. legacy = legacy_socket_scope(socket_path) - if legacy != scope and not chosen.exists(): + # Whether the canonical DATABASE exists, not whether its directory does. A directory is + # created by any command that writes beside the store - a stop request is enough - and + # testing for the directory let one such command hide a legacy store holding real + # assignments behind an empty folder. + if legacy != scope and not (chosen / "relay.sqlite3").exists(): previous = (base / "codex-session-relay" / legacy).absolute() if (previous / "relay.sqlite3").exists(): return StateSelection( diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 245f0ed..e21366c 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -453,6 +453,28 @@ def test_stop_refuses_a_recycled_pid(self): self.assertIn("reused", refused["detail"]) self.assertIsNone(child.poll()) + def test_stop_writes_no_request_for_a_lock_held_by_an_unidentified_process(self): + """A supervisor between taking the lock and writing its record has no identity yet. + + The stop request is not harmless there: a supervisor clears pending requests only + BEFORE taking the lock, so a request written after that is consumed by the very + service the caller was told it had not stopped. + """ + service = self.service("a") + child, _pid = self.holder(service) + service.record_path.unlink() + self.assertTrue(service.lock_is_held()) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertFalse( + service.stop_request_path.exists(), + "a refused stop must not leave a request the service will act on", + ) + self.assertIsNone(child.poll()) + def test_stop_refuses_when_the_process_cannot_be_aimed_at(self): service = self.service("a") child, pid = self.holder(service) diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 12555b5..255a59b 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -169,7 +169,7 @@ def test_a_store_already_in_use_keeps_its_directory_after_the_hash_changed(self) empty database while its assignments, generations and pending deliveries sat in the old directory, invisible. """ - from codex_session_relay.store import legacy_socket_scope + from codex_session_relay.store import legacy_socket_scope, socket_scope here = os.getcwd() os.chdir(self.tmp) @@ -189,6 +189,19 @@ def test_a_store_already_in_use_keeps_its_directory_after_the_hash_changed(self) self.assertEqual(str(chosen.path), legacy, "the store already in use keeps its directory") self.assertIn("already using", chosen.detail) + + # An empty canonical DIRECTORY is not a canonical store. Any command that writes + # beside the database creates one - a stop request is enough - and testing for + # the directory let that hide the legacy store behind an empty folder. + os.makedirs( + os.path.join(self.home, ".local", "state", "codex-session-relay", + socket_scope(relative)), + exist_ok=True, + ) + self.assertEqual( + str(resolve_state_dir(None, relative).path), legacy, + "an empty canonical directory must not hide a real store", + ) finally: os.chdir(here) From a3f3dc0f2de799a22f8c94ca7719ee97102eb89f Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 01:42:36 +0900 Subject: [PATCH 18/29] Close three ownership races the lock could already have settled All three are the same shape: a state was PROBED, and then acted on as if the probe were still true. Between the two, a supervisor can start. stop() asked lock_is_held(), which takes the lock and gives it straight back, and then wrote a stop request. supervise() clears pending requests BEFORE it acquires the lock, so a supervisor that acquires in that gap consumes the request and exits - while the stop that left it reported not_running. The decision is now taken while HOLDING the lock: holding it is the proof that nothing is running and nothing can start, and it returns not_running with nothing left behind. Failing to take it is itself the answer that someone is there, so the record is re-read and re-classified once. A record naming a worker never enters this path; that orphan is what stop is for. disable() checked ownership and then wrote enabled=false. A foreign supervisor starting in between reads enabled=true, takes the lock, and then sees the intent we changed afterwards and exits at its next boundary - a refused disable that stopped it anyway. The classification and the write now happen under the same lock. stop() is still called after releasing it, because stop takes the lock for its own decision and would misread one we were holding. Both use one new helper, daemon_lock_if_free, which keeps the lock for the duration of a decision rather than describing a moment that has already passed. The store selection had the same gap in a different currency. The legacy-directory fallback compares the hash of the CURRENT spelling with its canonical hash, so it only helps when this invocation uses the old spelling. A first post-upgrade command that happens to use the absolute path has legacy == scope, the comparison is a no-op, a fresh canonical database is created - and afterwards even the old relative spelling finds the new one, because it now exists. The pre-upgrade assignments are hidden for good. A hash cannot be inverted, so arithmetic cannot recover the old spelling. Stores now record which socket they serve, and selection asks the stores themselves before creating anything. That scan runs only when no canonical database exists yet, which is the one moment it matters. What this does NOT fix is stated rather than papered over: a store created before provenance existed says nothing about its socket and cannot be matched to one by anything but its directory hash. Both stores on this host are exactly that case - schema_meta holds only version, not even store_id. Adopting an unlabelled store on a guess is how the wrong store gets served, so doctor reports them under siblingStores instead and an operator decides with --state. Relay suite 645 passed, 30 skipped. validate.py, contracts.py, check_operations_contract.py, the scripts/ci/tests unittest suite, secrets.sh and git diff --check all exit 0. --- .../src/codex_session_relay/cli.py | 25 +++- .../src/codex_session_relay/service.py | 136 ++++++++++++++---- .../src/codex_session_relay/store.py | 98 ++++++++++++- .../codex-session-relay/tests/test_service.py | 99 +++++++++++++ .../codex-session-relay/tests/test_store.py | 55 +++++++ 5 files changed, 379 insertions(+), 34 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 9bcf1cd..02058b1 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -92,7 +92,9 @@ def state_directory(self): @property def store(self): if self._store is None: - self._store = Store(self.selection.db_path) + # The socket travels with the store so a later invocation can find it by socket + # rather than by the hash of the spelling that happened to create it. + self._store = Store(self.selection.db_path, socket_path=self.socket_path) return self._store @property @@ -766,6 +768,26 @@ def _contents(services, report) -> dict: "openAttempts": open_attempts, "detail": None} +def _sibling_stores(services) -> dict: + """Other stores beside this one that never recorded which socket they serve. + + A store is matched to a socket by provenance it records for itself. One created before + that existed can only be matched by its directory hash, so if this command is about to + create a fresh canonical database next to such a store, it may be hiding real data. That + is reported rather than resolved: adopting on a guess is how the wrong store gets served. + """ + from .store import stores_without_provenance + + root = services.selection.path.parent + if services.selection.source in ("flag", "env"): + # An explicit directory was chosen by a caller who already decided which participants + # share it, so its neighbours are not candidates for anything. + return {"checked": False, "reason": "the state directory was chosen explicitly", + "withoutProvenance": []} + without = stores_without_provenance(root, skip=services.selection.path.name) + return {"checked": True, "reason": None, "withoutProvenance": without} + + def cmd_doctor(services, args) -> dict: """What THIS process can actually do here, measured rather than assumed. @@ -783,6 +805,7 @@ def cmd_doctor(services, args) -> dict: report["ledger"] = _ledger_location(services) report["actorReachability"] = _reachability(services, report) report["contents"] = _contents(services, report) + report["siblingStores"] = _sibling_stores(services) nonce = nonce_lookup(services.selection, args.expect_nonce) if args.expect_nonce else None report["nonce"] = nonce diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index ab7fe3d..2fa0686 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -540,6 +540,35 @@ def lock_is_held(self) -> bool: finally: handle.close() + @contextmanager + def daemon_lock_if_free(self): + """Take the daemon lock and KEEP it for the duration of a decision. + + lock_is_held probes and releases, which is enough to describe a state but not to act + on one: between the probe and whatever the caller does next, a supervisor can acquire + the lock. Holding it across the decision closes that gap - a supervisor cannot start + while we hold it - and failing to take it is itself the answer that someone is there. + + Opened with 'a+' so the probe is atomic even on a state directory that has never run + a daemon: a supervisor starting there has to create and lock this same file. + """ + path = self.selection.path / DAEMON_LOCK + handle = None + try: + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(path, "a+") + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + if handle is not None: + handle.close() + handle = None + try: + yield handle + finally: + if handle is not None: + fcntl.flock(handle, fcntl.LOCK_UN) + handle.close() + # -------------------------------------------------------------- operations def authority_check(self, *, allow_isolated: bool): @@ -584,22 +613,40 @@ def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: # boundary: writing enabled=false and only then discovering the supervisor is not # ours refused the stop while still shutting that supervisor down at its next # boundary. A refusal that still has an effect is not a refusal. - record = self.record() - owner, handle, detail = self.ownership(record) - if handle is not None: - handle.close() - # The same cleanup window enable guards: a supervisor clears its pids before it - # releases the lock, and ownership answers none once both are absent. - markers = self._foreign_markers(record) if record else [] - if owner in (FOREIGN, UNVERIFIABLE) or (markers and self.lock_is_held()): - return {"ok": False, - "reason": "ownership_unverifiable" if owner == UNVERIFIABLE else "not_ours", - "detail": detail or "; ".join(markers), "intent": self.intent.read(), - "stop": {"ok": False, "reason": "refused", "supervisor": "untouched", - "worker": "untouched"}, - "note": "intent is shared with the owner of this state directory and was" - " left unchanged"} - written = self.intent.write(enabled=False, actor=actor) + # + # Decided and written UNDER the daemon lock. Checking ownership and then writing are + # two operations, and a foreign supervisor can start between them: it reads + # enabled=true, takes the lock, and then sees the intent we changed afterwards and + # exits at its next boundary. Holding the lock means no supervisor can start while we + # decide, and failing to take it means one is already there to be classified. + refusal = None + with self.daemon_lock_if_free() as held: + if held is None: + # Someone holds it. Re-read: a supervisor that started between our read and + # now has published its record, or is about to. + record = self.record() + owner, handle, detail = self.ownership(record) + if handle is not None: + handle.close() + markers = self._foreign_markers(record) if record else [] + if owner in (FOREIGN, UNVERIFIABLE) or markers: + refusal = { + "ok": False, + "reason": ("ownership_unverifiable" if owner == UNVERIFIABLE + else "not_ours"), + "detail": detail or "; ".join(markers), + "intent": self.intent.read(), + "stop": {"ok": False, "reason": "refused", + "supervisor": "untouched", "worker": "untouched"}, + "note": "intent is shared with the owner of this state directory and" + " was left unchanged", + } + if refusal is None: + written = self.intent.write(enabled=False, actor=actor) + if refusal is not None: + return refusal + # Outside the lock deliberately: stop() takes it for its own decision, and holding it + # here would make stop misread a free lock as ours. stopped = self.stop(actor=actor, timeout=timeout) # A stop that had nothing to stop is not a failure; the intent is what disable owns. failed = not stopped["ok"] and stopped["reason"] != "not_running" @@ -692,6 +739,16 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) """Signal only a process this installation owns, through a handle to that process.""" record = self.record() owner, handle, detail = self.ownership(record) + if owner == NONE and not (record or {}).get("workerPid"): + # Nothing verifiable is running. Deciding that and writing a stop request are two + # operations, and supervise() clears pending requests BEFORE it acquires the lock, + # so a supervisor that starts between them consumes the request we leave and exits + # - while this call reports not_running. Settle it while holding the lock instead. + settled, record, owner, handle, detail = self._settle_absent_owner(record, detail) + if settled is not None: + if handle is not None: + handle.close() + return settled try: if owner == FOREIGN: return {"ok": False, "reason": "not_ours", "detail": detail, @@ -701,18 +758,6 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # process gets killed after its number is reused. return {"ok": False, "reason": "ownership_unverifiable", "detail": detail, "supervisor": "untouched", "worker": "untouched"} - if (owner == NONE and self.lock_is_held() - and not (record or {}).get("workerPid")): - # Something holds the lock and has not published who it is: a supervisor - # between taking the lock and writing its record. Writing a stop request here - # is not harmless - it clears requests only BEFORE taking the lock, so it - # consumes this one and exits. A refusal that still stops the service. - # A record that names a WORKER is the other none: our own supervisor gone - # with its orphan holding the inherited lock, which is what stop must reach. - return {"ok": False, "reason": "ownership_unverifiable", - "detail": "the daemon lock is held by a process that has not yet" - " recorded its identity", - "supervisor": "untouched", "worker": "untouched"} # Only now. Writing the stop request before validating ownership left a refused # stop able to halt another installation's supervisor at its next boundary, which # is a refusal that still had an effect. @@ -751,6 +796,41 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) return {"ok": ok, "reason": None if ok else "did_not_exit", "detail": None, "supervisor": outcome, "worker": worker} + def _settle_absent_owner(self, record, detail): + """Decide 'nothing is running' while HOLDING the lock that would prove otherwise. + + Returns (response_or_None, record, owner, handle, detail). A response ends the stop; + None means the caller should continue with the re-read record and classification. + + A record naming a WORKER never reaches here: that none is our own supervisor gone + with its orphan holding the inherited lock, and reaching that orphan is what stop is + for. + """ + with self.daemon_lock_if_free() as held: + if held is not None: + # We hold it, so no supervisor is running and none can start while we decide. + # Returning without a request is the whole point: a request left behind here + # would be consumed by the next supervisor to start. + return ( + {"ok": False, "reason": "not_running", "detail": detail, + "supervisor": "gone", "worker": "gone"}, + record, NONE, None, detail, + ) + # Not free: something took it. Re-read - it may have published its identity by now. + again = self.record() + owner, handle, fresh = self.ownership(again) + if owner == NONE and not (again or {}).get("workerPid"): + if handle is not None: + handle.close() + return ( + {"ok": False, "reason": "ownership_unverifiable", + "detail": "the daemon lock is held by a process that has not yet" + " recorded its identity", + "supervisor": "untouched", "worker": "untouched"}, + again, owner, None, fresh, + ) + return (None, again, owner, handle, fresh) + def _terminate(self, handle, *, timeout, grace) -> str: if not handle.send(signal.SIGTERM): return "signal_refused" diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 85d7d70..66b93d5 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -494,9 +494,12 @@ def socket_scope(socket_path) -> str: # gave a relative path and a symlink alias for one socket two different default state # directories, so the second invocation opened a different store and was then refused # by the scope registry as a foreign owner instead of joining the service already there. - return hashlib.sha256( - str(Path(socket_path).expanduser().absolute().resolve()).encode() - ).hexdigest()[:16] + return hashlib.sha256(canonical_socket(socket_path).encode()).hexdigest()[:16] + + +def canonical_socket(socket_path) -> str: + """One spelling for one socket, the same way ScopeRegistry.key resolves it.""" + return str(Path(socket_path).expanduser().absolute().resolve()) def legacy_socket_scope(socket_path) -> str: @@ -557,6 +560,8 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: detail = f"default under {Path.home() / '.local' / 'state'}" scope = socket_scope(socket_path) chosen = (base / "codex-session-relay" / scope).absolute() + if (chosen / "relay.sqlite3").exists(): + return StateSelection(chosen, source, detail, scope) # An existing store keeps its directory. Canonicalising the socket changed this hash, so # a relative or symlinked socket that had been running would otherwise point at a fresh # empty database while its assignments, generations and pending deliveries sat in the @@ -567,7 +572,7 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: # created by any command that writes beside the store - a stop request is enough - and # testing for the directory let one such command hide a legacy store holding real # assignments behind an empty folder. - if legacy != scope and not (chosen / "relay.sqlite3").exists(): + if legacy != scope: previous = (base / "codex-session-relay" / legacy).absolute() if (previous / "relay.sqlite3").exists(): return StateSelection( @@ -575,16 +580,91 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: f"{detail}; kept the directory this socket was already using", legacy, ) + # The legacy hash only helps when THIS invocation used the old spelling. A first + # post-upgrade command that happens to use the absolute path has legacy == scope, so the + # comparison above never looks at the store the relative spelling created - and creating + # a canonical database here would hide it for good, because afterwards even the old + # spelling finds the new one. So before creating anything, ask the stores themselves. + # Only reached when no canonical database exists yet, which is the one moment it matters. + adopted = discover_store_for_socket(base / "codex-session-relay", socket_path, skip=scope) + if adopted is not None: + return StateSelection( + adopted, source, + f"{detail}; adopted the store already recorded for this socket", + adopted.name, + ) return StateSelection(chosen, source, detail, scope) +def store_socket(db_path) -> str | None: + """The canonical socket a store recorded for itself, or None if it never recorded one.""" + try: + connection = sqlite3.connect(f"{Path(db_path).as_uri()}?mode=ro", uri=True, timeout=5) + except (OSError, sqlite3.Error, ValueError): + return None + try: + row = connection.execute( + "SELECT value FROM schema_meta WHERE key = 'socket_path'" + ).fetchone() + except sqlite3.Error: + return None + finally: + connection.close() + return row[0] if row else None + + +def discover_store_for_socket(root, socket_path, *, skip=None): + """The directory holding the store this socket already has, under any spelling. + + Provenance rather than arithmetic: a hash cannot be inverted, so a store created under a + spelling we cannot guess is only findable if it says which socket it belongs to. Stores + record that from now on; one created before it did says nothing and is reported by doctor + instead of being adopted on a guess. + """ + if not socket_path: + return None + try: + candidates = sorted(p for p in Path(root).iterdir() if p.is_dir()) + except OSError: + return None + wanted = canonical_socket(socket_path) + for directory in candidates: + if skip is not None and directory.name == skip: + continue + database = directory / "relay.sqlite3" + if not database.exists(): + continue + if store_socket(database) == wanted: + return directory + return None + + +def stores_without_provenance(root, *, skip=None) -> list: + """Store directories that never recorded which socket they serve. + + They cannot be matched to a socket by anything but their directory hash, so a command + that creates a fresh canonical database beside one of them may be hiding real data. + Reported rather than adopted: adopting on a guess is how the wrong store gets served. + """ + try: + candidates = sorted(p for p in Path(root).iterdir() if p.is_dir()) + except OSError: + return [] + found = [] + for directory in candidates: + if skip is not None and directory.name == skip: + continue + database = directory / "relay.sqlite3" + if database.exists() and store_socket(database) is None: + found.append(str(directory)) + return found def state_dir(socket_path: str | None = None) -> Path: """The directory the environment alone would choose. Kept for callers that have no flag.""" return resolve_state_dir(None, socket_path).path class Store: - def __init__(self, path): + def __init__(self, path, socket_path=None): path = Path(path) path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) descriptor = os.open(path, os.O_CREAT | os.O_RDWR, 0o600) @@ -608,6 +688,14 @@ def __init__(self, path): self.db.execute( "INSERT OR IGNORE INTO schema_meta VALUES ('store_created_at', ?)", (_now_iso(),) ) + if socket_path: + # Provenance, so this store is findable by the socket it serves rather than only + # by the hash of whichever spelling created it. INSERT OR IGNORE: the first + # recording wins, so re-opening through a different spelling never rewrites it. + self.db.execute( + "INSERT OR IGNORE INTO schema_meta VALUES ('socket_path', ?)", + (canonical_socket(socket_path),), + ) # Tests set this to prove a transition rolls back; nothing in production assigns it. self.fault_hook = None diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index e21366c..3124aa1 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -4,6 +4,7 @@ flock; only a second process trying to start proves that the first one is actually excluded. """ +import contextlib import json import os import shutil @@ -421,6 +422,57 @@ def test_a_takeover_cannot_displace_a_live_owner(self): def _never_launch_here(self, *_a, **_k): # pragma: no cover - reaching it is the failure self.fail("a takeover must not launch past a live owner") + def test_disable_decides_and_writes_under_the_lock(self): + """A foreign supervisor starting between the check and the write read enabled=true. + + It took the lock, then saw the intent we changed afterwards and exited at its next + boundary - a refused disable that still stopped it. The classification and the write + happen under the lock now, so no supervisor can start between them. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), installationId="someone-else")) + + refused = service.disable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertTrue( + service.intent.read()["enabled"], + "a refused disable must not change the intent a foreign supervisor reads", + ) + self.assertIsNone(child.poll()) + + def test_disable_takes_the_lock_before_touching_shared_intent(self): + """The ordering itself, so a later edit cannot quietly put the write back first.""" + service = self.service("b") + service.enable(actor="owner") + order = [] + original_lock = service.daemon_lock_if_free + original_write = service.intent.write + + @contextlib.contextmanager + def watched_lock(): + order.append("lock") + with original_lock() as held: + yield held + + def watched_write(**kwargs): + order.append("write") + return original_write(**kwargs) + + service.daemon_lock_if_free = watched_lock + service.intent.write = watched_write + try: + service.disable(actor="owner") + finally: + service.daemon_lock_if_free = original_lock + service.intent.write = original_write + + self.assertEqual(order[:2], ["lock", "write"]) + self.assertFalse(service.intent.read()["enabled"]) + def test_enable_still_works_when_nothing_is_running_here(self): """A stopped registration is not a reason to make a state directory unusable.""" service = self.service("b") @@ -453,6 +505,53 @@ def test_stop_refuses_a_recycled_pid(self): self.assertIn("reused", refused["detail"]) self.assertIsNone(child.poll()) + def test_stop_leaves_no_request_when_a_supervisor_starts_in_the_window(self): + """The probe and the write were two operations, with a startup in between. + + supervise() clears pending requests BEFORE it acquires the lock, so a request left + by a stop that saw the lock free is consumed by the supervisor that took it - which + then exits, although the stop reported not_running. + """ + service = self.service("a") + service.enable(actor="test") + original = service.daemon_lock_if_free + calls = {"n": 0} + + @contextlib.contextmanager + def taken_by_a_starting_supervisor(): + calls["n"] += 1 + # The lock is free when the old code probed it and held by the time the decision + # is actually made. Holding it across the decision is what closes the window. + yield None + + service.daemon_lock_if_free = taken_by_a_starting_supervisor + try: + refused = service.stop() + finally: + service.daemon_lock_if_free = original + + self.assertEqual(calls["n"], 1, "the decision must be taken under the lock") + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertFalse( + service.stop_request_path.exists(), + "a stop that could not establish ownership must leave nothing behind", + ) + + def test_stop_reports_not_running_without_a_request_when_the_lock_is_free(self): + """Holding the lock IS the proof that nothing is running and nothing can start.""" + service = self.service("a") + service.enable(actor="test") + + outcome = service.stop() + + self.assertFalse(outcome["ok"]) + self.assertEqual(outcome["reason"], "not_running") + self.assertFalse( + service.stop_request_path.exists(), + "nothing was running, so nothing needs to be told to stop", + ) + def test_stop_writes_no_request_for_a_lock_held_by_an_unidentified_process(self): """A supervisor between taking the lock and writing its record has no identity yet. diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 255a59b..4cae356 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -205,6 +205,61 @@ def test_a_store_already_in_use_keeps_its_directory_after_the_hash_changed(self) finally: os.chdir(here) + def test_a_store_is_found_by_the_socket_it_recorded_not_by_its_hash(self): + """The legacy comparison only helps when THIS invocation uses the old spelling. + + A first post-upgrade command that happens to use the absolute path has + legacy == scope, so the comparison never looks at the store the relative spelling + created - and creating a canonical database then hides it for good, because + afterwards even the old spelling finds the new one. + """ + from codex_session_relay.store import legacy_socket_scope, socket_scope + + os.makedirs(os.path.join(self.tmp, "run"), exist_ok=True) + absolute = os.path.join(self.tmp, "run", "app-server.sock") + open(absolute, "w").close() + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = os.path.join("run", "app-server.sock") + # The store the pre-upgrade installation left behind, under the RELATIVE hash. + previous = os.path.join( + self.home, ".local", "state", "codex-session-relay", + legacy_socket_scope(relative), + ) + os.makedirs(previous) + Store(Path(previous) / "relay.sqlite3", socket_path=relative).close() + + # The first post-upgrade command uses the ABSOLUTE spelling, so the legacy + # comparison is a no-op: legacy == scope. + self.assertEqual( + legacy_socket_scope(absolute), socket_scope(absolute), + "the fixture needs the case the hash comparison cannot see", + ) + chosen = resolve_state_dir(None, absolute) + finally: + os.chdir(here) + + self.assertEqual(str(chosen.path), previous, + "the store recorded for this socket must be adopted, not hidden") + self.assertIn("adopted", chosen.detail) + + def test_a_store_with_no_recorded_socket_is_never_adopted_on_a_guess(self): + """Provenance or nothing. Adopting an unlabelled store is how the wrong one is served.""" + from codex_session_relay.store import socket_scope, stores_without_provenance + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + stranger = os.path.join(root, "0123456789abcdef") + os.makedirs(stranger) + Store(Path(stranger) / "relay.sqlite3").close() + + chosen = resolve_state_dir(None, "/run/brand-new.sock") + + self.assertEqual(chosen.socket_scope, socket_scope("/run/brand-new.sock")) + self.assertNotEqual(str(chosen.path), stranger) + # But it IS visible, so an operator is not left guessing why a store looks empty. + self.assertIn(stranger, stores_without_provenance(root, skip=chosen.path.name)) + def test_a_fresh_socket_uses_the_canonical_directory(self): """The fallback is for an existing store only; nothing new lands in the old name.""" from codex_session_relay.store import socket_scope From 9462291359e57b1d57424c7e5c8bd8867d3537a1 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 02:08:59 +0900 Subject: [PATCH 19/29] Four more places where a second launch was not accounted for Two of these are mine from the previous commit, which is the honest way to say it. stop() re-reads the daemon record after terminating the supervisor, so that a worker replaced mid-stop is the one actually reached. But a start that acquires the lock the moment the old supervisor lets go publishes ITS record there, and acting on that stopped the new launch's worker and then overwrote its record with the old launch's outcome - reporting success while the replacement was alive with a corrupted record. The re-read is now adopted only when the launch id still matches; when a different launch has published, neither its worker nor its record is touched. start() reported success by building a status from a RelayService whose store_id is still None on a first start: this process only probed a path that may not have existed yet, and the child is what minted the store. conflicts() therefore could not recognise the child's own scope registration as this store and reported the service we had just started as same_scope_different_store. The launch record carries the identity, so the parent adopts it before constructing the status. supervise() started its deadline timer after on_start returned. Recovery makes real App Server calls for unresolved attempts, so --deadline N could spend an arbitrary startup interval and then run for another N, spawning a worker past the bound it was given. The timer now covers the whole run. assert_assignment_delivery chose the recipient with an if/else on one kind, so any unrecognised kind fell through to the parent branch and was authorized as a completion. The parent then received a verification request that the acknowledgement path refuses - a delivery nobody can answer. Both directions are named explicitly and anything else is refused as having no authorized recipient. Relay suite 646 passed, 30 skipped. validate.py, contracts.py, the scripts/ci/tests unittest suite, secrets.sh and git diff --check all exit 0. --- .../src/codex_session_relay/scope.py | 17 +++++-- .../src/codex_session_relay/service.py | 29 ++++++++++-- .../codex-session-relay/tests/test_service.py | 45 +++++++++++++++++++ 3 files changed, 83 insertions(+), 8 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/scope.py b/packages/codex-session-relay/src/codex_session_relay/scope.py index cc273fa..d02b227 100644 --- a/packages/codex-session-relay/src/codex_session_relay/scope.py +++ b/packages/codex-session-relay/src/codex_session_relay/scope.py @@ -126,10 +126,19 @@ def assert_assignment_delivery(relationship, *, kind, recipient_task_id, RefusalReason.RECIPIENT_NOT_AUTHORIZED, f"event belongs to relationship {event_relationship_id!r}, not {rid!r}", ) - expected = ( - relationship["child"]["taskId"] if kind == "revision_request" - else relationship["parent"]["taskId"] - ) + # Named explicitly rather than defaulted. An unrecognised kind falling through to the + # parent branch was admitted as a completion, so the parent received a verification + # request the acknowledgement path then refuses - a delivery nobody can answer. + if kind == "revision_request": + expected = relationship["child"]["taskId"] + elif kind == "completion_event": + expected = relationship["parent"]["taskId"] + else: + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"{kind!r} is not a delivery direction this contract defines, so there is no" + " authorized recipient for it", + ) if recipient_task_id != expected: direction = "child" if kind == "revision_request" else "parent" raise ScopeError( diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 2fa0686..2ea3757 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -773,11 +773,22 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # first read and now means the pid we started with names a worker that has already # exited, and stopping that one would report success while the replacement, which # holds the inherited locks, is still delivering. - worker = self._stop_worker(self.record() or record, timeout=timeout, grace=grace) - record = self.record() or record + # + # But only when the re-read still describes the SAME launch. A start that acquires + # the lock after the old supervisor exits publishes its own record here, and acting + # on that would stop the new launch's worker and clear the new supervisor's pid using + # the old one's outcome - reporting success while the replacement is still alive. + fresh = self.record() + superseded_by_a_new_launch = ( + fresh is not None and record is not None + and fresh.get("launchId") != record.get("launchId") + ) + if fresh is not None and not superseded_by_a_new_launch: + record = fresh + worker = self._stop_worker(record, timeout=timeout, grace=grace) supervisor_done = outcome in ("exited", "gone") worker_done = worker in ("exited", "gone") - if record is not None: + if record is not None and not superseded_by_a_new_launch: # Identity is kept until termination is CONFIRMED. Clearing a pid we have not # seen exit would lose the only handle a later stop has to reach it. cleared = dict(record, stoppedBy=actor) @@ -936,6 +947,12 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl # running - or about to fail on an App Server connection - is not success either. if (record.get("pid") and record.get("readyAt") and record.get("launchId") == launch and self.lock_is_held()): + # The child minted or opened the store; this process only probed a path that + # may not have existed yet. Without adopting its identity, conflicts() cannot + # recognise the child's own scope registration as this store and reports the + # service we just started as same_scope_different_store. + if self.store_id is None and record.get("storeId"): + self.store_id = record["storeId"] return {"ok": True, "reason": None, "pid": record["pid"], "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), "status": self.status()} @@ -1102,13 +1119,17 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= scope_fd = claim["lockFd"] self.write_record(self.new_record(pid=os.getpid(), token=token)) try: + # Started BEFORE on_start, so the bound covers the whole supervisor run. + # Recovery can make real App Server calls for unresolved attempts, and timing + # only the loop let --deadline N spend an arbitrary startup interval first and + # then run for another N - even spawning a worker past the requested bound. + started = time.monotonic() if on_start is not None: on_start() # Only now is this service serving. Recovery runs before any worker can send, # and a caller told "started" while it was still in flight would go on to use # a service that might yet fail to initialise at all. self._note(readyAt=_now()) - started = time.monotonic() while True: if max_segments is not None and len(segments) >= max_segments: break diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 3124aa1..516a91d 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -27,6 +27,9 @@ REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) SOCKET = "/nonexistent-app-server.sock" +# A child that does nothing but stay alive, for tests that only need a live pid. +IDLE_CHILD = "import time\nwhile True:\n time.sleep(0.05)\n" + # A child that takes the claim and holds it until told to stop, so exclusion is observed # rather than asserted against a double. HOLDER = """ @@ -608,6 +611,48 @@ def test_stop_terminates_a_process_this_installation_owns(self): self.assertIsNone(service.record()["pid"]) self.assertFalse(service.lock_is_held()) + def test_stop_does_not_reach_into_a_replacement_launch(self): + """Re-reading after the supervisor exits can pick up a NEW launch's record. + + Acting on that stops the replacement's worker and clears the replacement's pid using + the outcome of the supervisor we actually stopped - reporting success while the new + service is still alive. + """ + service = self.service("a") + child, _pid = self.holder(service) + ours = dict(service.record(), launchId="launch-ours") + service.write_record(ours) + replacement = subprocess.Popen( + [sys.executable, "-c", IDLE_CHILD], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(replacement) + original = service._terminate + + def publish_a_new_launch(handle, **kwargs): + outcome = original(handle, **kwargs) + # A start that acquired the lock the moment ours let go. + service.write_record(dict( + ours, launchId="launch-theirs", pid=os.getpid(), + workerPid=replacement.pid, + workerStartTicks=service_module.start_ticks(replacement.pid), + )) + return outcome + + with mock.patch.object(service, "_terminate", publish_a_new_launch): + service.stop() + + child.wait(timeout=10) + self.assertIsNone( + replacement.poll(), + "the replacement launch's worker is not ours to signal", + ) + self.assertEqual( + service.record()["launchId"], "launch-theirs", + "and its record must not be overwritten with our outcome", + ) + self.assertEqual(service.record()["workerPid"], replacement.pid) + def test_stop_reaches_the_worker_the_record_names_now(self): """A supervisor replacing a worker while stop runs left the first read stale. From 903a2ce193c1aee4c85a9b813cceb61246bf92bb Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 02:22:23 +0900 Subject: [PATCH 20/29] Two launches with no launch id are still two launches The replacement guard I added compared launchId alone, and a direct 'service run' never has one - so two anonymous launches compared equal and the guard handed the replacement's worker and record straight back to the stop it was meant to protect them from. Identity is the launch id together with the supervisor's pid and start time, which differ across a replacement whether or not an id was ever assigned. And when a replacement IS detected, the stop now says so. It protected the new launch but still computed ok from the old launch's outcome, which told a caller the relay was down while it was still delivering. That case is a refusal: replaced_by_new_launch, naming what was left untouched. supervise published readyAt even when on_start had already consumed the whole deadline. The loop exits immediately in that case, so a waiting start() was being handed a service that was already on its way out. Readiness is not published when the bound is spent; start then reports did_not_report and reaps the child, which is what actually happened. Relay suite 647 passed, 30 skipped. --- .../src/codex_session_relay/service.py | 36 +++++++++++++++++-- .../codex-session-relay/tests/test_service.py | 34 ++++++++++++++++++ 2 files changed, 67 insertions(+), 3 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 2ea3757..dffddc0 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -779,16 +779,29 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # on that would stop the new launch's worker and clear the new supervisor's pid using # the old one's outcome - reporting success while the replacement is still alive. fresh = self.record() + # Identity, not just the launch id. A direct 'service run' carries no launch id at + # all, so comparing that field alone made two anonymous launches look like one and + # handed the replacement's worker and record straight back to this stop. The + # supervisor's own pid and start time distinguish them whether or not a launch id + # was ever assigned. superseded_by_a_new_launch = ( fresh is not None and record is not None - and fresh.get("launchId") != record.get("launchId") + and self._launch_identity(fresh) != self._launch_identity(record) ) if fresh is not None and not superseded_by_a_new_launch: record = fresh worker = self._stop_worker(record, timeout=timeout, grace=grace) + if superseded_by_a_new_launch: + # A replacement is running. Whatever we did to the launch we started from, the + # SERVICE is not stopped, and reporting success from the old launch's outcome + # would tell a caller the relay is down while it is still delivering. + return {"ok": False, "reason": "replaced_by_new_launch", + "detail": "a new launch acquired the daemon lock during this stop; it was" + " left untouched and is still running", + "supervisor": outcome, "worker": "untouched"} supervisor_done = outcome in ("exited", "gone") worker_done = worker in ("exited", "gone") - if record is not None and not superseded_by_a_new_launch: + if record is not None: # Identity is kept until termination is CONFIRMED. Clearing a pid we have not # seen exit would lose the only handle a later stop has to reach it. cleared = dict(record, stoppedBy=actor) @@ -807,6 +820,17 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) return {"ok": ok, "reason": None if ok else "did_not_exit", "detail": None, "supervisor": outcome, "worker": worker} + @staticmethod + def _launch_identity(record): + """What distinguishes one supervisor's run from the next one's. + + The launch id alone is not enough: a direct 'service run' never has one, so two + anonymous launches compare equal. The supervisor's pid and the moment it started + differ across a replacement whether or not a launch id was assigned. + """ + record = record or {} + return (record.get("launchId"), record.get("pid"), record.get("startedAt")) + def _settle_absent_owner(self, record, detail): """Decide 'nothing is running' while HOLDING the lock that would prove otherwise. @@ -1129,7 +1153,13 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= # Only now is this service serving. Recovery runs before any worker can send, # and a caller told "started" while it was still in flight would go on to use # a service that might yet fail to initialise at all. - self._note(readyAt=_now()) + # + # And not at all if the bound was spent getting ready: the loop below exits + # immediately in that case, so publishing readiness here would let a waiting + # start() report a service that is already on its way out. + expired = deadline is not None and time.monotonic() - started >= deadline + if not expired: + self._note(readyAt=_now()) while True: if max_segments is not None and len(segments) >= max_segments: break diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 516a91d..e6c0051 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -611,6 +611,40 @@ def test_stop_terminates_a_process_this_installation_owns(self): self.assertIsNone(service.record()["pid"]) self.assertFalse(service.lock_is_held()) + def test_two_anonymous_launches_are_still_distinguishable(self): + """A direct 'service run' carries no launch id, so comparing that field alone made + two different launches compare equal - and handed the replacement straight back.""" + service = self.service("a") + child, _pid = self.holder(service) + ours = dict(service.record(), launchId=None) + service.write_record(ours) + replacement = subprocess.Popen( + [sys.executable, "-c", IDLE_CHILD], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(replacement) + original = service._terminate + + def publish_anonymous_replacement(handle, **kwargs): + outcome = original(handle, **kwargs) + service.write_record(dict( + ours, launchId=None, pid=os.getpid(), startedAt="2099-01-01T00:00:00Z", + workerPid=replacement.pid, + workerStartTicks=service_module.start_ticks(replacement.pid), + )) + return outcome + + with mock.patch.object(service, "_terminate", publish_anonymous_replacement): + outcome = service.stop() + + child.wait(timeout=10) + self.assertIsNone(replacement.poll(), "the replacement's worker is not ours") + self.assertFalse( + outcome["ok"], + "a stop that left a replacement running has not stopped the service", + ) + self.assertEqual(outcome["reason"], "replaced_by_new_launch") + def test_stop_does_not_reach_into_a_replacement_launch(self): """Re-reading after the supervisor exits can pick up a NEW launch's record. From d9d688cd5c46ac5dce316e3cbe0a876de77d3a4e Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 02:35:28 +0900 Subject: [PATCH 21/29] Order launch identity by stability, so a normal exit is not a replacement The identity tuple I added compared launch id, pid and start time all at once. A supervisor finishing naturally clears its OWN pid during cleanup while keeping the launch id and the start time, so that ordinary lifecycle update looked like a different launch - and a stop that genuinely stopped the service reported replaced_by_new_launch instead of success. Taken in order of stability now: the launch id when there is one, the start time for an anonymous run, and the pid only for a record that has neither. That still separates two anonymous launches, which is what the tuple was widened for. Relay suite 648 passed, 30 skipped. --- .../src/codex_session_relay/service.py | 11 +++++++- .../codex-session-relay/tests/test_service.py | 25 +++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index dffddc0..bbc7e5d 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -827,9 +827,18 @@ def _launch_identity(record): The launch id alone is not enough: a direct 'service run' never has one, so two anonymous launches compare equal. The supervisor's pid and the moment it started differ across a replacement whether or not a launch id was assigned. + + Taken in order of stability rather than all at once. A supervisor finishing normally + clears its own pid during cleanup while keeping the launch id and start time, so a + tuple including the pid turned an ordinary exit into a phantom replacement and made + a successful stop report failure. """ record = record or {} - return (record.get("launchId"), record.get("pid"), record.get("startedAt")) + if record.get("launchId") is not None: + return ("launchId", record["launchId"]) + if record.get("startedAt") is not None: + return ("startedAt", record["startedAt"]) + return ("pid", record.get("pid")) def _settle_absent_owner(self, record, detail): """Decide 'nothing is running' while HOLDING the lock that would prove otherwise. diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index e6c0051..6da4501 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -611,6 +611,31 @@ def test_stop_terminates_a_process_this_installation_owns(self): self.assertIsNone(service.record()["pid"]) self.assertFalse(service.lock_is_held()) + def test_a_supervisor_finishing_normally_is_not_a_replacement(self): + """Cleanup clears the supervisor's own pid while keeping its launch id. + + A launch identity that included the pid turned that ordinary exit into a phantom + replacement, so a stop that genuinely stopped the service reported failure. + """ + service = self.service("a") + live = dict(service.new_record(pid=os.getpid()), launchId="launch-7") + cleaned = dict(live, pid=None) + + self.assertEqual( + service._launch_identity(live), service._launch_identity(cleaned), + "the same launch clearing its own pid is still the same launch", + ) + self.assertNotEqual( + service._launch_identity(live), + service._launch_identity(dict(live, launchId="launch-8")), + ) + # And anonymous runs still separate on when they started. + anon = dict(live, launchId=None, startedAt="2026-01-01T00:00:00Z") + self.assertNotEqual( + service._launch_identity(anon), + service._launch_identity(dict(anon, startedAt="2026-01-02T00:00:00Z")), + ) + def test_two_anonymous_launches_are_still_distinguishable(self): """A direct 'service run' carries no launch id, so comparing that field alone made two different launches compare equal - and handed the replacement straight back.""" From 471e50e746b1dc45c88aa89c413ef4e8bf9ea713 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 02:46:48 +0900 Subject: [PATCH 22/29] Two refinements to identity and discovery, both from review _now() records whole seconds, so two anonymous launches starting in the same second share a startedAt - and a replacement acquiring the lock inside that second compared equal to the launch being stopped, which is the case the identity tuple exists to separate. The anonymous branch carries the recorded start ticks alongside; the kernel's start-time counter for a pid does not collide. Socket discovery returned whichever matching directory sorted first. Two stores recording the same socket means a copy or separate explicit-state runs, and choosing between them silently operates on one set of assignments today and the other after a rename. Discovery adopts a single match and refuses an ambiguous one; doctor reports every store claiming the socket and flags the ambiguity, so the caller sees why a database looks unexpectedly empty instead of guessing. Adopting nothing is still the wrong database. It is the visible wrong database, which --state can correct. Relay suite 650 passed, 30 skipped. --- .../src/codex_session_relay/cli.py | 11 +++++- .../src/codex_session_relay/service.py | 6 +++- .../src/codex_session_relay/store.py | 27 +++++++++++++- .../codex-session-relay/tests/test_store.py | 36 +++++++++++++++++++ 4 files changed, 77 insertions(+), 3 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 02058b1..55dd34d 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -785,7 +785,16 @@ def _sibling_stores(services) -> dict: return {"checked": False, "reason": "the state directory was chosen explicitly", "withoutProvenance": []} without = stores_without_provenance(root, skip=services.selection.path.name) - return {"checked": True, "reason": None, "withoutProvenance": without} + from .store import stores_claiming_socket + + # More than one store recording this socket is an ambiguity discovery refuses to resolve, + # so it has to be visible here or a caller just gets a surprisingly empty database. + claiming = stores_claiming_socket( + root, services.socket_path, skip=services.selection.path.name, + ) + return {"checked": True, "reason": None, "withoutProvenance": without, + "claimingThisSocket": claiming, + "ambiguous": len(claiming) > 1} def cmd_doctor(services, args) -> dict: diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index bbc7e5d..34fe4a7 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -837,7 +837,11 @@ def _launch_identity(record): if record.get("launchId") is not None: return ("launchId", record["launchId"]) if record.get("startedAt") is not None: - return ("startedAt", record["startedAt"]) + # With the start ticks, because _now() records whole seconds: two anonymous + # launches inside the same second share a startedAt, and a replacement acquiring + # the lock in that second would compare equal to the launch being stopped. The + # kernel's start-time counter for that pid does not collide. + return ("startedAt", record["startedAt"], record.get("startTicks")) return ("pid", record.get("pid")) def _settle_absent_owner(self, record, detail): diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 66b93d5..18c5d57 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -628,6 +628,7 @@ def discover_store_for_socket(root, socket_path, *, skip=None): except OSError: return None wanted = canonical_socket(socket_path) + matches = [] for directory in candidates: if skip is not None and directory.name == skip: continue @@ -635,10 +636,34 @@ def discover_store_for_socket(root, socket_path, *, skip=None): if not database.exists(): continue if store_socket(database) == wanted: - return directory + matches.append(directory) + if len(matches) == 1: + return matches[0] + # Zero is the ordinary case. More than one means a copied store or separate explicit-state + # runs left two databases claiming the same socket, and picking whichever sorts first + # would silently operate on one set of assignments today and the other after a rename. + # Adopting nothing sends the caller to a fresh canonical store, which is wrong but VISIBLE: + # doctor reports the candidates and --state resolves it. return None +def stores_claiming_socket(root, socket_path, *, skip=None) -> list: + """Every store recording this socket. More than one is an ambiguity, not a choice.""" + if not socket_path: + return [] + try: + candidates = sorted(p for p in Path(root).iterdir() if p.is_dir()) + except OSError: + return [] + wanted = canonical_socket(socket_path) + return [ + str(directory) for directory in candidates + if (skip is None or directory.name != skip) + and (directory / "relay.sqlite3").exists() + and store_socket(directory / "relay.sqlite3") == wanted + ] + + def stores_without_provenance(root, *, skip=None) -> list: """Store directories that never recorded which socket they serve. diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 4cae356..3bb9a7c 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -244,6 +244,42 @@ def test_a_store_is_found_by_the_socket_it_recorded_not_by_its_hash(self): "the store recorded for this socket must be adopted, not hidden") self.assertIn("adopted", chosen.detail) + def test_two_stores_claiming_one_socket_are_not_silently_chosen_between(self): + """Picking whichever sorts first operates on one set of assignments today and the + other after a rename. Adopting nothing is wrong too, but it is visible.""" + from codex_session_relay.store import ( + discover_store_for_socket, stores_claiming_socket, + ) + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + socket = "/run/contested.sock" + both = [] + for name in ("aaaa000000000000", "bbbb000000000000"): + directory = os.path.join(root, name) + os.makedirs(directory) + Store(Path(directory) / "relay.sqlite3", socket_path=socket).close() + both.append(directory) + + self.assertIsNone( + discover_store_for_socket(Path(root), socket), + "an ambiguity is not a choice to make silently", + ) + self.assertEqual(sorted(stores_claiming_socket(Path(root), socket)), sorted(both)) + + def test_one_store_claiming_a_socket_is_still_adopted(self): + """The refusal must not swallow the case discovery exists for.""" + from codex_session_relay.store import discover_store_for_socket + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + socket = "/run/sole.sock" + directory = os.path.join(root, "cccc000000000000") + os.makedirs(directory) + Store(Path(directory) / "relay.sqlite3", socket_path=socket).close() + + self.assertEqual( + str(discover_store_for_socket(Path(root), socket)), directory, + ) + def test_a_store_with_no_recorded_socket_is_never_adopted_on_a_guess(self): """Provenance or nothing. Adopting an unlabelled store is how the wrong one is served.""" from codex_session_relay.store import socket_scope, stores_without_provenance From 5ee8b6d23ceb13d160b9dbc02a2616155aae7501 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 03:24:41 +0900 Subject: [PATCH 23/29] Adopting neither store was not the visible failure I argued it was Round three answered a contested socket by having discover_store_for_socket return None, on the reasoning that adopting neither candidate is wrong but at least visible. It is not visible. resolve_state_dir cannot tell that None from "nothing found", so it falls through to the canonical directory and the first command that writes there creates a THIRD empty database - after which the canonical-exists branch at the top of the function wins every later resolution and both real stores, with their assignments and pending deliveries, are hidden for good. That is worse than the first-match behaviour it replaced. So the ambiguity travels on the selection instead of being discarded at the bottom of a helper. resolve_state_dir asks stores_claiming_socket directly and branches on the count; discover_store_for_socket becomes the "exactly one" wrapper over that same walk, so there is one implementation of it rather than two. The command line refuses when a selection carries candidates, naming them and the sequence that resolves them. doctor and ack-proof are exempt for opposite reasons: doctor is how an operator learns which store to pass, and ack-proof opens no store at all. An explicit --state or environment override never reaches the check, because both return before discovery runs. The second change is the same kind of mistake in disable. A supervisor that has taken the daemon lock and not yet published its record leaves ownership() with nothing to classify, so the old condition - foreign or unverifiable or foreign markers - was false and the shared enabled=false was written anyway. The starting supervisor reads that at its first worker boundary and exits, which is a refusal that still had its effect. stop() already refuses this; disable did not, and neither did enable. The test is not whether the owner is ours but whether the holder can be attributed to us at all. An owned orphan still counts: a supervisor that died leaving our own worker alive answers none while that worker holds the lock through the inherited descriptor, and refusing there would leave an owner unable to disable their own orphan - which stop() deliberately still reaches. A recorded worker number is not enough on its own, though. It has to be alive, readable and still carrying the start time we recorded, and it has to have a boot id where the host has one, because ownership() answers none at its already-gone branch before its own missing-boot check ever runs. enable's other race - it reads ownership before probing the lock, so a foreign holder can replace an owned supervisor between the two - is older than this and is left recorded rather than folded in. --- .../src/codex_session_relay/cli.py | 43 +++++++++++ .../src/codex_session_relay/service.py | 67 +++++++++++++++-- .../src/codex_session_relay/store.py | 55 +++++++------- .../codex-session-relay/tests/test_cli.py | 65 ++++++++++++++++ .../codex-session-relay/tests/test_service.py | 75 +++++++++++++++++++ .../codex-session-relay/tests/test_store.py | 28 +++++++ 6 files changed, 299 insertions(+), 34 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 55dd34d..2dce5c5 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -1369,12 +1369,55 @@ def build_parser() -> argparse.ArgumentParser: return parser +def _refuse_ambiguous_state(services, args) -> None: + """Two stores already record this socket, so opening one of them would be a guess. + + Falling through to the canonical directory is not the neutral outcome it looks like. The + first command that writes there creates a THIRD empty database, and once that exists it + wins every later resolution and hides the assignments and pending deliveries in both of + the others. Refusing costs one command; the third store costs the state. + + doctor and ack-proof are exempt for opposite reasons. doctor is how an operator finds out + which store to pass to --state, so refusing it would remove the only way out. ack-proof is + a derivation over its own two arguments that opens no store at all. + + An explicit --state or environment override never arrives here: both return from + resolve_state_dir before any discovery runs, because a caller who named a directory has + already decided which participants share it. + + This guard is on the command line rather than on Services.store. A library caller that + builds Services itself bypasses it; every in-process caller in this package passes an + explicit directory, and raising from a property would turn a diagnostic into a crash. + """ + if not services.selection.ambiguous: + return + if getattr(args, "handler", None) in (cmd_doctor, cmd_ack_proof): + return + raise PayloadExit({ + "error": "refused", + "reason": "ambiguous_state_directory", + "detail": ( + "more than one store already records this socket, and creating a new one here" + " would hide them both" + ), + "socketPath": services.socket_path, + "candidates": list(services.selection.ambiguous), + "wouldHaveCreated": str(services.selection.db_path), + "recover": [ + "doctor lists the candidates", + "--state doctor identifies the store", + "--state service status shows which assignments it carries", + ], + }, EXIT_REFUSED) + + def main(argv=None) -> int: parser = build_parser() args = parser.parse_args(argv) services = None try: services = Services(args) + _refuse_ambiguous_state(services, args) payload = args.handler(services, args) print(json.dumps(payload, indent=2, default=str)) return EXIT_OK diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 34fe4a7..e028673 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -521,6 +521,53 @@ def _foreign_markers(self, record) -> list: markers.append("it is using a different store") return markers + def _worker_identified(self, record) -> bool: + """Whether the recorded worker pid provably names OUR worker, still running. + + The same three questions _stop_worker asks before it signals - the process exists, it + is readable, and its start time matches what we recorded - plus the boot question, + which ownership() does not always reach. When the recorded SUPERVISOR pid is gone it + answers none at the already-gone branch, before its own missing-boot check runs, so a + record written before a reboot arrives here naming a worker number that an unrelated + process may now hold. + """ + pid = (record or {}).get("workerPid") + if not pid: + return False + if record.get("bootId") is None and boot_id() is not None: + # Conditional on the host for the reason ownership() gives: where no boot id is + # available at all, every record lacks one, and refusing them all would leave a + # service unusable by its own owner. + return False + handle = ProcessHandle(pid) + try: + if handle.already_gone or not handle.usable: + return False + ticks = record.get("workerStartTicks") + current = start_ticks(pid) + return ticks is not None and current is not None and current == ticks + finally: + handle.close() + + def _holder_is_ours(self, record, owner, markers) -> bool: + """Whether whatever is holding this state directory can be attributed to us. + + Not the same question as "is the owner ours". A supervisor that died leaving our own + worker alive answers none, and that worker still holds the daemon lock through the + descriptor it inherited - refusing there would stop an owner from disabling their own + orphan, which stop() deliberately still reaches. + + Everything else that answers none while the lock is held is a holder we cannot name: + no record at all, a record whose process is gone, or a supervisor that has taken the + lock and not yet published. Writing the shared intent on any of those is how a refused + command still shuts down another installation. + """ + if markers: + return False + if owner == OURS: + return True + return owner == NONE and self._worker_identified(record) + def lock_is_held(self) -> bool: """Probed by trying to take it: the lock is the only honest liveness signal.""" path = self.selection.path / DAEMON_LOCK @@ -595,13 +642,17 @@ def enable(self, *, actor: str = "cli") -> dict: # clears its pids before releasing the lock, and ownership answers none once both are # absent - which is exactly the interval in which a foreign shutdown could be reversed. markers = self._foreign_markers(record) if record else [] - if self.lock_is_held() and (owner in (FOREIGN, UNVERIFIABLE) or markers): + if self.lock_is_held() and not self._holder_is_ours(record, owner, markers): # Only while something is actually RUNNING here. A stopped registration belonging # to someone else is not a reason to refuse an owner configuring their own # installation, and refusing then would make a state directory unusable forever. return {"ok": False, - "reason": "ownership_unverifiable" if owner == UNVERIFIABLE else "not_ours", - "detail": detail or "; ".join(markers), "intent": self.intent.read(), + "reason": ("not_ours" if (owner == FOREIGN or markers) + else "ownership_unverifiable"), + "detail": detail or "; ".join(markers) or ( + "the daemon lock is held but nothing here identifies its owner" + ), + "intent": self.intent.read(), "note": "intent is shared with the owner of this state directory and was" " left unchanged"} return {"ok": True, "reason": None, @@ -629,12 +680,14 @@ def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: if handle is not None: handle.close() markers = self._foreign_markers(record) if record else [] - if owner in (FOREIGN, UNVERIFIABLE) or markers: + if not self._holder_is_ours(record, owner, markers): refusal = { "ok": False, - "reason": ("ownership_unverifiable" if owner == UNVERIFIABLE - else "not_ours"), - "detail": detail or "; ".join(markers), + "reason": ("not_ours" if (owner == FOREIGN or markers) + else "ownership_unverifiable"), + "detail": detail or "; ".join(markers) or ( + "the daemon lock is held but nothing here identifies its owner" + ), "intent": self.intent.read(), "stop": {"ok": False, "reason": "refused", "supervisor": "untouched", "worker": "untouched"}, diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 18c5d57..4542911 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -522,6 +522,11 @@ class StateSelection: source: str detail: str socket_scope: str | None + # Every store that already records this socket, when there is more than one of them and + # no canonical store exists yet. Empty for every ordinary selection. It travels ON the + # selection because the decision not to create a store here has to reach whoever holds + # the path they would otherwise have created one at. + ambiguous: tuple = () @property def db_path(self) -> Path: @@ -531,7 +536,7 @@ def to_record(self) -> dict: return { "path": str(self.path), "dbPath": str(self.db_path), "source": self.source, "detail": self.detail, "socketScope": self.socket_scope, - "precedence": list(PRECEDENCE), + "precedence": list(PRECEDENCE), "ambiguous": list(self.ambiguous), } @@ -586,13 +591,26 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: # a canonical database here would hide it for good, because afterwards even the old # spelling finds the new one. So before creating anything, ask the stores themselves. # Only reached when no canonical database exists yet, which is the one moment it matters. - adopted = discover_store_for_socket(base / "codex-session-relay", socket_path, skip=scope) - if adopted is not None: + claims = stores_claiming_socket(base / "codex-session-relay", socket_path, skip=scope) + if len(claims) == 1: + adopted = Path(claims[0]) return StateSelection( adopted, source, f"{detail}; adopted the store already recorded for this socket", adopted.name, ) + if len(claims) > 1: + # Returning the canonical directory with nothing to say about the conflict is how a + # THIRD store gets made. The first command to write here creates it, and from that + # moment the canonical-exists branch at the top of this function wins every later + # resolution, so both of the real stores - with their assignments, generations and + # pending deliveries - are invisible. Choosing between them would be just as wrong in + # a quieter way. So the ambiguity travels with the path and the caller refuses. + return StateSelection( + chosen, source, + f"{detail}; {len(claims)} stores already record this socket", + scope, tuple(claims), + ) return StateSelection(chosen, source, detail, scope) @@ -620,31 +638,14 @@ def discover_store_for_socket(root, socket_path, *, skip=None): spelling we cannot guess is only findable if it says which socket it belongs to. Stores record that from now on; one created before it did says nothing and is reported by doctor instead of being adopted on a guess. + + Exactly one, because two stores claiming one socket is an ambiguity rather than a choice. + This answers the narrow question "is there a single store to adopt". A caller that has to + ACT on the difference between none and several reads stores_claiming_socket, which is the + one walk both of them share. """ - if not socket_path: - return None - try: - candidates = sorted(p for p in Path(root).iterdir() if p.is_dir()) - except OSError: - return None - wanted = canonical_socket(socket_path) - matches = [] - for directory in candidates: - if skip is not None and directory.name == skip: - continue - database = directory / "relay.sqlite3" - if not database.exists(): - continue - if store_socket(database) == wanted: - matches.append(directory) - if len(matches) == 1: - return matches[0] - # Zero is the ordinary case. More than one means a copied store or separate explicit-state - # runs left two databases claiming the same socket, and picking whichever sorts first - # would silently operate on one set of assignments today and the other after a rename. - # Adopting nothing sends the caller to a fresh canonical store, which is wrong but VISIBLE: - # doctor reports the candidates and --state resolves it. - return None + claims = stores_claiming_socket(root, socket_path, skip=skip) + return Path(claims[0]) if len(claims) == 1 else None def stores_claiming_socket(root, socket_path, *, skip=None) -> list: diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index 241faeb..3c64ae2 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -438,3 +438,68 @@ def test_closing_without_ever_using_the_store_creates_nothing(self): built = Services(argparse.Namespace(state=empty, socket=None)) built.close() self.assertFalse(os.path.exists(empty)) + + +class ContestedSocket(CliBase): + """Two stores recording one socket must not quietly become three. + + These runs deliberately pass no --state: the whole question is what the environment alone + resolves to, and an explicit directory answers it before discovery ever runs. + """ + + def contested(self, name): + """A home holding two stores that both record one socket.""" + from pathlib import Path + + from codex_session_relay.store import Store + + home = os.path.join(self.tmp, name) + root = os.path.join(home, ".local", "state", "codex-session-relay") + socket = os.path.join(self.tmp, f"{name}.sock") + for directory in ("aaaa444444444444", "bbbb444444444444"): + os.makedirs(os.path.join(root, directory)) + Store(Path(root) / directory / "relay.sqlite3", socket_path=socket).close() + return home, root, socket + + def run_in_home(self, home, *args, expect=0): + environment = dict(os.environ, PYTHONPATH=os.path.join(REPO, "src"), HOME=home) + for name in ("CODEX_SESSION_RELAY_STATE", "XDG_STATE_HOME"): + environment.pop(name, None) + completed = subprocess.run( + [sys.executable, "-m", "codex_session_relay.cli", *args], + capture_output=True, text=True, env=environment, timeout=60, + ) + self.assertEqual( + completed.returncode, expect, + f"exit {completed.returncode}: {completed.stdout}{completed.stderr}", + ) + return json.loads(completed.stdout) + + def test_an_ordinary_command_refuses_rather_than_creating_a_third_store(self): + home, root, socket = self.contested("contested-status") + + refused = self.run_in_home(home, "--socket", socket, "status", expect=2) + + self.assertEqual(refused["reason"], "ambiguous_state_directory") + self.assertEqual(len(refused["candidates"]), 2) + self.assertFalse( + os.path.exists(refused["wouldHaveCreated"]), + "the refusal must not leave behind the store it refused to choose", + ) + self.assertEqual(sorted(os.listdir(root)), ["aaaa444444444444", "bbbb444444444444"]) + + def test_doctor_still_describes_a_contested_socket(self): + home, _root, socket = self.contested("contested-doctor") + + report = self.run_in_home(home, "--socket", socket, "doctor") + + self.assertTrue(report["siblingStores"]["ambiguous"]) + self.assertEqual(len(report["siblingStores"]["claimingThisSocket"]), 2) + + def test_an_explicit_state_directory_resolves_the_contest(self): + home, root, socket = self.contested("contested-explicit") + chosen = os.path.join(root, "aaaa444444444444") + + answer = self.run_in_home(home, "--state", chosen, "--socket", socket, "status") + + self.assertEqual(answer["deliveries"], []) diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 6da4501..7559bdc 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -498,6 +498,81 @@ def test_disable_still_works_on_a_service_this_installation_owns(self): self.assertTrue(disabled["ok"], disabled) self.assertFalse(service.intent.read()["enabled"]) child.wait(timeout=10) + + def test_disable_refuses_a_lock_holder_nothing_here_can_identify(self): + """A supervisor that has taken the lock and not yet published its record. + + ownership() answers none, _foreign_markers has nothing to read from, and the old + condition asked only about foreign and unverifiable - so this wrote the shared + enabled=false anyway. The starting supervisor reads that at its first worker boundary + and exits, which is a refused disable that still shut another installation down. + """ + import fcntl + + service = self.service("c") + service.enable(actor="owner") + service.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(service.selection.path / service_module.DAEMON_LOCK, "a+") + self.addCleanup(handle.close) + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + refused = service.disable(actor="owner") + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertTrue( + service.intent.read()["enabled"], + "a refused disable must not change the intent the starting supervisor reads", + ) + + def test_a_stale_worker_number_does_not_make_a_held_lock_ours(self): + """A recorded worker pid is not ownership until its start time still matches. + + Once that process is gone the number proves nothing, and something else is holding + this directory - which is the case the guard exists for. + """ + import fcntl + + service = self.service("d") + service.enable(actor="owner") + service.write_record(dict( + service.new_record(pid=os.getpid()), pid=None, workerPid=os.getpid(), + workerStartTicks=(service_module.start_ticks(os.getpid()) or 0) + 1, + )) + handle = open(service.selection.path / service_module.DAEMON_LOCK, "a+") + self.addCleanup(handle.close) + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + refused = service.disable(actor="owner") + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertTrue(service.intent.read()["enabled"]) + + def test_disable_still_reaches_an_orphaned_worker_of_our_own(self): + """The guard must not refuse the case stop() deliberately still supports. + + A supervisor that died leaving our own worker alive answers none, and the worker holds + the lock through the descriptor it inherited. Refusing here would leave an owner unable + to disable their own orphan. + """ + service = self.service("e") + service.enable(actor="owner") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + disabled = service.disable(actor="owner") + + child.wait(timeout=10) + self.assertTrue(disabled["ok"], disabled) + self.assertFalse(service.intent.read()["enabled"]) def test_stop_refuses_a_recycled_pid(self): service = self.service("a") child, pid = self.holder(service) diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 3bb9a7c..1c456ba 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -280,6 +280,34 @@ def test_one_store_claiming_a_socket_is_still_adopted(self): str(discover_store_for_socket(Path(root), socket)), directory, ) + def test_two_stores_claiming_one_socket_do_not_produce_a_third(self): + """Adopting neither is not the visible failure it was argued to be. + + The caller lands on the canonical directory, the first write there creates a THIRD + empty database, and from that moment the canonical-exists branch wins every later + resolution - so both real stores are hidden for good. The selection has to carry the + conflict instead, which is what lets the command line refuse. + """ + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + socket = "/run/contested-selection.sock" + both = [] + for name in ("aaaa333333333333", "bbbb333333333333"): + directory = os.path.join(root, name) + os.makedirs(directory) + Store(Path(directory) / "relay.sqlite3", socket_path=socket).close() + both.append(directory) + + chosen = resolve_state_dir(None, socket) + + self.assertEqual(sorted(chosen.ambiguous), sorted(both)) + self.assertEqual(sorted(chosen.to_record()["ambiguous"]), sorted(both)) + self.assertNotIn(str(chosen.path), both, "neither is adopted on a guess") + + def test_an_ordinary_selection_carries_no_ambiguity(self): + """The field is a conflict report, not a list of neighbours.""" + self.assertEqual(resolve_state_dir(None, "/run/quiet.sock").ambiguous, ()) + self.assertEqual(resolve_state_dir(os.path.join(self.tmp, "flag")).ambiguous, ()) + def test_a_store_with_no_recorded_socket_is_never_adopted_on_a_guess(self): """Provenance or nothing. Adopting an unlabelled store is how the wrong one is served.""" from codex_session_relay.store import socket_scope, stores_without_provenance From f416667bfe9743e24b292eec5a09e0ad8ec64eaf Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 03:50:42 +0900 Subject: [PATCH 24/29] Four more ways a lifecycle command acted on something it had not identified Review round five, all four in this PR's own service surface. enable classified ownership and then probed the lock, with the intent write outside it. A foreign supervisor that passes its own enabled-intent check and takes the lock between those two steps gets enabled=true written over a disable that happened during the handoff, which leaves it running and eligible to restart. It now decides and writes under the lock exactly as disable does. I had recorded this one as a follow-up after my own audit raised it; a reviewer found it independently, which is enough to say the boundary was in the wrong place. start launched the supervisor and then waited for it to report. Anything that left that wait without a confirmed result - an exception, a Ctrl-C while recovery is still initialising - skipped the abandon that the timeout path already did. The default launcher puts the child in its own session, so no terminal signal reaches it: it would finish starting up, take both locks and serve a launch the caller had been told nothing about. The wait is its own method now, wrapped so every exit that is not a confirmed result stops the child. ScopeRegistry.claim took the scope lock and then wrote the registration. A failed write left the handle open, so the lock went on refusing every later start in the same process on behalf of a registration that was never published. It closes the handle and clears it before propagating. A supervised worker validated the token, the parent, the paths and the inherited descriptors, but never compared the store it had just opened with the storeId its supervisor recorded. A database deleted or atomically replaced between worker segments is a different store, and without the comparison the worker serves it while the supervisor and the scope registration still name the original - so every participant asking whether they agree is told yes. It is refused now, before the descriptor checks. Each has a regression test verified failing against the pre-fix source. --- .../src/codex_session_relay/cli.py | 8 + .../src/codex_session_relay/service.py | 89 +++++++---- .../codex-session-relay/tests/test_service.py | 146 ++++++++++++++++++ 3 files changed, 215 insertions(+), 28 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 2dce5c5..02d3520 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -919,6 +919,14 @@ def refuse(reason, detail): refuse("supervised_state_mismatch", "the record names a different state directory") if record.get("socketPath") not in (None, service.socket_path): refuse("supervised_socket_mismatch", "the record names a different operating scope") + # The supervisor recorded which store it registered the scope for. This worker opened + # whatever relay.sqlite3 the path resolves to NOW, and a database deleted or atomically + # replaced between segments is a different one - so without this the worker would serve an + # empty or unrelated store while the supervisor and the scope registration still name the + # original, and every participant comparing identities would be told they agree. + if record.get("storeId") not in (None, service.store_id): + refuse("supervised_store_mismatch", + "the record names a different store than this worker opened") try: want = os.stat(service.selection.path / DAEMON_LOCK) got = os.fstat(lock_fd) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index e028673..2ded8c3 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -187,7 +187,18 @@ def claim(self, socket_path, record: dict, *, shared: bool = False): os.set_inheritable(handle.fileno(), True) payload = dict(record, scopeKey=self.key(socket_path), scopeAuthority=self.authority, socketPath=str(socket_path), registeredAt=_now()) - self.record_path(socket_path).write_text(json.dumps(payload, indent=2), encoding="utf-8") + try: + self.record_path(socket_path).write_text( + json.dumps(payload, indent=2), encoding="utf-8", + ) + except OSError: + # The lock is taken but the registration it stands for was never published. Left + # open, this handle goes on refusing every later start in this process on behalf + # of a registration that does not exist - and nothing releases it before the + # object is collected. + self._handle = None + handle.close() + raise return {"ok": True, "reason": None, "record": payload, "lockFd": handle.fileno()} def release(self, socket_path, *, shared: bool = False) -> None: @@ -630,33 +641,42 @@ def authority_check(self, *, allow_isolated: bool): return {"ok": True, "reason": None} def enable(self, *, actor: str = "cli") -> dict: - # The same ownership question disable asks, for the same reason: service.json is - # shared by everything pointed at this state directory. An owner who has just - # disabled a service whose supervisor is still exiting must not have that reversed - # by another installation, which would leave it eligible to restart. - record = self.record() - owner, handle, detail = self.ownership(record) - if handle is not None: - handle.close() - # Read from the record as well as from the classification. A supervisor on its way out - # clears its pids before releasing the lock, and ownership answers none once both are - # absent - which is exactly the interval in which a foreign shutdown could be reversed. - markers = self._foreign_markers(record) if record else [] - if self.lock_is_held() and not self._holder_is_ours(record, owner, markers): - # Only while something is actually RUNNING here. A stopped registration belonging - # to someone else is not a reason to refuse an owner configuring their own - # installation, and refusing then would make a state directory unusable forever. - return {"ok": False, - "reason": ("not_ours" if (owner == FOREIGN or markers) - else "ownership_unverifiable"), - "detail": detail or "; ".join(markers) or ( - "the daemon lock is held but nothing here identifies its owner" - ), - "intent": self.intent.read(), - "note": "intent is shared with the owner of this state directory and was" - " left unchanged"} - return {"ok": True, "reason": None, - "intent": self.intent.write(enabled=True, actor=actor)} + # The same ownership question disable asks, and decided and written under the SAME + # lock for the same reason. service.json is shared by everything pointed at this + # state directory, and classifying then writing are two operations: a foreign + # supervisor can pass its own enabled-intent check and take the lock in between, and + # this call would then write enabled=true over a disable that happened during the + # handoff - leaving that supervisor running and eligible to restart. Holding the lock + # across both means no supervisor can start while we decide, and failing to take it + # is itself the answer that one is already there to be classified. + with self.daemon_lock_if_free() as held: + if held is None: + record = self.record() + owner, handle, detail = self.ownership(record) + if handle is not None: + handle.close() + # Read from the record as well as from the classification. A supervisor on + # its way out clears its pids before releasing the lock, and ownership + # answers none once both are absent - exactly the interval in which a foreign + # shutdown could be reversed. + markers = self._foreign_markers(record) if record else [] + if not self._holder_is_ours(record, owner, markers): + return {"ok": False, + "reason": ("not_ours" if (owner == FOREIGN or markers) + else "ownership_unverifiable"), + "detail": detail or "; ".join(markers) or ( + "the daemon lock is held but nothing here identifies its" + " owner" + ), + "intent": self.intent.read(), + "note": "intent is shared with the owner of this state directory" + " and was left unchanged"} + # Either nothing is running here - a stopped registration belonging to someone + # else is not a reason to refuse an owner configuring their own installation, and + # refusing then would make a state directory unusable forever - or what is + # running is ours. + return {"ok": True, "reason": None, + "intent": self.intent.write(enabled=True, actor=actor)} def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: # Ownership BEFORE the intent write. service.json is shared by everything pointed at @@ -1025,6 +1045,19 @@ def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cl segment_seconds=segment_seconds, max_segments=max_segments, ) deadline_at = time.monotonic() + timeout + try: + return self._await_launch(child, launch, deadline_at, timeout=timeout, poll=poll) + except BaseException: + # Anything that leaves this call without a confirmed result - an exception, a + # KeyboardInterrupt while recovery is still initialising - leaves a child the + # default launcher put in its own session, so no terminal signal reaches it. It + # would finish starting up, take both locks and begin serving a launch the caller + # was never told had succeeded. + self._abandon(child, timeout=timeout) + raise + + def _await_launch(self, child, launch, deadline_at, *, timeout, poll): + """Wait for the child to publish a record this call can recognise as its own launch.""" while time.monotonic() < deadline_at: record = self.record() or {} # Matched on the launch id, not on the pid changing. After a crash the OS can diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 7559bdc..0fd4877 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -499,6 +499,119 @@ def test_disable_still_works_on_a_service_this_installation_owns(self): self.assertFalse(service.intent.read()["enabled"]) child.wait(timeout=10) + def test_enable_takes_the_lock_before_touching_shared_intent(self): + """The ordering itself, so a later edit cannot quietly put the write back first. + + Classifying and then writing are two operations. A foreign supervisor that passes its + own enabled-intent check and takes the lock in between would have enabled=true written + over a disable that happened during the handoff, leaving it running and eligible to + restart. disable already decides under the lock; enable did not. + """ + service = self.service("f") + order = [] + original_lock = service.daemon_lock_if_free + original_write = service.intent.write + + @contextlib.contextmanager + def watched_lock(): + order.append("lock") + with original_lock() as held: + yield held + + def watched_write(**kwargs): + order.append("write") + return original_write(**kwargs) + + service.daemon_lock_if_free = watched_lock + service.intent.write = watched_write + try: + service.enable(actor="owner") + finally: + service.daemon_lock_if_free = original_lock + service.intent.write = original_write + + self.assertEqual(order[:2], ["lock", "write"]) + self.assertTrue(service.intent.read()["enabled"]) + + def test_enable_refuses_a_lock_holder_nothing_here_can_identify(self): + """The same hole disable had: a supervisor that has the lock and has published + nothing is not a supervisor whose owner asked for this.""" + import fcntl + + service = self.service("g") + service.disable(actor="owner") + service.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(service.selection.path / service_module.DAEMON_LOCK, "a+") + self.addCleanup(handle.close) + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + refused = service.enable(actor="owner") + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertFalse( + service.intent.read()["enabled"], + "a refused enable must not reverse the owner's disable", + ) + + def test_an_interrupted_start_does_not_leave_its_child_running(self): + """The default launcher puts the supervisor in its own session, so a terminal signal + never reaches it. Anything that leaves start() without a confirmed result - an + exception, a Ctrl-C while recovery is still initialising - has to stop it, or it comes + up afterwards holding both locks for a launch the caller was told nothing about.""" + service = self.service("h") + service.enable(actor="owner") + + class Interrupted: + returncode = None + + def __init__(self): + self.stopped = [] + self.polls = 0 + + def poll(self): + self.polls += 1 + if self.polls == 1: + raise KeyboardInterrupt + return self.returncode + + def terminate(self): + self.stopped.append("terminate") + self.returncode = -15 + + def kill(self): # pragma: no cover - terminate already settled it + self.stopped.append("kill") + + def wait(self, timeout=None): + return self.returncode + + child = Interrupted() + with self.assertRaises(KeyboardInterrupt): + service.start(allow_isolated=True, launcher=lambda *a, **k: child, timeout=5.0) + + self.assertIn( + "terminate", child.stopped, + "an unconfirmed child was left running after the start was interrupted", + ) + + def test_a_scope_claim_that_cannot_publish_its_record_releases_the_lock(self): + """A lock with no record is a claim on behalf of a registration that does not exist, + and it goes on refusing every later start in this process until the object dies.""" + registry = ScopeRegistry(Path(self.scopes), ISOLATED) + registry.prepare() + with mock.patch.object(service_module.Path, "write_text", + side_effect=OSError("no space left on device")): + with self.assertRaises(OSError): + registry.claim(SOCKET, {"storeId": "a-store"}) + + self.assertIsNone(registry._handle, "the lock outlived the claim that failed") + self.assertTrue( + registry.claim(SOCKET, {"storeId": "a-store"})["ok"], + "a later claim in the same process was refused by the abandoned lock", + ) + def test_disable_refuses_a_lock_holder_nothing_here_can_identify(self): """A supervisor that has taken the lock and not yet published its record. @@ -1470,7 +1583,40 @@ def test_a_partial_supervised_invocation_is_refused(self): self.adopt(service, scope_fd=None) self.assertEqual(caught.exception.payload["reason"], "supervised_invocation_incomplete") + def test_a_record_naming_another_store_is_refused(self): + """The supervisor recorded which store it registered the scope for. + + This worker opened whatever relay.sqlite3 the path resolves to now, and a database + deleted or atomically replaced between worker segments is a different one. Without + this the worker serves an empty or unrelated store while the supervisor and the scope + registration still name the original - and every participant comparing identities is + told they agree. + """ + from codex_session_relay.cli import PayloadExit + + service = self.prepared(storeId="the-supervisors-store") + service.store_id = "a-replacement-store" + with self.assertRaises(PayloadExit) as caught: + self.adopt(service) + self.assertEqual(caught.exception.payload["reason"], "supervised_store_mismatch") + + def test_a_record_naming_the_same_store_is_not_stopped_by_the_store_check(self): + """The refusal must not swallow the ordinary supervised run. + + This gets as far as the descriptor check, which is where a test with no genuinely + inherited descriptor has to stop. What it establishes is that the store check was not + what stopped it. + """ + from codex_session_relay.cli import PayloadExit + + service = self.prepared(storeId="one-store") + service.store_id = "one-store" + with self.assertRaises(PayloadExit) as caught: + self.adopt(service) + self.assertEqual(caught.exception.payload["reason"], "supervised_fd_mismatch") + def test_a_wrong_token_is_refused(self): + from codex_session_relay.cli import PayloadExit service = self.prepared() From 2f36f0246b078ba7f75d2e58c63d3666a91a4316 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 04:09:28 +0900 Subject: [PATCH 25/29] Three more from round six, two of them about who was holding what A store older than socket provenance records nothing about which socket it serves, and a directory hash cannot be inverted. If the first post-upgrade command uses a spelling we cannot reconstruct, the legacy-hash comparison misses it, discovery finds no claim, and a fresh canonical database is created beside it - which then wins every later resolution and hides its assignments for good. That is the same permanent loss two contested stores cause, arriving by a different route, and doctor was the only place it showed. Selections about to CREATE a store now carry the unidentified siblings, and the command line refuses with unidentified_state_directory. An existing canonical store returns before the check, because it has already answered the question. The way out is the same --state that resolves a contested socket, and it is needed once. supervise checked the recorded intent before it held anything. A disable landing between that check and the instance lock would start a supervisor its owner had already turned off - and one the disabling caller never examined, because what it classified was the previous holder. The intent is re-read under the lock now, so whoever holds it is the one whose intent decides. Last, the segment bound was tested only at the top of the loop, so a finite run slept one restart delay it had no use for before returning. After repeated failures that delay is the backoff cap, which turned a two-segment run into a five-minute wait for a replacement that was never going to be spawned. The bound is checked after the exit code is recorded, along with the deadline. test_a_clean_segment_resets_the_failure_count grew a fourth segment: the reset it asserts was visible only in the delay after the last worker, which is the delay that no longer exists. --- .../src/codex_session_relay/cli.py | 20 ++++++-- .../src/codex_session_relay/service.py | 18 +++++++ .../src/codex_session_relay/store.py | 19 +++++++ .../codex-session-relay/tests/test_cli.py | 20 ++++++++ .../codex-session-relay/tests/test_service.py | 50 ++++++++++++++++++- .../codex-session-relay/tests/test_store.py | 36 +++++++++++++ 6 files changed, 157 insertions(+), 6 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 02d3520..30fd40f 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -1385,6 +1385,11 @@ def _refuse_ambiguous_state(services, args) -> None: wins every later resolution and hides the assignments and pending deliveries in both of the others. Refusing costs one command; the third store costs the state. + The same refusal covers a store that records NO socket. Its directory hash cannot be + inverted, so if it is this socket's - created from a spelling we cannot reconstruct - then + creating a canonical database beside it hides it just as permanently. That case fires only + when a store would be created; an existing canonical store has already settled it. + doctor and ack-proof are exempt for opposite reasons. doctor is how an operator finds out which store to pass to --state, so refusing it would remove the only way out. ack-proof is a derivation over its own two arguments that opens no store at all. @@ -1397,24 +1402,31 @@ def _refuse_ambiguous_state(services, args) -> None: builds Services itself bypasses it; every in-process caller in this package passes an explicit directory, and raising from a property would turn a diagnostic into a crash. """ - if not services.selection.ambiguous: + selection = services.selection + if not (selection.ambiguous or selection.unidentified): return if getattr(args, "handler", None) in (cmd_doctor, cmd_ack_proof): return + contested = bool(selection.ambiguous) raise PayloadExit({ "error": "refused", - "reason": "ambiguous_state_directory", + "reason": ("ambiguous_state_directory" if contested + else "unidentified_state_directory"), "detail": ( "more than one store already records this socket, and creating a new one here" " would hide them both" + ) if contested else ( + "a store here records no socket, so it cannot be ruled out as this one's;" + " creating a new store beside it would hide it permanently" ), "socketPath": services.socket_path, - "candidates": list(services.selection.ambiguous), - "wouldHaveCreated": str(services.selection.db_path), + "candidates": list(selection.ambiguous or selection.unidentified), + "wouldHaveCreated": str(selection.db_path), "recover": [ "doctor lists the candidates", "--state doctor identifies the store", "--state service status shows which assignments it carries", + "--state once, to adopt it deliberately", ], }, EXIT_REFUSED) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 2ded8c3..ac0bff0 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -1231,6 +1231,16 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= segments, failures, degraded = [], 0, None outstanding = None with SingleInstance(self.selection.path, shared=True) as lock: + # Re-read UNDER the lock. The check above ran while nothing was held, so a disable + # landing between them is missed: this supervisor starts, and the disabling caller + # - which classified the PREVIOUS holder and wrote outside the lock - has written + # enabled=false at a supervisor it never examined, which this one then obeys at its + # first worker boundary. Whoever holds this lock is the one whose intent decides. + if not self.intent.read()["enabled"]: + raise ServiceRefused( + "service_disabled", + "this service was disabled while this supervisor was starting", + ) scope_fd = None if self.socket_path: claim = self.scope.claim( @@ -1299,6 +1309,14 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= self._note(consecutiveFailures=failures, degraded=degraded) if self.stop_requested() or not self.intent.read()["enabled"]: break + if max_segments is not None and len(segments) >= max_segments: + # The top of the loop stops for this too, but only AFTER the restart + # delay below - so a finite run outran its own bound by up to the + # backoff cap, which after repeated failures is five minutes, before + # returning. There is nothing left to wait for. + break + if deadline is not None and time.monotonic() - started >= deadline: + break wait = policy.restart_delay_for(failures) if deadline is not None: # Clamped the same way a worker segment is. An unclamped delay - up to diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index 4542911..16d95c7 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -527,6 +527,10 @@ class StateSelection: # selection because the decision not to create a store here has to reach whoever holds # the path they would otherwise have created one at. ambiguous: tuple = () + # Stores that record NO socket at all, when this selection is about to create a new one + # beside them. They predate provenance, their directory hash cannot be inverted, and one + # of them may be this socket's - so creating here may be hiding real assignments. + unidentified: tuple = () @property def db_path(self) -> Path: @@ -537,6 +541,7 @@ def to_record(self) -> dict: "path": str(self.path), "dbPath": str(self.db_path), "source": self.source, "detail": self.detail, "socketScope": self.socket_scope, "precedence": list(PRECEDENCE), "ambiguous": list(self.ambiguous), + "unidentified": list(self.unidentified), } @@ -611,6 +616,20 @@ def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: f"{detail}; {len(claims)} stores already record this socket", scope, tuple(claims), ) + # Nothing claims this socket, so a store is about to be created here. A store that + # predates provenance records no socket at all and its directory hash cannot be inverted, + # so if one of them IS this socket's - created from a spelling we cannot reconstruct - + # creating a canonical database now hides it permanently, exactly the way a third store + # would. Reporting them through doctor alone was not enough, because ordinary commands do + # not run doctor. Only when we would CREATE: an existing canonical store has already + # answered the question and returned above. + unidentified = stores_without_provenance(base / "codex-session-relay", skip=scope) + if unidentified: + return StateSelection( + chosen, source, + f"{detail}; {len(unidentified)} stores here record no socket", + scope, (), tuple(unidentified), + ) return StateSelection(chosen, source, detail, scope) diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index 3c64ae2..7e140d3 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -496,6 +496,26 @@ def test_doctor_still_describes_a_contested_socket(self): self.assertTrue(report["siblingStores"]["ambiguous"]) self.assertEqual(len(report["siblingStores"]["claimingThisSocket"]), 2) + def test_a_store_recording_no_socket_also_refuses_before_creating_one(self): + """A store older than provenance cannot be matched to a socket by anything but its + directory hash, which cannot be inverted. Reporting it through doctor was not enough, + because an ordinary command does not run doctor and creates the store anyway.""" + from pathlib import Path + + from codex_session_relay.store import Store + + home = os.path.join(self.tmp, "unlabelled-home") + root = os.path.join(home, ".local", "state", "codex-session-relay") + os.makedirs(os.path.join(root, "0123456789abcdef")) + Store(Path(root) / "0123456789abcdef" / "relay.sqlite3").close() + socket = os.path.join(self.tmp, "unlabelled.sock") + + refused = self.run_in_home(home, "--socket", socket, "status", expect=2) + + self.assertEqual(refused["reason"], "unidentified_state_directory") + self.assertFalse(os.path.exists(refused["wouldHaveCreated"])) + self.assertEqual(os.listdir(root), ["0123456789abcdef"], "no store was created") + def test_an_explicit_state_directory_resolves_the_contest(self): home, root, socket = self.contested("contested-explicit") chosen = os.path.join(root, "aaaa444444444444") diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 0fd4877..1cd45ab 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -1388,6 +1388,38 @@ def test_every_worker_inherits_the_descriptors_the_supervisor_holds(self): self.assertIsNotNone(launches[0]["lock_fd"]) self.assertIsNotNone(launches[0]["scope_fd"]) + def test_supervision_refuses_a_disable_that_landed_while_it_was_starting(self): + """The first intent check runs while nothing is held. + + A disable that lands between it and the instance lock would otherwise start a + supervisor its owner had already turned off - and one the disabling caller never + examined, because what it classified was the PREVIOUS holder. Whoever holds the lock + is the one whose intent decides, so the answer is re-read under it. + """ + service = self.service("a") + service.enable(actor="test") + enabled = service.intent.read() + reads = [] + + def once_then_disabled(): + reads.append(1) + return enabled if len(reads) == 1 else dict(enabled, enabled=False) + + service.intent.read = once_then_disabled + launches = [] + + def spawn(**call): # pragma: no cover - reaching it is the failure + launches.append(call) + return FakeWorker(0) + + with self.assertRaises(ServiceRefused) as caught: + service.supervise( + allow_isolated=True, spawn=spawn, sleeper=lambda _s: None, max_segments=1, + ) + + self.assertEqual(caught.exception.reason, "service_disabled") + self.assertEqual(launches, [], "a disabled service was supervised anyway") + def test_repeated_failure_backs_off_and_is_reported(self): service = self.service("a") service.enable(actor="test") @@ -1403,10 +1435,24 @@ def test_repeated_failure_backs_off_and_is_reported(self): def test_a_clean_segment_resets_the_failure_count(self): service = self.service("a") service.enable(actor="test") - outcome, _launches, slept = self.supervised(service, codes=[1, 1, 0]) - self.assertEqual(outcome["consecutiveFailures"], 0) + # A fourth segment so the reset is observable as a delay. The delay after the LAST + # segment no longer exists - there is nothing left to wait for - so a three-code run + # would show the backoff climbing and never show it come back down. + outcome, _launches, slept = self.supervised(service, codes=[1, 1, 0, 1]) + self.assertEqual(outcome["consecutiveFailures"], 1) self.assertEqual(slept[:3], [2.0, 4.0, 2.0]) + def test_the_last_allowed_segment_does_not_wait_to_restart_nothing(self): + """The segment bound was only checked at the top of the loop, so every finite run + slept one restart delay it had no use for - up to the backoff cap after repeated + failures - before returning.""" + service = self.service("a") + service.enable(actor="test") + _outcome, launches, slept = self.supervised(service, codes=[1, 1]) + self.assertEqual(len(launches), 2) + self.assertEqual(slept, [2.0], "the run waited after the segment it was never going" + " to replace") + def test_disabling_the_service_ends_supervision_at_the_boundary(self): service = self.service("a") service.enable(actor="test") diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 1c456ba..66c5550 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -324,6 +324,42 @@ def test_a_store_with_no_recorded_socket_is_never_adopted_on_a_guess(self): # But it IS visible, so an operator is not left guessing why a store looks empty. self.assertIn(stranger, stores_without_provenance(root, skip=chosen.path.name)) + def test_a_store_recording_no_socket_blocks_creating_one_beside_it(self): + """Its directory hash cannot be inverted, so it cannot be ruled out as this socket's. + + Creating a canonical database next to it hides it exactly the way a third store hides + two contested ones, and reporting it through doctor alone did not help: ordinary + commands do not run doctor. + """ + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + stranger = os.path.join(root, "fedcba9876543210") + os.makedirs(stranger) + Store(Path(stranger) / "relay.sqlite3").close() + + chosen = resolve_state_dir(None, "/run/first-after-upgrade.sock") + + self.assertEqual(list(chosen.unidentified), [stranger]) + self.assertEqual(chosen.ambiguous, ()) + self.assertEqual(list(chosen.to_record()["unidentified"]), [stranger]) + + def test_an_existing_canonical_store_settles_the_question_already(self): + """The refusal is about CREATING one. A store that is already here has answered.""" + from codex_session_relay.store import socket_scope + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + stranger = os.path.join(root, "fedcba9876543211") + os.makedirs(stranger) + Store(Path(stranger) / "relay.sqlite3").close() + socket = "/run/already-here.sock" + mine = os.path.join(root, socket_scope(socket)) + os.makedirs(mine) + Store(Path(mine) / "relay.sqlite3", socket_path=socket).close() + + chosen = resolve_state_dir(None, socket) + + self.assertEqual(str(chosen.path), mine) + self.assertEqual(chosen.unidentified, ()) + def test_a_fresh_socket_uses_the_canonical_directory(self): """The fallback is for an existing store only; nothing new lands in the old name.""" from codex_session_relay.store import socket_scope From b64ceff0b1d76a0e45934638239ffdca4de8ae49 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 04:29:26 +0900 Subject: [PATCH 26/29] Round seven: a stop that could erase a live launch, and two smaller things stop already refused to act on a replacement it detected, but the detection was a snapshot. The daemon lock goes free the moment the old supervisor and worker are gone, so a new launch can acquire it after the re-read and publish its own record - and the clearing write then erased the identity of a service that is running, leaving every later status and stop with no handle on it. The write happens under the lock now: holding it means no replacement can start while the stop finalises, and failing to take it means one already did, which is reported rather than overwritten. A lock taken by something that has published nothing is refused for the same reason disable refuses it. A stopped supervisor kept nextRestartAt in its record, so a service that had ended read as though a restart were still scheduled. Cleared with the pid. Last, a store records the socket it serves and the first recording wins, which is right - nothing should silently rewrite it. But an explicit --state or CODEX_SESSION_RELAY_STATE reused with a different App Server is a real disagreement rather than a re-opening: the service claims and serves the new socket while the database goes on attributing itself to the old one, so one installation's assignments can be exposed through another and later discovery still matches the store to a socket it no longer serves. The command line refuses that, with both sockets named. doctor and ack-proof stay exempt, as they are for the other state refusals, because describing the disagreement is how an operator resolves it. --- .../src/codex_session_relay/cli.py | 27 +++++++++- .../src/codex_session_relay/service.py | 31 +++++++++++- .../codex-session-relay/tests/test_cli.py | 29 +++++++++++ .../codex-session-relay/tests/test_service.py | 49 +++++++++++++++++++ 4 files changed, 133 insertions(+), 3 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 30fd40f..00352cd 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -29,7 +29,8 @@ from .registry import Registry, contract_record as relationship_record, record_settings from .settings import REQUIRED as REQUIRED_SETTINGS from .store import ( - Store, compare_store, nonce_lookup, probe, resolve_state_dir, state_dir, + Store, canonical_socket, compare_store, nonce_lookup, probe, resolve_state_dir, + state_dir, store_socket, ) from .sync import SyncOutbox, render_progress_summary @@ -1403,6 +1404,30 @@ def _refuse_ambiguous_state(services, args) -> None: explicit directory, and raising from a property would turn a diagnostic into a crash. """ selection = services.selection + # A store records the socket it serves, and the first recording wins so nothing rewrites + # it silently. But an explicit --state or CODEX_SESSION_RELAY_STATE reused with a + # DIFFERENT App Server is a real disagreement: the service would claim and serve the new + # socket while the database goes on attributing itself to the old one, so assignments from + # one App Server can be exposed through another and later discovery still matches the + # store to the socket it no longer serves. Explicit selections reach this even though they + # carry no discovery, because choosing a directory is not choosing what is already in it. + if services.socket_path and selection.db_path.exists(): + recorded = store_socket(selection.db_path) + wanted = canonical_socket(services.socket_path) + if recorded is not None and recorded != wanted and ( + getattr(args, "handler", None) not in (cmd_doctor, cmd_ack_proof) + ): + raise PayloadExit({ + "error": "refused", + "reason": "state_directory_serves_another_socket", + "detail": ( + "this store records a different App Server socket; serving the requested" + " one from it would expose one installation's assignments through another" + ), + "recordedSocket": recorded, + "requestedSocket": wanted, + "stateDirectory": str(selection.path), + }, EXIT_REFUSED) if not (selection.ambiguous or selection.unidentified): return if getattr(args, "handler", None) in (cmd_doctor, cmd_ack_proof): diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index ac0bff0..492efc8 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -885,7 +885,32 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) cleared["workerStartTicks"] = None if supervisor_done and worker_done: cleared["stoppedAt"] = _now() - self.write_record(cleared) + # Written UNDER the lock, or not at all. The re-read above is a snapshot, and the + # daemon lock is free the moment the old supervisor and worker are gone - so a + # replacement can acquire it after that read and publish its own record, and this + # write would erase the identity of a launch that is running, leaving every later + # status and stop with no handle on it. Holding the lock means no replacement can + # start while we finalise; failing to take it means one already did. + with self.daemon_lock_if_free() as held: + latest = self.record() + replaced = ( + latest is not None + and self._launch_identity(latest) != self._launch_identity(record) + ) + if replaced: + return {"ok": False, "reason": "replaced_by_new_launch", + "detail": "a new launch published its record during this stop; it" + " was left untouched and is still running", + "supervisor": outcome, "worker": worker} + if held is None and latest is None: + # The lock is held by something that has published nothing. It is not + # ours to describe, and writing a stopped record for it would say this + # state directory is idle while that process starts serving. + return {"ok": False, "reason": "ownership_unverifiable", + "detail": "the daemon lock was taken during this stop by a" + " process that has published no record", + "supervisor": outcome, "worker": worker} + self.write_record(cleared) if owner == NONE and worker == "gone": return {"ok": False, "reason": "not_running", "detail": detail, "supervisor": "gone", "worker": "gone"} @@ -1333,7 +1358,9 @@ def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments= # state directory claim this socket beside the orphan. self.scope.release(self.socket_path, shared=True) current = self.record() or {} - cleared = dict(current, pid=None, stoppedAt=_now()) + # nextRestartAt with it: it names a restart this supervisor is no longer going + # to make, and leaving it behind let a stopped service report a pending one. + cleared = dict(current, pid=None, stoppedAt=_now(), nextRestartAt=None) if outstanding is None: cleared["workerPid"] = None cleared["workerStartTicks"] = None diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index 7e140d3..b848267 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -496,6 +496,35 @@ def test_doctor_still_describes_a_contested_socket(self): self.assertTrue(report["siblingStores"]["ambiguous"]) self.assertEqual(len(report["siblingStores"]["claimingThisSocket"]), 2) + def test_a_state_directory_recording_another_socket_is_refused(self): + """An explicit directory reused with a different App Server. + + Choosing a directory is not choosing what is already in it: the service would claim + and serve the new socket while the database went on attributing itself to the old + one, so one installation's assignments could be exposed through another and later + discovery would still match the store to the socket it no longer serves. + """ + from pathlib import Path + + from codex_session_relay.store import Store + + state = os.path.join(self.tmp, "reused-state") + first = os.path.join(self.tmp, "first.sock") + second = os.path.join(self.tmp, "second.sock") + Store(Path(state) / "relay.sqlite3", socket_path=first).close() + + environment = dict(os.environ, PYTHONPATH=os.path.join(REPO, "src")) + completed = subprocess.run( + [sys.executable, "-m", "codex_session_relay.cli", "--state", state, + "--socket", second, "status"], + capture_output=True, text=True, env=environment, timeout=60, + ) + + self.assertEqual(completed.returncode, 2, completed.stdout + completed.stderr) + refused = json.loads(completed.stdout) + self.assertEqual(refused["reason"], "state_directory_serves_another_socket") + self.assertEqual(refused["recordedSocket"], first) + def test_a_store_recording_no_socket_also_refuses_before_creating_one(self): """A store older than provenance cannot be matched to a socket by anything but its directory hash, which cannot be inverted. Reporting it through doctor was not enough, diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 1cd45ab..49b57f2 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -612,6 +612,47 @@ def test_a_scope_claim_that_cannot_publish_its_record_releases_the_lock(self): "a later claim in the same process was refused by the abandoned lock", ) + def test_stop_finalises_the_record_under_the_lock(self): + """The re-read that detects a replacement is a snapshot. + + The daemon lock is free the moment the old supervisor and worker are gone, so a + replacement can acquire it after that read and publish its own record - and the + clearing write would erase the identity of a launch that is running, leaving every + later status and stop with no handle on it. + """ + service = self.service("i") + service.enable(actor="owner") + child, _pid = self.holder(service) + order = [] + original_lock = service.daemon_lock_if_free + original_write = service.write_record + + @contextlib.contextmanager + def watched_lock(): + order.append("lock") + with original_lock() as held: + yield held + + def watched_write(payload): + order.append("write") + return original_write(payload) + + service.daemon_lock_if_free = watched_lock + service.write_record = watched_write + try: + stopped = service.stop(actor="owner") + finally: + service.daemon_lock_if_free = original_lock + service.write_record = original_write + + child.wait(timeout=10) + self.assertTrue(stopped["ok"], stopped) + self.assertEqual( + order[-2:], ["lock", "write"], + "the stopped record was published without holding the lock that keeps a" + " replacement out", + ) + def test_disable_refuses_a_lock_holder_nothing_here_can_identify(self): """A supervisor that has taken the lock and not yet published its record. @@ -1420,6 +1461,14 @@ def spawn(**call): # pragma: no cover - reaching it is the failure self.assertEqual(caught.exception.reason, "service_disabled") self.assertEqual(launches, [], "a disabled service was supervised anyway") + def test_a_stopped_supervisor_reports_no_pending_restart(self): + """nextRestartAt named a restart this supervisor is no longer going to make, so a + stopped service read as though one were still scheduled.""" + service = self.service("a") + service.enable(actor="test") + self.supervised(service, codes=[1, 1]) + self.assertIsNone(service.record()["nextRestartAt"]) + def test_repeated_failure_backs_off_and_is_reported(self): service = self.service("a") service.enable(actor="test") From 2c516bcd4a589914c93985d225a34d3a10c7bbc6 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 04:53:45 +0900 Subject: [PATCH 27/29] Two windows my last round left open The boot rule was applied to one stale record shape and not the other. When a supervisor's pid is cleared but a worker number remains, ownership already refused a record with no boot id on a host that has one - because _stop_worker validates start ticks only, and a reboot resets those along with the pid space. The already-gone path is exactly as stale: the supervisor's pid is still in the record and the process behind it is gone. It answered none, stop fell through to the worker, and an unrelated process holding the old number after a reboot would have been signalled. Same rule, same reason, now on both paths. The other is the guard I added last round for a replacement publishing during a stop. It compared the re-read record's launch identity, which cannot see the case that actually happens: a replacement that has taken the lock and not yet published leaves the OLD record in place, so the comparison finds no difference and the write went ahead anyway. By that point both processes this stop was acting on are confirmed gone, so nothing it knows about can be holding the lock - which makes the failed acquisition itself the evidence, and the only evidence there is in that window. A lock that cannot be taken after both exits is read as a replacement. That subsumes the published-nothing case the previous commit handled separately, so its narrower check is gone rather than left to disagree with this one. --- .../src/codex_session_relay/service.py | 26 +++++++---- .../codex-session-relay/tests/test_service.py | 44 +++++++++++++++++++ 2 files changed, 62 insertions(+), 8 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 492efc8..3157384 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -487,6 +487,17 @@ def ownership(self, record=None): if handle.already_gone: if foreign: return FOREIGN, handle, "; ".join(foreign) + if (record.get("workerPid") and record.get("bootId") is None + and boot_id() is not None): + # The same rule the cleared-pid path above already applies, and this path is + # just as stale: the supervisor is gone but its worker number is still in the + # record, so stop() falls through to _stop_worker - which validates start + # ticks only. A reboot resets those along with the pid space, so an unrelated + # process holding the old worker number would be signalled. + return UNVERIFIABLE, handle, ( + "no boot id is recorded, so this worker pid cannot be distinguished from" + " one reused after a reboot" + ) return NONE, handle, "the recorded process is gone" if not handle.usable: if foreign: @@ -897,19 +908,18 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) latest is not None and self._launch_identity(latest) != self._launch_identity(record) ) + if held is None and supervisor_done and worker_done: + # Both of the processes this record named are confirmed gone, so whatever + # holds the lock now is not one of them. A replacement usually has not + # published yet, which means latest is still the OLD record and the + # identity comparison cannot see it - so the failed acquisition is itself + # the evidence, and it is the only evidence available in that window. + replaced = True if replaced: return {"ok": False, "reason": "replaced_by_new_launch", "detail": "a new launch published its record during this stop; it" " was left untouched and is still running", "supervisor": outcome, "worker": worker} - if held is None and latest is None: - # The lock is held by something that has published nothing. It is not - # ours to describe, and writing a stopped record for it would say this - # state directory is idle while that process starts serving. - return {"ok": False, "reason": "ownership_unverifiable", - "detail": "the daemon lock was taken during this stop by a" - " process that has published no record", - "supervisor": outcome, "worker": worker} self.write_record(cleared) if owner == NONE and worker == "gone": return {"ok": False, "reason": "not_running", "detail": detail, diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index 49b57f2..dd96058 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -653,6 +653,50 @@ def watched_write(payload): " replacement out", ) + def test_a_gone_supervisor_with_no_boot_id_leaves_its_worker_unverifiable(self): + """stop() falls through to _stop_worker when ownership answers none, and that + validates start ticks only. A reboot resets those along with the pid space, so an + unrelated process holding the old worker number would be signalled. The cleared-pid + path already refused this; the already-gone path is just as stale and did not. + """ + if service_module.boot_id() is None: + self.skipTest("this host records no boot id, so the rule cannot apply") + service = self.service("j") + finished = subprocess.Popen([sys.executable, "-c", "pass"]) + finished.wait(timeout=10) + service.write_record(dict( + service.new_record(pid=os.getpid()), pid=finished.pid, workerPid=os.getpid(), + workerStartTicks=service_module.start_ticks(os.getpid()), bootId=None, + )) + + owner, handle, detail = service.ownership() + if handle is not None: + handle.close() + + self.assertEqual(owner, service_module.UNVERIFIABLE) + self.assertIn("boot id", detail) + + def test_stop_reads_a_lock_it_cannot_take_after_both_exits_as_a_replacement(self): + """A replacement usually has not published yet, so latest still reads the OLD record + and the identity comparison cannot see it. Both recorded processes are confirmed gone + by this point, so nothing this stop was acting on can be holding the lock, and the + failed acquisition is the only evidence available in that window. + """ + service = self.service("k") + service.enable(actor="owner") + child, _pid = self.holder(service) + + @contextlib.contextmanager + def never_free(): + yield None + + service.daemon_lock_if_free = never_free + stopped = service.stop(actor="owner") + + child.wait(timeout=10) + self.assertFalse(stopped["ok"], stopped) + self.assertEqual(stopped["reason"], "replaced_by_new_launch") + def test_disable_refuses_a_lock_holder_nothing_here_can_identify(self): """A supervisor that has taken the lock and not yet published its record. From d243e636e57698fc1f9cecaf085d0b6d113c0402 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 05:19:56 +0900 Subject: [PATCH 28/29] One rule for the stop record, after three narrower ones each left a window The finalizing write has now been guarded three ways: compare the re-read launch identity, then also treat a failed acquisition after both exits as a replacement, then also handle an absent record. Each asked what the world looked like at one instant, and a replacement can publish immediately after any instant - so each left a different window, and the third left the one where termination is not confirmed and no record is present. Holding the lock is the only thing that excludes a replacement for the duration of the write rather than at a moment, so that is the condition now: the record is written only while this process holds the daemon lock. A lock it cannot take after both processes are confirmed gone is a replacement. A lock it cannot take while the worker is unconfirmed is most likely that worker, through the descriptor it inherited - and nothing is written, which loses nothing, because the record already names the processes a later stop has to reach. That is exactly what the write would have preserved. daemon_lock_if_free was also mapping every OSError to "someone holds it". flock reports contention with EACCES or EAGAIN; an unreadable directory or an exhausted descriptor table is an operational failure and not a statement about who owns this state directory. Callers act on that answer - stop announces a replacement, disable refuses an owner - so the other errors are raised and reported as themselves. It is the rule ScopeRegistry.claim already applies to the scope lock. --- .../src/codex_session_relay/service.py | 46 ++++++++++------ .../codex-session-relay/tests/test_service.py | 52 +++++++++++++++++++ 2 files changed, 83 insertions(+), 15 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index 3157384..e141458 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -620,6 +620,12 @@ def daemon_lock_if_free(self): Opened with 'a+' so the probe is atomic even on a state directory that has never run a daemon: a supervisor starting there has to create and lock this same file. + + Only CONTENTION yields None. Mapping every OSError to "someone holds it" made an + unreadable directory or an exhausted descriptor table indistinguishable from a running + supervisor, and callers act on that answer - stop reports a replacement, disable + refuses an owner. An operational failure is not a statement about who owns this + directory, so it is raised and reported as itself. """ path = self.selection.path / DAEMON_LOCK handle = None @@ -627,10 +633,12 @@ def daemon_lock_if_free(self): self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) handle = open(path, "a+") fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) - except OSError: + except OSError as error: if handle is not None: handle.close() handle = None + if error.errno not in (errno.EACCES, errno.EAGAIN): + raise try: yield handle finally: @@ -902,25 +910,33 @@ def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) # write would erase the identity of a launch that is running, leaving every later # status and stop with no handle on it. Holding the lock means no replacement can # start while we finalise; failing to take it means one already did. + # One rule: this record is written only while we HOLD the lock. Three narrower + # versions of this guard each left a window - a replacement that had published, one + # that had not, an absent record - because each asked what the world looked like at + # a moment rather than excluding change for the duration. Holding the lock is the + # only thing that actually excludes a replacement, so that is the condition. with self.daemon_lock_if_free() as held: - latest = self.record() - replaced = ( - latest is not None - and self._launch_identity(latest) != self._launch_identity(record) - ) - if held is None and supervisor_done and worker_done: - # Both of the processes this record named are confirmed gone, so whatever - # holds the lock now is not one of them. A replacement usually has not - # published yet, which means latest is still the OLD record and the - # identity comparison cannot see it - so the failed acquisition is itself - # the evidence, and it is the only evidence available in that window. - replaced = True - if replaced: + if held is None: + if supervisor_done and worker_done: + # Both processes this stop acted on are confirmed gone, so whatever + # holds the lock is neither of them. + return {"ok": False, "reason": "replaced_by_new_launch", + "detail": "the daemon lock was taken during this stop; that" + " launch was left untouched and is still running", + "supervisor": outcome, "worker": worker} + # Otherwise the worker we could not confirm is the likeliest holder, + # through the descriptor it inherited. Nothing is written, and nothing + # needs to be: the record still names the processes a later stop must + # reach, which is exactly what this write would have preserved. + elif (latest := self.record()) is not None and ( + self._launch_identity(latest) != self._launch_identity(record) + ): return {"ok": False, "reason": "replaced_by_new_launch", "detail": "a new launch published its record during this stop; it" " was left untouched and is still running", "supervisor": outcome, "worker": worker} - self.write_record(cleared) + else: + self.write_record(cleared) if owner == NONE and worker == "gone": return {"ok": False, "reason": "not_running", "detail": detail, "supervisor": "gone", "worker": "gone"} diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py index dd96058..9106c8f 100644 --- a/packages/codex-session-relay/tests/test_service.py +++ b/packages/codex-session-relay/tests/test_service.py @@ -5,6 +5,7 @@ """ import contextlib +import errno import json import os import shutil @@ -697,6 +698,57 @@ def never_free(): self.assertFalse(stopped["ok"], stopped) self.assertEqual(stopped["reason"], "replaced_by_new_launch") + def test_stop_writes_nothing_when_it_cannot_hold_the_lock(self): + """One rule, because three narrower ones each left a window. + + A replacement that has published, one that has not, and an absent record are all the + same situation: this process does not hold the state directory, so it does not get to + write the record. Holding the lock is the only thing that excludes a replacement for + the duration of the write rather than at one instant. + """ + service = self.service("m") + service.enable(actor="owner") + child, _pid = self.holder(service) + before = service.record() + + @contextlib.contextmanager + def never_free(): + yield None + + writes = [] + original_write = service.write_record + service.daemon_lock_if_free = never_free + service.write_record = lambda payload: (writes.append(payload), + original_write(payload))[1] + try: + stopped = service.stop(actor="owner") + finally: + service.write_record = original_write + + child.wait(timeout=10) + self.assertFalse(stopped["ok"], stopped) + self.assertEqual(stopped["reason"], "replaced_by_new_launch") + self.assertEqual(writes, [], "a stop that holds nothing wrote the shared record") + self.assertEqual(service.record()["launchId"], before["launchId"]) + + def test_a_lock_that_fails_operationally_is_not_reported_as_a_replacement(self): + """flock reports contention with EACCES or EAGAIN. Every other OSError is an + operational failure - an unreadable directory, no descriptors left - and mapping it + to "someone holds it" made stop announce a replacement that does not exist, which a + restart then refuses to work around.""" + service = self.service("n") + service.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + + def refuse_to_open(*_args, **_kwargs): + raise OSError(errno.EMFILE, "too many open files") + + with mock.patch("builtins.open", side_effect=refuse_to_open): + with self.assertRaises(OSError) as caught: + with service.daemon_lock_if_free(): + pass + + self.assertEqual(caught.exception.errno, errno.EMFILE) + def test_disable_refuses_a_lock_holder_nothing_here_can_identify(self): """A supervisor that has taken the lock and not yet published its record. From 5681e26517b721af02549658dbe37d44e0da4ef6 Mon Sep 17 00:00:00 2001 From: thisisjun786 <259586770+thisisjun786@users.noreply.github.com> Date: Thu, 17 Sep 2026 06:25:57 +0900 Subject: [PATCH 29/29] open() reports EACCES too, and that is not contention Last commit stopped daemon_lock_if_free reading every OSError as "someone holds this directory", but left opening the lock file inside the same handler - and open() raises EACCES of its own, for a lock file that has become unreadable. So the mistake survived one level down: an unreadable daemon.lock still answered "held", and after a stop had actually terminated both processes that answer became replaced_by_new_launch, which a restart then refuses to work around. Opening the file sits outside the contention handler now. Only flock's own EACCES and EAGAIN mean contention; every other failure, at either call, is operational and is raised as itself. --- .../src/codex_session_relay/service.py | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py index e141458..a237bd0 100644 --- a/packages/codex-session-relay/src/codex_session_relay/service.py +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -625,18 +625,19 @@ def daemon_lock_if_free(self): unreadable directory or an exhausted descriptor table indistinguishable from a running supervisor, and callers act on that answer - stop reports a replacement, disable refuses an owner. An operational failure is not a statement about who owns this - directory, so it is raised and reported as itself. + directory, so it is raised and reported as itself. Which means opening the file has to + sit OUTSIDE the contention handler: open() raises EACCES too, for a lock file that has + become unreadable, and reading that as contention is the same mistake one level down. """ path = self.selection.path / DAEMON_LOCK - handle = None + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(path, "a+") try: - self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) - handle = open(path, "a+") fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) except OSError as error: - if handle is not None: - handle.close() + handle.close() handle = None + # Only flock's contention errnos. Anything else from flock is operational too. if error.errno not in (errno.EACCES, errno.EAGAIN): raise try: