diff --git a/packages/codex-session-relay/README.md b/packages/codex-session-relay/README.md index 64ad2e9..624ec28 100644 --- a/packages/codex-session-relay/README.md +++ b/packages/codex-session-relay/README.md @@ -56,13 +56,19 @@ activate a service on another host or establish that an earlier test process is Runtime state lives outside any repository, in `$XDG_STATE_HOME/codex-session-relay//` (mode 0700), holding `relay.sqlite3`, -`daemon.lock`, `daemon.pid` and `daemon.log`. `CODEX_SESSION_RELAY_STATE` overrides the directory. +`daemon.lock`, `daemon.pid` and `daemon.log`. Precedence, highest first: `--state`, +`CODEX_SESSION_RELAY_STATE`, `XDG_STATE_HOME`, then `~/.local/state`. **Every process must point at the same state directory.** The child emitting, the parent acknowledging and the daemon delivering share one store; a mismatched `--state` means they simply do not see each other. The endpoint hash is derived from the socket path, so passing the same `--socket` is enough. +`doctor` reports which rule won, the database it resolved to and the access this process really +has. To prove two participants share one store rather than two copies of one, write a nonce with +`store-challenge --write` and check it from the other side with `doctor --expect-nonce`; an +identifier alone is copied along with the file. See [docs/operations.md](docs/operations.md). + ## Authorized execution settings A send carries the settings the recipient task was actually created with. This is not defensive @@ -336,3 +342,5 @@ These are recorded because behaviour depends on them. - `docs/protocol-v1.md` — the wire and record protocol, derived from the frozen contract. - `docs/invariants.md` — every invariant and the code that enforces it. +- `docs/operations.md` — where the state lives, who owns the daemon, and how to read a + stuck delivery. Each section says whether the behaviour is implemented or planned. diff --git a/packages/codex-session-relay/docs/operations.md b/packages/codex-session-relay/docs/operations.md new file mode 100644 index 0000000..d2f92ec --- /dev/null +++ b/packages/codex-session-relay/docs/operations.md @@ -0,0 +1,155 @@ +# Operating the relay + +Who owns the state, who owns the daemon, and what an operator can ask it. The delivery +contract itself is in [protocol-v1.md](protocol-v1.md) and the refusals that enforce it are +in [invariants.md](invariants.md). + +Each section carries a status, because this document is written alongside the work that +makes it true: **implemented** means the behaviour exists and is tested in this package, +**planned** means the contract is agreed and the code lands in a named pull request. +Nothing here describes an installed runtime, a registered service or a live App Server; +source in this package changes none of those. + +## Where the state lives + +Every participant of one assignment must read and write the same SQLite store. The +directory is chosen by the first rule that applies: + +| Precedence | Source | Notes | +|---|---|---| +| 1 | `--state ` | explicit, wins over everything | +| 2 | `CODEX_SESSION_RELAY_STATE` | also read independently by the bridge adapter for its transport ledger | +| 3 | `XDG_STATE_HOME/codex-session-relay/` | the scope is a hash of the App Server socket path | +| 4 | `~/.local/state/codex-session-relay/` | the default | + +The store is `/relay.sqlite3`. `codex-session-relay doctor` reports which rule won, the +value that won, the resolved database path and the measured read/write access, so a +participant never has to infer its own configuration. + +Setting `--state` alone is not enough for an isolated run. The bridge adapter resolves its +transport ledger from `CODEX_SESSION_RELAY_STATE` independently, so a run that overrides only +the flag splits the relay store from the ledger that carries send idempotency. Set both. + +Status: implemented. `resolve_state_dir` returns the winning rule and `doctor` reports it with +the resolved database and the measured access. + +## Proving two participants share one store + +A path string is not proof: symlinks, bind mounts and per-sandbox mounts all make equal +paths unequal and unequal paths equal. A stored identifier alone is not proof either, +because copying the database copies the identifier. + +| Evidence | Verdict | +|---|---| +| a nonce written by one participant is readable by the other | proven | +| equal store id and equal device/inode | proven | +| equal store id, different inode, no nonce | unproven - a copy is possible | +| different store id | mismatch | + +`doctor --expect-store ` exits non-zero on a mismatch. An unproven result is never +reported as healthy. + +Status: implemented. Unproven also exits non-zero, because a caller that asked whether this is +the same store must not read exit 0 as yes. Each participant runs the check in its own sandbox; +one invocation cannot establish another participant's access. + +## The service + +Two process roles. A **worker** is the existing bounded daemon: it runs a set number of ticks +or until a deadline and then exits, and cannot be constructed unbounded (I-64). A +**supervisor** owns the locks and launches successive workers, which is what carries an +assignment past any single process bound. + +| Command | Effect | +|---|---| +| `service status` | intent, liveness, ownership, store, conflicts, projects, observation health | +| `service enable` / `disable` | records the owner's intent; disable also stops a running service | +| `service start` | refuses if already running, if another store owns the operating scope, or if intent is disabled | +| `service stop` | refuses unless this installation owns the process; confirms both supervisor and worker exited | +| `service restart` | stop then start, preserving the recorded intent | +| `service run` | the foreground supervisor; what `start` launches | + +An update tool should read `service status`, act on `stop` or `restart`, and rely on one +guarantee: **a service that was disabled is never enabled as a side effect**. Intent lives in +`service.json` and absence means never configured, which is not enabled. + +Status: implemented. `service run` is the supervisor: it holds the locks once and replaces +bounded workers, so an assignment continues on the same store and generation past any single +process lifetime. A clean segment waits the restart interval; repeated failure backs off +exponentially to a cap and is reported as `degraded` rather than retried silently. + +### Ownership + +Single ownership is enforced at two levels, because either alone has a hole. + +- `/daemon.lock` prevents two daemons per state directory. +- A per-user scope registry, keyed by the App Server socket and located independently of the + state directory, prevents two daemons **with different state directories** from serving the + same App Server. A file lock in one state directory cannot see a rival that chose another. + +Both locks are held by the supervisor and inherited by its worker, so ownership persists +until every process of the service has exited. A dead supervisor with a live worker does not +release the scope. + +Termination is bound to a process handle rather than a pid, and identity includes the boot +id, the installation and the store, so a reused pid is never signalled by mistake. Where a +stable handle is unavailable, ownership reports `unverifiable` and stop refuses. + +Status: implemented. The supervisor acquires both locks, marks them inheritable and passes +them to each worker, which adopts them rather than taking a second lock. A worker is +authenticated by a token recorded in `daemon.json` plus a device/inode check on each +descriptor, arms `PR_SET_PDEATHSIG` in its own bootstrap and immediately re-checks its +parent, and refuses to serve if the supervisor has already gone. + +## One service, several projects + +A single supervisor and a single store serve every assignment on that host, user and App +Server, across projects and repositories. Each delivery is checked against its own +assignment before any transport call: the recipient must be that assignment's own parent for +a completion, or its own child for a revision. Membership in the authorized recipient list +alone is not sufficient, because two assignments may legitimately list the same recipient. +`service status` groups by project so one service carrying several projects is visible. + +Cancelling, pausing or archiving one assignment removes it from the loop. It does not stop +the shared service: only the supervisor's own bound, an explicit stop, or disabled intent +does that. + +Status: implemented. The direction check runs at both `enqueue` and `attempt`, before any +transport call. + +## Reading a stuck delivery + +`status` distinguishes the stages that all previously read as one withheld state: + +| Phase | Meaning | +|---|---| +| `awaiting_receipt` | the child has not produced a completion receipt yet | +| `parent_busy` | the parent is mid-turn; it is never interrupted | +| `settings_rejected` | the host would not confirm the authorized execution settings | +| `turn_accepted` | the transport started a turn | +| `awaiting_ack` | delivered, acknowledgement outstanding | +| `channel_closed` | the push channel itself is unavailable; stored, not woken | +| `superseded` | a newer generation or revision replaced this one | + +Each carries the most recent failed operation, its concrete error, the exact settings +difference where there is one, and the next retry time. + +Health is separate from liveness. A running process with a growing observation backlog is +reported as stalled: staged event age, when each current anchor was last successfully +polled, and the backlog per assignment are all exposed, and a live pid is never counted as +working. + +Status: planned in PR-B. + +## What a restart preserves + +Assignments, generations and anchors, queued and deferred deliveries, attempt history and +frozen message bytes, acknowledgements, verdicts and the Linear outbox all live in the store +and survive any process boundary. On start the relay reconciles unresolved attempts before +doing anything else, and reconciliation establishes what happened without sending: an +attempt whose response was lost stays uncertain until evidence resolves it, and is never +resent on the strength of elapsed time. + +Status: implemented. The supervisor runs recovery before its first worker, and an expired +lease returns its delivery to `held_uncertain` for the reconciler to judge, never to the +send queue, because a queued row would be eligible to send again on no evidence. diff --git a/packages/codex-session-relay/src/codex_session_relay/cli.py b/packages/codex-session-relay/src/codex_session_relay/cli.py index 6fbc5ad..00352cd 100644 --- a/packages/codex-session-relay/src/codex_session_relay/cli.py +++ b/packages/codex-session-relay/src/codex_session_relay/cli.py @@ -28,19 +28,26 @@ from .reconcile import Reconciler from .registry import Registry, contract_record as relationship_record, record_settings from .settings import REQUIRED as REQUIRED_SETTINGS -from .store import Store, state_dir +from .store import ( + Store, canonical_socket, compare_store, nonce_lookup, probe, resolve_state_dir, + state_dir, store_socket, +) from .sync import SyncOutbox, render_progress_summary EXIT_OK, EXIT_REFUSED, EXIT_HOST, EXIT_USAGE = 0, 2, 3, 4 # Which commands need to reach the App Server, and which do not. Reported by doctor, because a # caller should learn this from one command instead of from a failure halfway through. -HOST_REQUIRED_COMMANDS = ("daemon", "deliver", "reconcile", "recover", "verify-acks") +HOST_REQUIRED_COMMANDS = ( + "daemon", "deliver", "reconcile", "recover", "service run", "service start", + "service restart", "verify-acks", +) OFFLINE_COMMANDS = ( "ack", "ack-proof", "admit-turn", "assignment-show", "claim", "criteria-register", "criteria-show", "doctor", "emit", "generation-bind", "generation-open", "register", "relationship-resume", "relationship-status", "revision-head", "settings-record", - "settings-show", "show", "status", "verdict", + "settings-show", "show", "status", "store-challenge", "store-identity", "verdict", + "service status", "service enable", "service disable", "service stop", ) @@ -55,30 +62,108 @@ def __getattr__(self, name): class Services: + """Everything a command might need, built only when the command actually needs it. + + Nothing is constructed here on purpose. doctor exists to describe a host where the store + cannot be opened, and a Store built during construction opens the file O_RDWR, switches on + WAL and runs the schema script - so it would raise before doctor could report why. Each + dependency is cached after first use, so laziness never means two stores in one process. + """ + def __init__(self, args): - directory = Path(args.state) if args.state else state_dir(args.socket) - self.store = Store(directory / "relay.sqlite3") + self.selection = resolve_state_dir(getattr(args, "state", None), args.socket) self.clock = SystemClock() self.socket_path = args.socket self.adapter_requested = bool(args.socket) + self._store = None self._adapter = None - self.criteria = CriteriaService(self.store, self.clock) - self.registry = Registry(self.store, self.clock) - self.intake = ReceiptIntake( - self.store, self.registry, self.clock, - admission=AnchorOrExplicit(_LazyAdapter(self) if self.adapter_requested else None), - ) - self.delivery = DeliveryService(self.store, self.registry, self.intake, self.clock) - self.ack = AckService( - self.store, self.registry, self.intake, self.delivery, self.clock, - criteria=self.criteria, - ) - self.reconciler = Reconciler(self.store, self.registry, self.delivery, self.clock) - self.sync = SyncOutbox(self.store, self.clock) - self.ack.sync = self.sync - self.assignments = AssignmentView( - self.store, self.registry, self.clock, criteria=self.criteria - ) + self._criteria = None + self._registry = None + self._intake = None + self._delivery = None + self._ack = None + self._reconciler = None + self._sync = None + self._assignments = None + + @property + def state_directory(self): + return self.selection.path + + @property + def store(self): + if self._store is None: + # The socket travels with the store so a later invocation can find it by socket + # rather than by the hash of the spelling that happened to create it. + self._store = Store(self.selection.db_path, socket_path=self.socket_path) + return self._store + + @property + def criteria(self): + if self._criteria is None: + self._criteria = CriteriaService(self.store, self.clock) + return self._criteria + + @property + def registry(self): + if self._registry is None: + self._registry = Registry(self.store, self.clock) + return self._registry + + @property + def intake(self): + if self._intake is None: + self._intake = ReceiptIntake( + self.store, self.registry, self.clock, + admission=AnchorOrExplicit( + _LazyAdapter(self) if self.adapter_requested else None + ), + ) + return self._intake + + @property + def delivery(self): + if self._delivery is None: + self._delivery = DeliveryService( + self.store, self.registry, self.intake, self.clock + ) + return self._delivery + + @property + def ack(self): + if self._ack is None: + service = AckService( + self.store, self.registry, self.intake, self.delivery, self.clock, + criteria=self.criteria, + ) + # Wired BEFORE the service is published. record_verdict skips its outbox + # obligation when sync is absent, so an ack handed out unwired would drop the + # obligation silently rather than fail. + service.sync = self.sync + self._ack = service + return self._ack + + @property + def reconciler(self): + if self._reconciler is None: + self._reconciler = Reconciler( + self.store, self.registry, self.delivery, self.clock + ) + return self._reconciler + + @property + def sync(self): + if self._sync is None: + self._sync = SyncOutbox(self.store, self.clock) + return self._sync + + @property + def assignments(self): + if self._assignments is None: + self._assignments = AssignmentView( + self.store, self.registry, self.clock, criteria=self.criteria + ) + return self._assignments @property def adapter(self): @@ -107,7 +192,11 @@ def close(self): except Exception: pass self._adapter = None - self.store.close() + # Only a store that was actually built is closed. Reading the property here would + # construct one during teardown, on the very host where constructing it fails. + if self._store is not None: + self._store.close() + self._store = None def _require_adapter(services): @@ -121,6 +210,20 @@ def __init__(self, message, code): self.code = code +class PayloadExit(Exception): + """A completed answer that is still a refusal. + + doctor has to print its whole diagnosis AND exit non-zero when it cannot prove two + participants share a store. A plain refusal would throw the diagnosis away, and a plain + return would let exit 0 be read as yes. + """ + + def __init__(self, payload, code): + super().__init__(payload.get("detail", "refused")) + self.payload = payload + self.code = code + + # --------------------------------------------------------------------- commands @@ -591,40 +694,349 @@ def wait(seconds): def cmd_daemon(services, args) -> dict: _require_adapter(services) - from .daemon import RelayDaemon, SingleInstance + # A bounded daemon run is an explicit operator action, so it does NOT require the managed + # service's enable intent - but it does take the same scope claim, or two standalone runs + # with different state directories could serve one App Server and never see each other. + return _run_bounded(services, _service_for(services), args, require_intent=False) + + +def cmd_store_identity(services, args) -> dict: + """One line each participant can emit, for a comparison to consume.""" + return {"stateSelection": services.selection.to_record(), "store": services.store.locate()} + - directory = Path(args.state) if args.state else state_dir(args.socket) - daemon = RelayDaemon( - services.store, services.registry, services.intake, services.delivery, services.ack, - services.reconciler, services.adapter, clock=services.clock, +def cmd_store_challenge(services, args) -> dict: + """Write a nonce here, or look for one another participant wrote. + + This is the only evidence that survives a copied database: the identifier inside a copy is + identical, but a value written AFTER the copy exists in exactly one of the two files. + """ + if args.write: + return services.store.write_challenge(actor=args.actor or "cli") + if not args.read: + raise SystemExit2("store-challenge needs --write or --read ", EXIT_USAGE) + return services.store.read_challenge(args.read) + + +def _ledger_location(services) -> dict: + """Where the transport ledger will actually live, which --state does not move. + + bridge_adapter._build resolves it with state_dir(socket_path), reading the environment + only, so a run that overrides --state alone splits the relay store from the ledger that + carries send idempotency. Mirrors codex_thread_bridge.ledger.open_endpoint_ledger, which + cannot be called here because opening it is a side effect. + """ + import hashlib + + if not services.socket_path: + return {"configured": False, "directory": None, "path": None, "split": False} + directory = Path(state_dir(services.socket_path)).expanduser() + canonical = Path(services.socket_path).expanduser().absolute().resolve() + endpoint = hashlib.sha256(str(canonical).encode()).hexdigest()[:16] + split = directory.resolve() != services.selection.path.resolve() + return { + "configured": True, "directory": str(directory), + "path": str(directory / f"operations-{endpoint}.sqlite3"), "split": split, + } + + +def _contents(services, report) -> dict: + """Counts, but only when the store can actually be opened for them.""" + from .store import read_only_rows + + if not report["access"]["dbReadable"]: + return {"available": False, "relationships": None, "openAttempts": None, + "detail": "the database is not readable from this process"} + # Read through the probe's own read-only connection. services.store would construct a + # Store, and Store.__init__ opens O_RDWR, switches on WAL and runs the whole schema + # script - so asking doctor to COUNT rows in an empty, legacy or unrelated readable + # relay.sqlite3 quietly turned it into a relay database. Diagnosis writes nothing. + counted = read_only_rows( + services.selection, + "SELECT (SELECT COUNT(*) FROM relationships) AS relationships," + " (SELECT COUNT(*) FROM attempts a" + " JOIN deliveries d ON d.event_id = a.event_id" + " WHERE a.internal_state = 'in_flight'" + " OR (a.state = 'held_uncertain'" + " AND d.state IN ('held_uncertain','sending'))) AS open_attempts", ) - deadline = services.clock.now() + args.deadline if args.deadline else None - with SingleInstance(directory): - reports = daemon.run( - max_ticks=args.max_ticks, deadline=deadline, - sleep=_scheduler_wait(services.clock, deadline), - ) - return {"ticks": [report.as_dict() for report in reports]} + if not counted["readable"] or counted["detail"] or not counted["rows"]: + return {"available": False, "relationships": None, "openAttempts": None, + "detail": counted["detail"] or "the store could not be read"} + relationships = counted["rows"][0]["relationships"] + open_attempts = counted["rows"][0]["open_attempts"] + return {"available": True, "relationships": relationships, + "openAttempts": open_attempts, "detail": None} + + +def _sibling_stores(services) -> dict: + """Other stores beside this one that never recorded which socket they serve. + + A store is matched to a socket by provenance it records for itself. One created before + that existed can only be matched by its directory hash, so if this command is about to + create a fresh canonical database next to such a store, it may be hiding real data. That + is reported rather than resolved: adopting on a guess is how the wrong store gets served. + """ + from .store import stores_without_provenance + + root = services.selection.path.parent + if services.selection.source in ("flag", "env"): + # An explicit directory was chosen by a caller who already decided which participants + # share it, so its neighbours are not candidates for anything. + return {"checked": False, "reason": "the state directory was chosen explicitly", + "withoutProvenance": []} + without = stores_without_provenance(root, skip=services.selection.path.name) + from .store import stores_claiming_socket + + # More than one store recording this socket is an ambiguity discovery refuses to resolve, + # so it has to be visible here or a caller just gets a surprisingly empty database. + claiming = stores_claiming_socket( + root, services.socket_path, skip=services.selection.path.name, + ) + return {"checked": True, "reason": None, "withoutProvenance": without, + "claimingThisSocket": claiming, + "ambiguous": len(claiming) > 1} def cmd_doctor(services, args) -> dict: + """What THIS process can actually do here, measured rather than assumed. + + Constructs no Store: probe() answers from stat, a read-only connection and a rolled-back + write transaction, so a missing, unreadable or read-only state directory is an answer + instead of the failure that would otherwise replace it. + """ import os - return { - "stateDirectory": str(services.store.path.parent), - "procAvailable": os.path.isdir("/proc/self/fd"), - "adapter": "bridge" if services.adapter_requested else "none (read-only, no --socket)", - "relationships": services.store.one( - "SELECT COUNT(*) AS c FROM relationships" - )["c"], - "openAttempts": len(Reconciler( - services.store, services.registry, services.delivery, services.clock - ).open_attempts()), - "actorReachability": _reachability(services), - } + report = probe(services.selection) + report["procAvailable"] = os.path.isdir("/proc/self/fd") + report["adapter"] = ( + "bridge" if services.adapter_requested else "none (read-only, no --socket)" + ) + report["ledger"] = _ledger_location(services) + report["actorReachability"] = _reachability(services, report) + report["contents"] = _contents(services, report) + report["siblingStores"] = _sibling_stores(services) + + nonce = nonce_lookup(services.selection, args.expect_nonce) if args.expect_nonce else None + report["nonce"] = nonce + comparison = compare_store( + report["store"], expect_store=args.expect_store, expect_inode=args.expect_inode, + nonce=nonce, + ) + asked = any((args.expect_store, args.expect_inode, args.expect_nonce)) + report.update(comparison) + if asked and comparison["sameStore"] != "proven": + # A caller that asked whether this is the same store and got no proof must not read + # exit 0 as yes. Unproven is refused for the same reason a mismatch is: the criterion + # is that a different database is never reported as healthy. + raise PayloadExit(report, EXIT_REFUSED) + return report + + +def _service_for(services): + """Built from the probe, so status stays an offline command that constructs no Store.""" + from .service import RelayService + + return RelayService( + services.selection, socket_path=services.socket_path, + store_id=probe(services.selection)["store"]["storeId"], + ) + + +def _refuse_unless_ok(payload: dict) -> dict: + if payload.get("ok"): + return payload + raise PayloadExit(payload, EXIT_REFUSED) + + +def _run_bounded(services, service, args, *, require_intent: bool) -> dict: + """Hold ownership for exactly as long as this process serves, then let it go. + + The daemon is constructed INSIDE the claim so a run that loses the race never opens a + transport connection it is about to abandon. + """ + from .daemon import RelayDaemon + from .service import ServiceRefused, owned_service + + deadline = services.clock.now() + args.deadline if args.deadline else None + allow_isolated = getattr(args, "allow_isolated_scope", False) + # Before the claim, for the same reason _supervise does it: the probe that built this + # service answers from a file that may not exist yet, and a scope registration recorded + # with a null store id can later be overwritten by a different store. + service.store_id = services.store.identity + adopted = _adopt_supervised(service, args) + try: + with owned_service( + service, allow_isolated=allow_isolated, require_intent=require_intent, + adopt_lock_fd=adopted.get("lockFd"), adopt_scope_fd=adopted.get("scopeFd"), + ) as record: + daemon = RelayDaemon( + services.store, services.registry, services.intake, services.delivery, + services.ack, services.reconciler, services.adapter, clock=services.clock, + ) + reports = daemon.run( + max_ticks=args.max_ticks, deadline=deadline, + sleep=_scheduler_wait(services.clock, deadline), + ) + except ServiceRefused as refusal: + raise PayloadExit( + {"ok": False, "reason": refusal.reason, "detail": refusal.detail}, EXIT_REFUSED, + ) from refusal + return {"ok": True, "reason": None, "pid": record["pid"], + "ticks": [report.as_dict() for report in reports]} + + +def _adopt_supervised(service, args) -> dict: + """Validate a supervised invocation, or refuse it. Never fall back to an unlocked run. + + An fstat match proves the descriptor points at the right FILE, not that it shares the + supervisor's open file description - an independently opened descriptor for the same path + passes it. The token recorded in daemon.json and the recorded-parent check are what + actually establish that this process was launched by that supervisor. + """ + from .service import DAEMON_LOCK, arm_parent_death_signal + + import os + + token = getattr(args, "supervised_token", None) + lock_fd = getattr(args, "supervised_lock_fd", None) + scope_fd = getattr(args, "supervised_scope_fd", None) + supplied = [value for value in (token, lock_fd, scope_fd) if value is not None] + if not supplied: + return {} + + def refuse(reason, detail): + raise PayloadExit( + {"ok": False, "reason": reason, "detail": detail}, EXIT_REFUSED, + ) + + if token is None or lock_fd is None or scope_fd is None: + refuse("supervised_invocation_incomplete", + "a supervised worker needs the token and both descriptors together") + record = service.record() or {} + if not record.get("token") or record["token"] != token: + refuse("supervised_token_mismatch", "the token does not match this state directory") + if record.get("stateDir") not in (None, str(service.selection.path)): + refuse("supervised_state_mismatch", "the record names a different state directory") + if record.get("socketPath") not in (None, service.socket_path): + refuse("supervised_socket_mismatch", "the record names a different operating scope") + # The supervisor recorded which store it registered the scope for. This worker opened + # whatever relay.sqlite3 the path resolves to NOW, and a database deleted or atomically + # replaced between segments is a different one - so without this the worker would serve an + # empty or unrelated store while the supervisor and the scope registration still name the + # original, and every participant comparing identities would be told they agree. + if record.get("storeId") not in (None, service.store_id): + refuse("supervised_store_mismatch", + "the record names a different store than this worker opened") + try: + want = os.stat(service.selection.path / DAEMON_LOCK) + got = os.fstat(lock_fd) + except OSError as error: + refuse("supervised_fd_unreadable", f"{type(error).__name__}: {error}") + if (got.st_dev, got.st_ino) != (want.st_dev, want.st_ino): + refuse("supervised_fd_mismatch", "the inherited descriptor is not this daemon lock") + death = arm_parent_death_signal(record.get("pid") or -1) + if death["orphaned"]: + refuse("supervisor_already_gone", + f"parent is {death['parent']}, not the recorded supervisor {record.get('pid')}") + if not death["armed"]: + # Recorded rather than refused. The getppid check above closes the window that + # matters here - a supervisor that is ALREADY gone - and refusing outright would make + # the relay unusable on any host without prctl. What is lost is the later case: if + # the supervisor crashes mid-segment the kernel will not signal this worker, so it + # runs to the end of its bounded segment holding the inherited locks. Bounded, but + # real, and an operator can see it in the record instead of assuming it is armed. + # Appended to the log rather than written into daemon.json: the supervisor owns that + # record and rewrites it at every worker boundary, so a whole-document write from the + # worker would race it and could erase the workerPid a stop needs. + service.store_journal_note( + f"worker {os.getpid()} could not arm PR_SET_PDEATHSIG" + f" ({death.get('detail') or 'no detail'}); if the supervisor crashes this worker" + " runs to the end of its segment holding the inherited locks" + ) + return {"lockFd": lock_fd, "scopeFd": scope_fd if scope_fd >= 0 else None, + "parentDeathSignal": "armed" if death["armed"] else "unarmed"} + + +def cmd_service(services, args) -> dict: + service = _service_for(services) + action = args.service_command + if action == "status": + return service.status() + if action == "enable": + # Through the same refusal path as every other mutating service command: returning + # the payload directly exits zero, and automation would read a refused enable that + # deliberately changed nothing as a success. + return _refuse_unless_ok(service.enable(actor=args.actor or "cli")) + if action == "disable": + return _refuse_unless_ok(service.disable(actor=args.actor or "cli")) + if action == "stop": + return _refuse_unless_ok(service.stop(actor=args.actor or "cli")) + if action in ("start", "restart"): + _require_adapter(services) + call = service.start if action == "start" else service.restart + return _refuse_unless_ok(call( + allow_isolated=args.allow_isolated_scope, deadline=args.deadline, + segment_seconds=args.segment_seconds, max_segments=args.max_segments, + actor=args.actor or "cli", takeover=getattr(args, "takeover_scope", False), + )) + if action == "run": + _require_adapter(services) + return _supervise(services, service, args) + raise SystemExit2(f"unknown service action {action!r}", EXIT_USAGE) + + +def _supervise(services, service, args) -> dict: + """The supervisor: it holds the locks and replaces bounded workers.""" + from .service import ServiceRefused + + service.launch_id = getattr(args, "launch_id", None) + # This process is the one that claims the scope, so the flag has to be honoured here and + # not only in the parent that decided to pass it. + service.takeover = getattr(args, "takeover_scope", False) + # The probe that built this service answers from a file that may not exist yet, so on a + # fresh state directory it reports no store id at all. Opening the store HERE is not the + # side effect doctor and status refuse: a supervisor is about to use it either way. It + # matters because the scope registration is written next, and ScopeRegistry's mismatch + # guard needs both ids to be present - a registration recorded with a null id could be + # overwritten later by a different store, losing the evidence two stores served one socket. + service.store_id = services.store.identity + + def recover(): + # Establish what happened to anything in flight BEFORE a worker can send. Recovery + # itself sends nothing; it only decides what the evidence supports. + services.reconciler.recover_on_start(services.adapter) + _release_expired_leases(services) + + try: + return service.supervise( + allow_isolated=args.allow_isolated_scope, segment_seconds=args.segment_seconds, + max_segments=args.max_segments, deadline=args.deadline, on_start=recover, + ) + except ServiceRefused as refusal: + raise PayloadExit( + {"ok": False, "reason": refusal.reason, "detail": refusal.detail}, EXIT_REFUSED, + ) from refusal -def _reachability(services) -> dict: +def _release_expired_leases(services) -> None: + """An expired lease returns the attempt to reconciliation, never to the send queue. + + A sending row whose lease ran out may already have reached the recipient, so putting it + back to queued would make it eligible to send again on no evidence at all. held_uncertain + is where the reconciler can judge it (I-78). + """ + now = services.clock.now() + with services.store.transaction() as db: + db.execute( + "UPDATE deliveries SET state = 'held_uncertain', lease_owner = NULL," + " lease_until = NULL, updated_at = ?" + " WHERE state = 'sending' AND lease_until IS NOT NULL AND lease_until <= ?", + (services.clock.iso(), now), + ) + + +def _reachability(services, report) -> dict: """What THIS process can actually do here, measured rather than assumed. A workspace-write task cannot write the default state directory and cannot connect to the @@ -634,15 +1046,10 @@ def _reachability(services) -> dict: thread, which is why doctor can answer even where the bridge itself could not load. """ import socket - import tempfile - directory = services.store.path.parent - writable, detail = True, None - try: - with tempfile.NamedTemporaryFile(dir=directory, prefix=".reach-"): - pass - except OSError as error: - writable, detail = False, f"{type(error).__name__}: {error}" + # Reuses the probe's measurement rather than repeating it, so one command cannot report + # two different answers about the same directory. + access = report["access"] connect = "not configured" if services.socket_path: @@ -657,8 +1064,8 @@ def _reachability(services) -> dict: probe.close() return { - "stateDirectoryWritable": writable, - "stateDirectoryDetail": detail, + "stateDirectoryWritable": access["directoryWritable"], + "stateDirectoryDetail": access["detail"], "socketConfigured": bool(services.socket_path), "socketConnect": connect, "offlineCommands": list(OFFLINE_COMMANDS), @@ -924,18 +1331,138 @@ def build_parser() -> argparse.ArgumentParser: daemon = subparsers.add_parser("daemon") daemon.add_argument("--max-ticks", type=int) daemon.add_argument("--deadline", type=float) + daemon.add_argument("--allow-isolated-scope", action="store_true") + daemon.add_argument("--supervised-token") + daemon.add_argument("--supervised-lock-fd", type=int) + daemon.add_argument("--supervised-scope-fd", type=int) daemon.set_defaults(handler=cmd_daemon) - subparsers.add_parser("doctor").set_defaults(handler=cmd_doctor) + service = subparsers.add_parser("service") + actions = service.add_subparsers(dest="service_command", required=True) + for name in ("status", "enable", "disable", "stop"): + offline = actions.add_parser(name) + offline.add_argument("--actor") + for name in ("start", "restart", "run"): + hosted = actions.add_parser(name) + hosted.add_argument("--actor") + hosted.add_argument("--allow-isolated-scope", action="store_true") + # The WORKER's bound. The supervisor replaces workers; it is not itself bounded by + # this, or the service would end after a single segment. + hosted.add_argument("--segment-seconds", type=float) + # The supervisor's own optional bounds, for a test or a deliberately finite run. + hosted.add_argument("--max-segments", type=int) + hosted.add_argument("--deadline", type=float) + hosted.add_argument("--launch-id") + # For a registration whose store no longer exists - deleted, lost or deliberately + # replaced. Refused while anything is live on the scope, so this can only ever + # replace a registration nothing is running behind. + hosted.add_argument( + "--takeover-scope", action="store_true", + help="replace a stopped registration that names a store this one is not", + ) + service.set_defaults(handler=cmd_service) + + doctor = subparsers.add_parser("doctor") + doctor.add_argument("--expect-store", help="the store id another participant reported") + doctor.add_argument("--expect-inode", help="the device:inode another participant reported") + doctor.add_argument("--expect-nonce", help="a nonce another participant wrote here") + doctor.set_defaults(handler=cmd_doctor) + + subparsers.add_parser("store-identity").set_defaults(handler=cmd_store_identity) + + challenge = subparsers.add_parser("store-challenge") + challenge.add_argument("--write", action="store_true") + challenge.add_argument("--read") + challenge.add_argument("--actor") + challenge.set_defaults(handler=cmd_store_challenge) return parser +def _refuse_ambiguous_state(services, args) -> None: + """Two stores already record this socket, so opening one of them would be a guess. + + Falling through to the canonical directory is not the neutral outcome it looks like. The + first command that writes there creates a THIRD empty database, and once that exists it + wins every later resolution and hides the assignments and pending deliveries in both of + the others. Refusing costs one command; the third store costs the state. + + The same refusal covers a store that records NO socket. Its directory hash cannot be + inverted, so if it is this socket's - created from a spelling we cannot reconstruct - then + creating a canonical database beside it hides it just as permanently. That case fires only + when a store would be created; an existing canonical store has already settled it. + + doctor and ack-proof are exempt for opposite reasons. doctor is how an operator finds out + which store to pass to --state, so refusing it would remove the only way out. ack-proof is + a derivation over its own two arguments that opens no store at all. + + An explicit --state or environment override never arrives here: both return from + resolve_state_dir before any discovery runs, because a caller who named a directory has + already decided which participants share it. + + This guard is on the command line rather than on Services.store. A library caller that + builds Services itself bypasses it; every in-process caller in this package passes an + explicit directory, and raising from a property would turn a diagnostic into a crash. + """ + selection = services.selection + # A store records the socket it serves, and the first recording wins so nothing rewrites + # it silently. But an explicit --state or CODEX_SESSION_RELAY_STATE reused with a + # DIFFERENT App Server is a real disagreement: the service would claim and serve the new + # socket while the database goes on attributing itself to the old one, so assignments from + # one App Server can be exposed through another and later discovery still matches the + # store to the socket it no longer serves. Explicit selections reach this even though they + # carry no discovery, because choosing a directory is not choosing what is already in it. + if services.socket_path and selection.db_path.exists(): + recorded = store_socket(selection.db_path) + wanted = canonical_socket(services.socket_path) + if recorded is not None and recorded != wanted and ( + getattr(args, "handler", None) not in (cmd_doctor, cmd_ack_proof) + ): + raise PayloadExit({ + "error": "refused", + "reason": "state_directory_serves_another_socket", + "detail": ( + "this store records a different App Server socket; serving the requested" + " one from it would expose one installation's assignments through another" + ), + "recordedSocket": recorded, + "requestedSocket": wanted, + "stateDirectory": str(selection.path), + }, EXIT_REFUSED) + if not (selection.ambiguous or selection.unidentified): + return + if getattr(args, "handler", None) in (cmd_doctor, cmd_ack_proof): + return + contested = bool(selection.ambiguous) + raise PayloadExit({ + "error": "refused", + "reason": ("ambiguous_state_directory" if contested + else "unidentified_state_directory"), + "detail": ( + "more than one store already records this socket, and creating a new one here" + " would hide them both" + ) if contested else ( + "a store here records no socket, so it cannot be ruled out as this one's;" + " creating a new store beside it would hide it permanently" + ), + "socketPath": services.socket_path, + "candidates": list(selection.ambiguous or selection.unidentified), + "wouldHaveCreated": str(selection.db_path), + "recover": [ + "doctor lists the candidates", + "--state doctor identifies the store", + "--state service status shows which assignments it carries", + "--state once, to adopt it deliberately", + ], + }, EXIT_REFUSED) + + def main(argv=None) -> int: parser = build_parser() args = parser.parse_args(argv) services = None try: services = Services(args) + _refuse_ambiguous_state(services, args) payload = args.handler(services, args) print(json.dumps(payload, indent=2, default=str)) return EXIT_OK @@ -949,6 +1476,9 @@ def main(argv=None) -> int: except SystemExit2 as error: print(json.dumps({"error": "usage", "detail": str(error)}, indent=2)) return error.code + except PayloadExit as error: + print(json.dumps(error.payload, indent=2, default=str)) + return error.code except Exception as error: print(json.dumps({ "error": "host", "detail": f"{type(error).__name__}: {error}" diff --git a/packages/codex-session-relay/src/codex_session_relay/daemon.py b/packages/codex-session-relay/src/codex_session_relay/daemon.py index 27f10f0..4a45cb7 100644 --- a/packages/codex-session-relay/src/codex_session_relay/daemon.py +++ b/packages/codex-session-relay/src/codex_session_relay/daemon.py @@ -43,28 +43,53 @@ def as_dict(self) -> dict: class SingleInstance: - """One daemon per state directory. A second one exits rather than racing the first.""" + """One daemon per state directory. A second one exits rather than racing the first. - def __init__(self, directory): + A supervised worker ADOPTS the descriptor its supervisor already holds rather than taking a + second lock. flock belongs to the open file description, so the supervisor and its worker + share one and the lock stays held while either of them lives - which is what stops a + replacement from starting beside an orphaned worker. + + That sharing is also why a shared holder releases by CLOSING and never by LOCK_UN: + unlocking through any duplicate descriptor releases it for every holder at once. + """ + + def __init__(self, directory, *, shared: bool = False, adopt_fd=None): self.path = Path(directory) / "daemon.lock" + self.shared = bool(shared or adopt_fd is not None) + self.adopted = adopt_fd is not None + self._adopt_fd = adopt_fd self._handle = None def __enter__(self): + if self._adopt_fd is not None: + # Already locked by the supervisor through this same description. Re-acquiring + # would be a second lock on a file we already hold. + self._handle = os.fdopen(self._adopt_fd, "r+", closefd=True) + return self self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - self._handle = open(self.path, "w") + self._handle = open(self.path, "a+") try: fcntl.flock(self._handle, fcntl.LOCK_EX | fcntl.LOCK_NB) except OSError as error: self._handle.close() self._handle = None raise RuntimeError(f"another relay daemon already holds {self.path}") from error + self._handle.seek(0) + self._handle.truncate() self._handle.write(str(os.getpid())) self._handle.flush() + if self.shared: + os.set_inheritable(self._handle.fileno(), True) return self + def fileno(self): + return self._handle.fileno() if self._handle is not None else None + def __exit__(self, *_exc): if self._handle is not None: - fcntl.flock(self._handle, fcntl.LOCK_UN) + if not self.shared: + fcntl.flock(self._handle, fcntl.LOCK_UN) self._handle.close() self._handle = None diff --git a/packages/codex-session-relay/src/codex_session_relay/delivery.py b/packages/codex-session-relay/src/codex_session_relay/delivery.py index f6dcf24..01b80af 100644 --- a/packages/codex-session-relay/src/codex_session_relay/delivery.py +++ b/packages/codex-session-relay/src/codex_session_relay/delivery.py @@ -14,7 +14,7 @@ from .identity import request_id as derive_request_id from .lifecycle import UNKNOWN as LIFECYCLE_UNKNOWN, hold_reason_for, observe, record as record_lifecycle from .policy import RetryPolicy -from .scope import check_recipient +from .scope import assert_assignment_delivery, check_recipient from .transport import ( DEFERRED_BUSY, DISPATCHED, @@ -79,8 +79,14 @@ def enqueue(self, event_id: str, *, kind: str = COMPLETION, recipient_task_id=No relationship["child"]["taskId"] if kind == REVISION else relationship["parent"]["taskId"] ) - # Refused before any transport call, and re-checked inside attempt(). - check_recipient(recipient_task_id, relationship["authorizedScope"]["allowedRecipients"]) + # Refused before any transport call, and re-checked inside attempt(). Checked BEFORE + # the idempotent early return, so re-enqueueing cannot smuggle a cross delivery past + # a row that already exists. + assert_assignment_delivery( + relationship, kind=kind, recipient_task_id=recipient_task_id, + event_relationship_id=event["relationship_id"], + manifest_paths=_manifest_paths(event), + ) existing = self.find(event_id) if existing is not None: return dict(existing) @@ -353,7 +359,12 @@ def attempt(self, event_id: str, adapter, *, now=None, owner: str = "relay"): return None if row["next_eligible_at"] is not None and row["next_eligible_at"] > now: return None - check_recipient(recipient, relationship["authorizedScope"]["allowedRecipients"]) + assert_assignment_delivery( + relationship, kind=row["kind"], recipient_task_id=recipient, + recipient_thread_id=row["recipient_thread_id"], + event_relationship_id=row["relationship_id"], + manifest_paths=_manifest_paths(self.intake.row(event_id)), + ) if self._rate_limited(recipient, now): self._reschedule( event_id, row["state"], now + self.policy.min_send_interval_seconds, @@ -751,3 +762,19 @@ def _status_for_record(observation) -> str: class _NotClaimable(Exception): pass + + +def _manifest_paths(event_row): + """The declared paths a receipt carries, or none. The manifest lives inside the receipt + JSON rather than in a column of its own.""" + if event_row is None: + return () + try: + receipt = json.loads(event_row["receipt"]) + except (TypeError, ValueError, IndexError, KeyError): + return () + entries = receipt.get("manifest") or () + return tuple( + entry["path"] for entry in entries + if isinstance(entry, dict) and isinstance(entry.get("path"), str) + ) diff --git a/packages/codex-session-relay/src/codex_session_relay/policy.py b/packages/codex-session-relay/src/codex_session_relay/policy.py index e617625..329caaf 100644 --- a/packages/codex-session-relay/src/codex_session_relay/policy.py +++ b/packages/codex-session-relay/src/codex_session_relay/policy.py @@ -29,6 +29,35 @@ class RetryPolicy: poll_interval_seconds: float = 20.0 max_sends_per_tick: int = 4 max_reconciles_per_tick: int = 8 + # Supervision cadence. A worker is bounded by segment_seconds; the supervisor replaces it, + # which is what carries an assignment past any single process lifetime. + segment_seconds: float = 3600.0 + restart_interval_seconds: float = 2.0 + restart_base_seconds: float = 2.0 + restart_backoff_max_seconds: float = 300.0 + repeat_failure_threshold: int = 3 + + def restart_delay_for(self, consecutive_failures: int) -> float: + """A clean segment waits the plain interval; repeated failure backs off and caps.""" + import math + + if consecutive_failures <= 0: + return self.restart_interval_seconds + base, ceiling = self.restart_base_seconds, self.restart_backoff_max_seconds + # Bounded before the exponent is evaluated, because a supervisor whose worker fails on + # every segment keeps counting and base * 2 ** (n - 1) eventually builds an integer too + # large to convert to a float - so the supervisor would die of its own backoff instead + # of retrying at the cap. The bound is DERIVED from this policy rather than fixed: a + # small base needs more doublings to reach its ceiling, and a constant would have sent + # a customized supervisor straight to the cap while its own backoff still had room. + if base <= 0 or ceiling <= base: + return ceiling + if consecutive_failures - 1 >= math.ceil(math.log2(ceiling / base)): + return ceiling + return min( + self.restart_backoff_max_seconds, + self.restart_base_seconds * (2 ** (consecutive_failures - 1)), + ) def delay_for(self, attempt_no: int, reason: str) -> float: """Capped exponential backoff, so repeated failure slows down instead of hammering.""" diff --git a/packages/codex-session-relay/src/codex_session_relay/registry.py b/packages/codex-session-relay/src/codex_session_relay/registry.py index 9b7874b..43a003f 100644 --- a/packages/codex-session-relay/src/codex_session_relay/registry.py +++ b/packages/codex-session-relay/src/codex_session_relay/registry.py @@ -483,6 +483,22 @@ def contract_record(record: dict) -> dict: return clean +def project_key(record: dict) -> str: + """Which project an assignment belongs to, for grouping a shared service's work. + + Grouping information, never authorization: the cross-delivery refusal is decided from the + relationship's own endpoints, not from this. A parent with no recorded cwd falls back to + its host, so every assignment lands under some key. + """ + parent = record.get("parent") or {} + cwd = parent.get("cwd") + if cwd: + import posixpath + + return posixpath.normpath(cwd) + return f"host:{parent.get('hostId')}" + + def record_settings(store, clock, task_id: str, settings: dict, *, source: str) -> dict: """Record the execution settings a task was actually created with. diff --git a/packages/codex-session-relay/src/codex_session_relay/scope.py b/packages/codex-session-relay/src/codex_session_relay/scope.py index f842546..d02b227 100644 --- a/packages/codex-session-relay/src/codex_session_relay/scope.py +++ b/packages/codex-session-relay/src/codex_session_relay/scope.py @@ -108,6 +108,56 @@ def check_recipient(task_id: str, allowed) -> None: ) +def assert_assignment_delivery(relationship, *, kind, recipient_task_id, + recipient_thread_id=None, event_relationship_id=None, + manifest_paths=()): + """A delivery belongs to ONE assignment, and goes to that assignment's own endpoint. + + Membership in allowed_recipients is not sufficient by itself. Two assignments on one host + may legitimately authorize the same recipient, so a completion belonging to assignment A + addressed to assignment B's parent passes a membership check and is still a cross + delivery. The DIRECTION is what ties a message to its own assignment: a completion travels + to this relationship's parent, a revision to this relationship's child, and nothing else + is an authorized destination. + """ + rid = relationship["relationshipId"] + if event_relationship_id is not None and event_relationship_id != rid: + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"event belongs to relationship {event_relationship_id!r}, not {rid!r}", + ) + # Named explicitly rather than defaulted. An unrecognised kind falling through to the + # parent branch was admitted as a completion, so the parent received a verification + # request the acknowledgement path then refuses - a delivery nobody can answer. + if kind == "revision_request": + expected = relationship["child"]["taskId"] + elif kind == "completion_event": + expected = relationship["parent"]["taskId"] + else: + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"{kind!r} is not a delivery direction this contract defines, so there is no" + " authorized recipient for it", + ) + if recipient_task_id != expected: + direction = "child" if kind == "revision_request" else "parent" + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"a {kind} for {rid!r} goes to its own {direction} {expected!r}, not to " + f"{recipient_task_id!r}", + ) + if recipient_thread_id is not None and recipient_thread_id != recipient_task_id: + raise ScopeError( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + f"the native thread {recipient_thread_id!r} is not the recipient task " + f"{recipient_task_id!r}", + ) + check_recipient(recipient_task_id, relationship["authorizedScope"]["allowedRecipients"]) + roots = relationship["authorizedScope"]["artifactRoots"] + for path in manifest_paths: + assert_within(path, roots) + + @dataclass(frozen=True) class PathBinding: declared: str diff --git a/packages/codex-session-relay/src/codex_session_relay/service.py b/packages/codex-session-relay/src/codex_session_relay/service.py new file mode 100644 index 0000000..a237bd0 --- /dev/null +++ b/packages/codex-session-relay/src/codex_session_relay/service.py @@ -0,0 +1,1474 @@ +"""Who owns the daemon, and which process may be told to stop. + +Two questions the state directory cannot answer on its own. + +A flock on /daemon.lock proves one daemon per state directory. It says nothing about a +second installation that chose a DIFFERENT state directory for the same App Server, and those +two would each hold their own lock and each believe they were alone. Single ownership +therefore lives in a scope registry keyed by the operating scope - the socket - in a location +neither installation chose. + +And a pid is not a process. Verifying /proc/ and then calling os.kill leaves a window in +which that process exits and its number is reused, so the signal lands somewhere else. Every +signal here goes through a pidfd opened BEFORE the identity check and kept open until the +process is gone, so the thing verified and the thing signalled are the same thing. +""" + +import errno +import fcntl +import hashlib +import json +import os +import pwd +import signal +import time +import uuid +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path + +SCOPE_ENV = "CODEX_SESSION_RELAY_SCOPE_DIR" +PRODUCTION, ISOLATED = "production", "isolated" + +DAEMON_LOCK = "daemon.lock" +DAEMON_RECORD = "daemon.json" +SERVICE_INTENT = "service.json" +DAEMON_LOG = "daemon.log" +STOP_REQUEST = "stop.request" + +OURS, FOREIGN, UNVERIFIABLE, NONE = "ours", "foreign", "unverifiable", "none" + + +def _now() -> str: + from .store import _now_iso + + return _now_iso() + + +def boot_id(): + """Without it a pid recorded before a reboot can match a live unrelated pid today.""" + try: + return Path("/proc/sys/kernel/random/boot_id").read_text(encoding="utf-8").strip() + except OSError: + return None + + +def _stat_fields(pid: int): + """The fields of /proc//stat after the comm. + + The comm can itself contain spaces and brackets, so the split starts after the LAST + closing bracket rather than at the second whitespace-separated field. + """ + try: + raw = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8") + return raw[raw.rindex(")") + 2:].split() + except (OSError, ValueError): + return None + + +def start_ticks(pid: int): + """Field 22. Pids repeat; a pid together with its start time does not.""" + fields = _stat_fields(pid) + try: + return int(fields[19]) + except (TypeError, ValueError, IndexError): + return None + + +def process_state(pid: int): + """Field 3: R, S, D, Z, T and friends.""" + fields = _stat_fields(pid) + return fields[0] if fields else None + + +def production_scope_root() -> Path: + """The one ownership authority a real launch may use. + + Read from the passwd database rather than $HOME, so it does not move with the launch + environment - which is the whole point, since the environment is what distinguishes two + installations. Deliberately not under ~/.codex, which this package never writes. + """ + return Path(pwd.getpwuid(os.geteuid()).pw_dir) / ".codex-session-relay" / "scopes" + + +def resolve_scope_root(): + """(root, authority). An override never silently becomes the authority. + + An environment override alone would put the hole back one level down: two real launches + could set it to different directories, or one could set it and the other not, and both + would claim the same socket while passing every ownership check. So an override marks the + registry ISOLATED, and the commands that start a daemon refuse an isolated authority + unless the caller passes --allow-isolated-scope and means it. + """ + override = os.environ.get(SCOPE_ENV) + if override: + return Path(override).expanduser().absolute(), ISOLATED + return production_scope_root(), PRODUCTION + + +class ScopeUnavailable(Exception): + """The ownership record cannot be reached, so ownership cannot be established.""" + + +@dataclass +class ScopeRegistry: + """One claim per operating scope, in a place neither state directory chose.""" + + root: Path + authority: str = PRODUCTION + _handle: object = field(default=None, repr=False) + + def key(self, socket_path) -> str: + # Resolved, not merely absolute. A socket reachable through a symlink or a relative + # alias is the SAME operating scope, and hashing the supplied spelling would give the + # two launches different keys - so both would take a lock and both would serve one + # App Server, which is the exact thing this registry exists to prevent. + canonical = str(Path(socket_path).expanduser().absolute().resolve()) + digest = hashlib.sha256(canonical.encode()).hexdigest()[:16] + if self.authority == PRODUCTION: + return digest + # Namespaced so an isolated record can never be read as the production one. This is + # NOT socket isolation: two deliberately isolated services still share the socket. + salt = hashlib.sha256(str(self.root).encode()).hexdigest()[:8] + return f"isolated-{salt}-{digest}" + + def prepare(self) -> Path: + """Create and validate the directory, or refuse. There is no second location.""" + try: + self.root.mkdir(mode=0o700, parents=True, exist_ok=True) + info = os.stat(self.root) + except OSError as error: + raise ScopeUnavailable(f"{self.root}: {type(error).__name__}: {error}") from error + if info.st_uid != os.geteuid(): + raise ScopeUnavailable(f"{self.root} is owned by uid {info.st_uid}, not this user") + if info.st_mode & 0o022: + raise ScopeUnavailable(f"{self.root} is group or world writable") + return self.root + + def record_path(self, socket_path) -> Path: + return self.root / f"{self.key(socket_path)}.json" + + def read(self, socket_path): + try: + return json.loads(self.record_path(socket_path).read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + def claim(self, socket_path, record: dict, *, shared: bool = False): + """Take the scope lock and write the record, or report who already holds it. + + Liveness is the LOCK, never a recorded pid. A supervisor that dies leaving a worker + alive still owns the scope through the descriptor the worker inherited, and a dead + pid in the record is not permission to take it. + """ + self.prepare() + # A stopped registration naming a DIFFERENT store is still that store's registration. + # Taking the free lock and overwriting the record would erase the only evidence that + # two stores have served this socket, which is the duplicate this registry exists to + # surface. Replacing it has to be deliberate. + existing = self.read(socket_path) + if (existing and record.get("storeId") and existing.get("storeId") + and existing["storeId"] != record["storeId"] + and not record.get("takeover")): + return {"ok": False, "reason": "scope_registered_to_other_store", + "held_by": existing, "scopeKey": self.key(socket_path)} + path = self.root / f"{self.key(socket_path)}.lock" + handle = open(path, "a+") + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as error: + handle.close() + if error.errno not in (errno.EACCES, errno.EAGAIN): + raise ScopeUnavailable(f"{path}: {type(error).__name__}: {error}") from error + return {"ok": False, "reason": "scope_owned_by_other_store", + "held_by": self.read(socket_path), "scopeKey": self.key(socket_path)} + self._handle = handle + if shared: + os.set_inheritable(handle.fileno(), True) + payload = dict(record, scopeKey=self.key(socket_path), scopeAuthority=self.authority, + socketPath=str(socket_path), registeredAt=_now()) + try: + self.record_path(socket_path).write_text( + json.dumps(payload, indent=2), encoding="utf-8", + ) + except OSError: + # The lock is taken but the registration it stands for was never published. Left + # open, this handle goes on refusing every later start in this process on behalf + # of a registration that does not exist - and nothing releases it before the + # object is collected. + self._handle = None + handle.close() + raise + return {"ok": True, "reason": None, "record": payload, "lockFd": handle.fileno()} + + def release(self, socket_path, *, shared: bool = False) -> None: + """Keep the record, drop the claim. + + The record outlives the process on purpose: a STOPPED duplicate registration on a + different store is still a duplicate, and live-only scanning could not see it. + """ + existing = self.read(socket_path) + if existing is not None: + existing.update(pid=None, workerPid=None, startedAt=None, releasedAt=_now()) + try: + self.record_path(socket_path).write_text( + json.dumps(existing, indent=2), encoding="utf-8") + except OSError: + pass + handle, self._handle = self._handle, None + if handle is not None: + # Closing releases it when this is the last descriptor. Never LOCK_UN: unlocking + # through any duplicate would release a lock a supervised worker still relies on. + if not shared: + try: + fcntl.flock(handle, fcntl.LOCK_UN) + except OSError: + pass + handle.close() + + def conflicts(self, socket_path, *, store_id=None, state_dir=None) -> list: + """Every record for this scope that names a different store, alive or not.""" + out = [] + try: + entries = sorted(self.root.glob("*.json")) + except OSError: + return out + mine = self.key(socket_path) + for entry in entries: + try: + found = json.loads(entry.read_text(encoding="utf-8")) + except (OSError, ValueError): + continue + if found.get("scopeKey") != mine: + continue + same_store = store_id is not None and found.get("storeId") == store_id + same_dir = state_dir is not None and found.get("stateDir") == str(state_dir) + if same_store and same_dir: + continue + out.append({ + "stateDir": found.get("stateDir"), "storeId": found.get("storeId"), + "installationId": found.get("installationId"), "pid": found.get("pid"), + "live": _is_live(found), "reason": "same_scope_different_store", + }) + return out + + +def _is_live(record) -> bool: + pid, recorded_boot = record.get("pid"), record.get("bootId") + if not pid: + return False + if recorded_boot is not None and recorded_boot != boot_id(): + # A pid from a previous boot cannot be this record's process. + return False + if process_state(int(pid)) in (None, "Z"): + return False + ticks = start_ticks(int(pid)) + if ticks is None: + return False + recorded = record.get("startTicks") + return recorded is None or int(recorded) == ticks + + +def installation_id(state_dir) -> str: + """This checkout plus this state directory. A second of either is a second install.""" + package = Path(__file__).resolve().parent + material = f"{package}\0{Path(state_dir).expanduser().absolute().resolve()}" + return hashlib.sha256(material.encode()).hexdigest()[:16] + + +PR_SET_PDEATHSIG = 1 + + +def arm_parent_death_signal(expected_parent: int) -> dict: + """Ask the kernel to signal this process when its parent dies, then check it is not late. + + PR_SET_PDEATHSIG is not delivered retrospectively, so a parent that died before the child + armed it leaves an orphan. The getppid check immediately afterwards is what closes that + window, and it can only be done here, in the child. + + Armed in the child's own bootstrap rather than through preexec_fn, which is unsafe once + the supervisor has started the adapter's transport thread. + """ + armed, detail = False, None + try: + import ctypes + + libc = ctypes.CDLL("libc.so.6", use_errno=True) + armed = libc.prctl(PR_SET_PDEATHSIG, signal.SIGTERM, 0, 0, 0) == 0 + if not armed: + detail = f"prctl failed: errno {ctypes.get_errno()}" + except (OSError, AttributeError, ValueError) as error: + detail = f"{type(error).__name__}: {error}" + actual = os.getppid() + orphaned = actual != expected_parent + return {"armed": armed, "detail": detail, "parent": actual, "orphaned": orphaned} + + +class ProcessHandle: + """A handle to a specific process, not to a number. + + Opened FIRST, verified second, signalled third, all through the same descriptor, and kept + open until the process is gone - including across the grace period before SIGKILL. That + ordering is the point: a pid verified and then signalled can belong to a different process + by the time the signal lands. Holding the descriptor also reserves the pid, so /proc stays + truthful for us while we wait. + + Where pidfd is unavailable this refuses rather than falling back to os.kill. A signal that + cannot be aimed is worse than no signal. + """ + + def __init__(self, pid): + self.pid = int(pid) + self.fd = None + self.detail = None + self.already_gone = False + opener = getattr(os, "pidfd_open", None) + if opener is None or getattr(signal, "pidfd_send_signal", None) is None: + self.detail = "this build has no pidfd support, so a signal cannot be aimed" + return + try: + self.fd = opener(self.pid) + except ProcessLookupError: + self.already_gone = True + self.detail = "the process is already gone" + except OSError as error: + self.detail = f"{type(error).__name__}: {error}" + + @property + def usable(self) -> bool: + return self.fd is not None + + def alive(self) -> bool: + """A zombie is not alive. + + Its /proc entry survives until its parent reaps it, and holding a pidfd keeps the pid + reserved, so a liveness check that only asks whether /proc/ exists would wait out + the whole grace period and then report a terminated process as still running. + """ + state = process_state(self.pid) + return state is not None and state != "Z" + + def send(self, sig) -> bool: + if self.fd is None: + return False + try: + signal.pidfd_send_signal(self.fd, sig) + except ProcessLookupError: + return True + except OSError as error: + self.detail = f"{type(error).__name__}: {error}" + return False + return True + + def close(self) -> None: + if self.fd is not None: + os.close(self.fd) + self.fd = None + + +@dataclass +class ServiceIntent: + """What the owner asked for, independent of whether anything is running. + + Absent means never configured, which is NOT enabled. Nothing here writes enabled=True as a + side effect of starting; only an explicit enable does. + """ + + path: Path + + def read(self) -> dict: + try: + data = json.loads(self.path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {"enabled": False, "configured": False, "changedAt": None, "changedBy": None} + return {"enabled": bool(data.get("enabled")), "configured": True, + "changedAt": data.get("changedAt"), "changedBy": data.get("changedBy")} + + def write(self, *, enabled: bool, actor: str) -> dict: + self.path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + payload = {"enabled": bool(enabled), "changedAt": _now(), "changedBy": actor} + # Replaced atomically, for the same reason the daemon record is. read() treats an + # unreadable document as not configured and therefore disabled, and the supervisor + # re-reads intent at every worker boundary - so a reader landing in the truncated + # middle of this write would stop a service its owner had left enabled. + temporary = self.path.with_name(f".{self.path.name}.{os.getpid()}") + try: + temporary.write_text(json.dumps(payload, indent=2), encoding="utf-8") + os.replace(temporary, self.path) + except OSError: + temporary.unlink(missing_ok=True) + raise + return dict(payload, configured=True) + + +class RelayService: + """Start, stop and describe the daemon that belongs to THIS installation.""" + + def __init__(self, selection, *, socket_path=None, scope=None, store_id=None): + self.selection = selection + self.socket_path = socket_path + self.store_id = store_id + self.launch_id = None + # Set only by an explicit --takeover, and only ever read by ScopeRegistry.claim. + self.takeover = False + if scope is None: + root, authority = resolve_scope_root() + scope = ScopeRegistry(root, authority) + self.scope = scope + self.intent = ServiceIntent(selection.path / SERVICE_INTENT) + self.installation_id = installation_id(selection.path) + + @property + def record_path(self) -> Path: + return self.selection.path / DAEMON_RECORD + + def record(self): + try: + return json.loads(self.record_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + def write_record(self, payload: dict) -> dict: + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + # Replaced atomically. write_text truncates first, and _note rewrites this file on + # every worker boundary, so a concurrent stop could read the empty or half-written + # middle, call a running supervisor absent and return not_running without signalling + # its worker - or miss the foreign markers and write a stop request for someone else. + temporary = self.record_path.with_name(f".{self.record_path.name}.{os.getpid()}") + try: + temporary.write_text(json.dumps(payload, indent=2), encoding="utf-8") + os.replace(temporary, self.record_path) + except OSError: + temporary.unlink(missing_ok=True) + raise + return payload + + def new_record(self, *, pid, token=None, worker_pid=None) -> dict: + return { + "pid": pid, "startTicks": start_ticks(pid), "bootId": boot_id(), + "workerPid": worker_pid, "token": token, + "launchId": self.launch_id, + "takeover": self.takeover or None, + # Written by supervise once on_start has returned. Absent means "not serving yet". + "readyAt": None, + "storeId": self.store_id, "installationId": self.installation_id, + "stateDir": str(self.selection.path), "socketPath": self.socket_path, + "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), + "startedAt": _now(), "restarts": 0, "consecutiveFailures": 0, "lastExit": None, + "nextRestartAt": None, + } + + def ownership(self, record=None): + """ours | foreign | unverifiable | none, decided through a real process handle.""" + record = self.record() if record is None else record + if not record or not record.get("pid"): + if record and record.get("workerPid"): + # A supervisor that died leaving its worker alive keeps the worker identity + # on purpose, so this record still names something stop() will reach. It has + # to be classified first: answering none here sent the orphan path straight + # at another installation's worker. + foreign = self._foreign_markers(record) + if foreign: + return FOREIGN, None, "; ".join(foreign) + if record.get("bootId") is None and boot_id() is not None: + # The same rule the live-supervisor path gets. _stop_worker checks only + # start ticks, and a reboot resets those along with the pid space, so an + # unrelated process holding the old worker number would be signalled. + return UNVERIFIABLE, None, ( + "no boot id is recorded, so this worker pid cannot be" + " distinguished from one reused after a reboot" + ) + return NONE, None, "no daemon record" + # Read from the record itself, so they survive the supervisor's death. A foreign + # installation whose supervisor is gone can still have a live worker recorded, and + # answering none there sent stop() down the orphan path and straight at it. + foreign = self._foreign_markers(record) + handle = ProcessHandle(record["pid"]) + if handle.already_gone: + if foreign: + return FOREIGN, handle, "; ".join(foreign) + if (record.get("workerPid") and record.get("bootId") is None + and boot_id() is not None): + # The same rule the cleared-pid path above already applies, and this path is + # just as stale: the supervisor is gone but its worker number is still in the + # record, so stop() falls through to _stop_worker - which validates start + # ticks only. A reboot resets those along with the pid space, so an unrelated + # process holding the old worker number would be signalled. + return UNVERIFIABLE, handle, ( + "no boot id is recorded, so this worker pid cannot be distinguished from" + " one reused after a reboot" + ) + return NONE, handle, "the recorded process is gone" + if not handle.usable: + if foreign: + return FOREIGN, handle, "; ".join(foreign) + return UNVERIFIABLE, handle, handle.detail + mismatches = list(foreign) + ticks = start_ticks(record["pid"]) + if mismatches: + # Decided BEFORE the start-time question. Installation, store and boot each prove + # the record is foreign on their own, and answering unverifiable would discard a + # definite answer in favour of an uncertain one. + return FOREIGN, handle, "; ".join(mismatches) + if record.get("bootId") is None and boot_id() is not None: + # A reboot resets both the pid space and the start-tick counter, so a record with + # no boot written on a host that HAS one cannot be ruled out as pre-reboot: an + # unrelated process can hold the old number with a matching tick count. + # Conditional on the host, because where no boot id is available at all every + # record lacks one, and refusing them all would leave a service unstoppable by + # its own owner - trading a narrow risk for a certain failure. + return UNVERIFIABLE, handle, ( + "no boot id is recorded, so a pid from before a reboot cannot be ruled out" + ) + if record.get("startTicks") is None or ticks is None: + # Nothing proves it foreign, and without a start time a pid is just a number that + # anything could be holding now. Unverifiable is the honest answer; stop refuses. + return UNVERIFIABLE, handle, ( + "no start time is available for this pid, so identity cannot be established" + ) + if ticks != record["startTicks"]: + mismatches.append("the pid was reused by a different process") + if mismatches: + return FOREIGN, handle, "; ".join(mismatches) + return OURS, handle, None + + def _foreign_markers(self, record) -> list: + """What the record alone proves about who owns it, with no live process required.""" + markers = [] + if record.get("bootId") not in (None, boot_id()): + markers.append("recorded before a different boot") + if record.get("installationId") != self.installation_id: + markers.append("another installation owns it") + if self.store_id is not None and record.get("storeId") not in (None, self.store_id): + markers.append("it is using a different store") + return markers + + def _worker_identified(self, record) -> bool: + """Whether the recorded worker pid provably names OUR worker, still running. + + The same three questions _stop_worker asks before it signals - the process exists, it + is readable, and its start time matches what we recorded - plus the boot question, + which ownership() does not always reach. When the recorded SUPERVISOR pid is gone it + answers none at the already-gone branch, before its own missing-boot check runs, so a + record written before a reboot arrives here naming a worker number that an unrelated + process may now hold. + """ + pid = (record or {}).get("workerPid") + if not pid: + return False + if record.get("bootId") is None and boot_id() is not None: + # Conditional on the host for the reason ownership() gives: where no boot id is + # available at all, every record lacks one, and refusing them all would leave a + # service unusable by its own owner. + return False + handle = ProcessHandle(pid) + try: + if handle.already_gone or not handle.usable: + return False + ticks = record.get("workerStartTicks") + current = start_ticks(pid) + return ticks is not None and current is not None and current == ticks + finally: + handle.close() + + def _holder_is_ours(self, record, owner, markers) -> bool: + """Whether whatever is holding this state directory can be attributed to us. + + Not the same question as "is the owner ours". A supervisor that died leaving our own + worker alive answers none, and that worker still holds the daemon lock through the + descriptor it inherited - refusing there would stop an owner from disabling their own + orphan, which stop() deliberately still reaches. + + Everything else that answers none while the lock is held is a holder we cannot name: + no record at all, a record whose process is gone, or a supervisor that has taken the + lock and not yet published. Writing the shared intent on any of those is how a refused + command still shuts down another installation. + """ + if markers: + return False + if owner == OURS: + return True + return owner == NONE and self._worker_identified(record) + + def lock_is_held(self) -> bool: + """Probed by trying to take it: the lock is the only honest liveness signal.""" + path = self.selection.path / DAEMON_LOCK + if not path.exists(): + return False + try: + handle = open(path, "a+") + except OSError: + return False + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + return True + else: + fcntl.flock(handle, fcntl.LOCK_UN) + return False + finally: + handle.close() + + @contextmanager + def daemon_lock_if_free(self): + """Take the daemon lock and KEEP it for the duration of a decision. + + lock_is_held probes and releases, which is enough to describe a state but not to act + on one: between the probe and whatever the caller does next, a supervisor can acquire + the lock. Holding it across the decision closes that gap - a supervisor cannot start + while we hold it - and failing to take it is itself the answer that someone is there. + + Opened with 'a+' so the probe is atomic even on a state directory that has never run + a daemon: a supervisor starting there has to create and lock this same file. + + Only CONTENTION yields None. Mapping every OSError to "someone holds it" made an + unreadable directory or an exhausted descriptor table indistinguishable from a running + supervisor, and callers act on that answer - stop reports a replacement, disable + refuses an owner. An operational failure is not a statement about who owns this + directory, so it is raised and reported as itself. Which means opening the file has to + sit OUTSIDE the contention handler: open() raises EACCES too, for a lock file that has + become unreadable, and reading that as contention is the same mistake one level down. + """ + path = self.selection.path / DAEMON_LOCK + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(path, "a+") + try: + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError as error: + handle.close() + handle = None + # Only flock's contention errnos. Anything else from flock is operational too. + if error.errno not in (errno.EACCES, errno.EAGAIN): + raise + try: + yield handle + finally: + if handle is not None: + fcntl.flock(handle, fcntl.LOCK_UN) + handle.close() + + # -------------------------------------------------------------- operations + + def authority_check(self, *, allow_isolated: bool): + if self.scope.authority == ISOLATED and not allow_isolated: + return { + "ok": False, "reason": "isolated_scope_not_allowed", + "detail": f"{SCOPE_ENV} is set, which moves the ownership record away from the" + " one authority every launch shares. Pass --allow-isolated-scope if" + " that is deliberate; single ownership is not enforced across" + " differently isolated roots.", + } + return {"ok": True, "reason": None} + + def enable(self, *, actor: str = "cli") -> dict: + # The same ownership question disable asks, and decided and written under the SAME + # lock for the same reason. service.json is shared by everything pointed at this + # state directory, and classifying then writing are two operations: a foreign + # supervisor can pass its own enabled-intent check and take the lock in between, and + # this call would then write enabled=true over a disable that happened during the + # handoff - leaving that supervisor running and eligible to restart. Holding the lock + # across both means no supervisor can start while we decide, and failing to take it + # is itself the answer that one is already there to be classified. + with self.daemon_lock_if_free() as held: + if held is None: + record = self.record() + owner, handle, detail = self.ownership(record) + if handle is not None: + handle.close() + # Read from the record as well as from the classification. A supervisor on + # its way out clears its pids before releasing the lock, and ownership + # answers none once both are absent - exactly the interval in which a foreign + # shutdown could be reversed. + markers = self._foreign_markers(record) if record else [] + if not self._holder_is_ours(record, owner, markers): + return {"ok": False, + "reason": ("not_ours" if (owner == FOREIGN or markers) + else "ownership_unverifiable"), + "detail": detail or "; ".join(markers) or ( + "the daemon lock is held but nothing here identifies its" + " owner" + ), + "intent": self.intent.read(), + "note": "intent is shared with the owner of this state directory" + " and was left unchanged"} + # Either nothing is running here - a stopped registration belonging to someone + # else is not a reason to refuse an owner configuring their own installation, and + # refusing then would make a state directory unusable forever - or what is + # running is ours. + return {"ok": True, "reason": None, + "intent": self.intent.write(enabled=True, actor=actor)} + + def disable(self, *, actor: str = "cli", timeout: float = 10.0) -> dict: + # Ownership BEFORE the intent write. service.json is shared by everything pointed at + # this state directory, and a foreign supervisor re-reads it at every worker + # boundary: writing enabled=false and only then discovering the supervisor is not + # ours refused the stop while still shutting that supervisor down at its next + # boundary. A refusal that still has an effect is not a refusal. + # + # Decided and written UNDER the daemon lock. Checking ownership and then writing are + # two operations, and a foreign supervisor can start between them: it reads + # enabled=true, takes the lock, and then sees the intent we changed afterwards and + # exits at its next boundary. Holding the lock means no supervisor can start while we + # decide, and failing to take it means one is already there to be classified. + refusal = None + with self.daemon_lock_if_free() as held: + if held is None: + # Someone holds it. Re-read: a supervisor that started between our read and + # now has published its record, or is about to. + record = self.record() + owner, handle, detail = self.ownership(record) + if handle is not None: + handle.close() + markers = self._foreign_markers(record) if record else [] + if not self._holder_is_ours(record, owner, markers): + refusal = { + "ok": False, + "reason": ("not_ours" if (owner == FOREIGN or markers) + else "ownership_unverifiable"), + "detail": detail or "; ".join(markers) or ( + "the daemon lock is held but nothing here identifies its owner" + ), + "intent": self.intent.read(), + "stop": {"ok": False, "reason": "refused", + "supervisor": "untouched", "worker": "untouched"}, + "note": "intent is shared with the owner of this state directory and" + " was left unchanged", + } + if refusal is None: + written = self.intent.write(enabled=False, actor=actor) + if refusal is not None: + return refusal + # Outside the lock deliberately: stop() takes it for its own decision, and holding it + # here would make stop misread a free lock as ours. + stopped = self.stop(actor=actor, timeout=timeout) + # A stop that had nothing to stop is not a failure; the intent is what disable owns. + failed = not stopped["ok"] and stopped["reason"] != "not_running" + return {"ok": not failed, "reason": stopped["reason"] if failed else None, + "intent": written, "stop": stopped} + + def status(self) -> dict: + record = self.record() + owner, handle, detail = self.ownership(record) + if handle is not None: + handle.close() + intent = self.intent.read() + held = self.lock_is_held() + return { + "enabled": intent["enabled"], "intentConfigured": intent["configured"], + "intentChangedAt": intent["changedAt"], "intentChangedBy": intent["changedBy"], + # Liveness is the lock, not the recorded pid: a supervisor that died leaving a + # worker alive still holds it through the descriptor the worker inherited. + "running": held, + "ownership": owner, "ownershipDetail": detail, + "pid": (record or {}).get("pid"), "workerPid": (record or {}).get("workerPid"), + "startedAt": (record or {}).get("startedAt"), + "storeId": (record or {}).get("storeId"), "socketPath": self.socket_path, + "installationId": self.installation_id, + "stateDirectory": str(self.selection.path), + "scopeAuthority": self.scope.authority, "scopeRoot": str(self.scope.root), + "lock": "held" if held else "free", + "staleRecord": bool(record and record.get("pid") and not held), + "restarts": (record or {}).get("restarts"), + "consecutiveFailures": (record or {}).get("consecutiveFailures"), + "lastExit": (record or {}).get("lastExit"), + "nextRestartAt": (record or {}).get("nextRestartAt"), + "conflicts": self.conflicts(), + "projects": self.projects(), + } + + def projects(self) -> dict: + """Which projects this one service is carrying, read without opening a Store. + + Grouping only. The cross-delivery refusal is decided per assignment from its own + endpoints; this exists so an operator can see that a single supervisor is serving + several repositories rather than having to infer it. + """ + from .registry import project_key + from .store import read_only_rows + + answer = read_only_rows( + self.selection, + "SELECT relationship_id, issue_key, status, parent_task_id, parent_host_id," + " parent_cwd" + " FROM relationships" + " WHERE superseded_by IS NULL", + ) + if not answer["readable"]: + return {"available": False, "detail": answer["detail"], "projects": []} + if answer["detail"]: + # The file opened and the query did not. An inventory we could not read is not an + # empty inventory, and reporting available with no projects says it is. + return {"available": False, "detail": answer["detail"], "projects": []} + grouped = {} + for row in answer["rows"]: + key = project_key({"parent": {"cwd": row["parent_cwd"], + "hostId": row["parent_host_id"]}}) + entry = grouped.setdefault( + key, {"project": key, "assignments": 0, "active": 0, "parents": set(), + "issues": set()}, + ) + entry["assignments"] += 1 + entry["active"] += 1 if row["status"] == "active" else 0 + entry["parents"].add(row["parent_task_id"]) + entry["issues"].add(row["issue_key"]) + projects = [ + {"project": e["project"], "assignments": e["assignments"], "active": e["active"], + "parents": sorted(e["parents"]), "issues": sorted(e["issues"])} + for e in sorted(grouped.values(), key=lambda e: e["project"]) + ] + return {"available": True, "detail": answer["detail"], "projects": projects} + + def conflicts(self) -> list: + if not self.socket_path: + return [] + try: + return self.scope.conflicts( + self.socket_path, store_id=self.store_id, state_dir=self.selection.path, + ) + except ScopeUnavailable: + return [] + + def stop(self, *, actor: str = "cli", timeout: float = 10.0, grace: float = 0.1) -> dict: + """Signal only a process this installation owns, through a handle to that process.""" + record = self.record() + owner, handle, detail = self.ownership(record) + if owner == NONE and not (record or {}).get("workerPid"): + # Nothing verifiable is running. Deciding that and writing a stop request are two + # operations, and supervise() clears pending requests BEFORE it acquires the lock, + # so a supervisor that starts between them consumes the request we leave and exits + # - while this call reports not_running. Settle it while holding the lock instead. + settled, record, owner, handle, detail = self._settle_absent_owner(record, detail) + if settled is not None: + if handle is not None: + handle.close() + return settled + try: + if owner == FOREIGN: + return {"ok": False, "reason": "not_ours", "detail": detail, + "supervisor": "untouched", "worker": "untouched"} + if owner == UNVERIFIABLE: + # Refusing is the point. Signalling on a pid match alone is how an unrelated + # process gets killed after its number is reused. + return {"ok": False, "reason": "ownership_unverifiable", "detail": detail, + "supervisor": "untouched", "worker": "untouched"} + # Only now. Writing the stop request before validating ownership left a refused + # stop able to halt another installation's supervisor at its next boundary, which + # is a refusal that still had an effect. + self.request_stop() + # NONE still falls through to the worker: a supervisor can die leaving its worker + # alive, and that orphan is exactly what a stop has to reach. + outcome = ("gone" if owner == NONE + else self._terminate(handle, timeout=timeout, grace=grace)) + finally: + if handle is not None: + handle.close() + # Re-read AFTER the supervisor is gone. A supervisor replacing a worker between our + # first read and now means the pid we started with names a worker that has already + # exited, and stopping that one would report success while the replacement, which + # holds the inherited locks, is still delivering. + # + # But only when the re-read still describes the SAME launch. A start that acquires + # the lock after the old supervisor exits publishes its own record here, and acting + # on that would stop the new launch's worker and clear the new supervisor's pid using + # the old one's outcome - reporting success while the replacement is still alive. + fresh = self.record() + # Identity, not just the launch id. A direct 'service run' carries no launch id at + # all, so comparing that field alone made two anonymous launches look like one and + # handed the replacement's worker and record straight back to this stop. The + # supervisor's own pid and start time distinguish them whether or not a launch id + # was ever assigned. + superseded_by_a_new_launch = ( + fresh is not None and record is not None + and self._launch_identity(fresh) != self._launch_identity(record) + ) + if fresh is not None and not superseded_by_a_new_launch: + record = fresh + worker = self._stop_worker(record, timeout=timeout, grace=grace) + if superseded_by_a_new_launch: + # A replacement is running. Whatever we did to the launch we started from, the + # SERVICE is not stopped, and reporting success from the old launch's outcome + # would tell a caller the relay is down while it is still delivering. + return {"ok": False, "reason": "replaced_by_new_launch", + "detail": "a new launch acquired the daemon lock during this stop; it was" + " left untouched and is still running", + "supervisor": outcome, "worker": "untouched"} + supervisor_done = outcome in ("exited", "gone") + worker_done = worker in ("exited", "gone") + if record is not None: + # Identity is kept until termination is CONFIRMED. Clearing a pid we have not + # seen exit would lose the only handle a later stop has to reach it. + cleared = dict(record, stoppedBy=actor) + if supervisor_done: + cleared["pid"] = None + if worker_done: + cleared["workerPid"] = None + cleared["workerStartTicks"] = None + if supervisor_done and worker_done: + cleared["stoppedAt"] = _now() + # Written UNDER the lock, or not at all. The re-read above is a snapshot, and the + # daemon lock is free the moment the old supervisor and worker are gone - so a + # replacement can acquire it after that read and publish its own record, and this + # write would erase the identity of a launch that is running, leaving every later + # status and stop with no handle on it. Holding the lock means no replacement can + # start while we finalise; failing to take it means one already did. + # One rule: this record is written only while we HOLD the lock. Three narrower + # versions of this guard each left a window - a replacement that had published, one + # that had not, an absent record - because each asked what the world looked like at + # a moment rather than excluding change for the duration. Holding the lock is the + # only thing that actually excludes a replacement, so that is the condition. + with self.daemon_lock_if_free() as held: + if held is None: + if supervisor_done and worker_done: + # Both processes this stop acted on are confirmed gone, so whatever + # holds the lock is neither of them. + return {"ok": False, "reason": "replaced_by_new_launch", + "detail": "the daemon lock was taken during this stop; that" + " launch was left untouched and is still running", + "supervisor": outcome, "worker": worker} + # Otherwise the worker we could not confirm is the likeliest holder, + # through the descriptor it inherited. Nothing is written, and nothing + # needs to be: the record still names the processes a later stop must + # reach, which is exactly what this write would have preserved. + elif (latest := self.record()) is not None and ( + self._launch_identity(latest) != self._launch_identity(record) + ): + return {"ok": False, "reason": "replaced_by_new_launch", + "detail": "a new launch published its record during this stop; it" + " was left untouched and is still running", + "supervisor": outcome, "worker": worker} + else: + self.write_record(cleared) + if owner == NONE and worker == "gone": + return {"ok": False, "reason": "not_running", "detail": detail, + "supervisor": "gone", "worker": "gone"} + ok = supervisor_done and worker_done + return {"ok": ok, "reason": None if ok else "did_not_exit", + "detail": None, "supervisor": outcome, "worker": worker} + + @staticmethod + def _launch_identity(record): + """What distinguishes one supervisor's run from the next one's. + + The launch id alone is not enough: a direct 'service run' never has one, so two + anonymous launches compare equal. The supervisor's pid and the moment it started + differ across a replacement whether or not a launch id was assigned. + + Taken in order of stability rather than all at once. A supervisor finishing normally + clears its own pid during cleanup while keeping the launch id and start time, so a + tuple including the pid turned an ordinary exit into a phantom replacement and made + a successful stop report failure. + """ + record = record or {} + if record.get("launchId") is not None: + return ("launchId", record["launchId"]) + if record.get("startedAt") is not None: + # With the start ticks, because _now() records whole seconds: two anonymous + # launches inside the same second share a startedAt, and a replacement acquiring + # the lock in that second would compare equal to the launch being stopped. The + # kernel's start-time counter for that pid does not collide. + return ("startedAt", record["startedAt"], record.get("startTicks")) + return ("pid", record.get("pid")) + + def _settle_absent_owner(self, record, detail): + """Decide 'nothing is running' while HOLDING the lock that would prove otherwise. + + Returns (response_or_None, record, owner, handle, detail). A response ends the stop; + None means the caller should continue with the re-read record and classification. + + A record naming a WORKER never reaches here: that none is our own supervisor gone + with its orphan holding the inherited lock, and reaching that orphan is what stop is + for. + """ + with self.daemon_lock_if_free() as held: + if held is not None: + # We hold it, so no supervisor is running and none can start while we decide. + # Returning without a request is the whole point: a request left behind here + # would be consumed by the next supervisor to start. + return ( + {"ok": False, "reason": "not_running", "detail": detail, + "supervisor": "gone", "worker": "gone"}, + record, NONE, None, detail, + ) + # Not free: something took it. Re-read - it may have published its identity by now. + again = self.record() + owner, handle, fresh = self.ownership(again) + if owner == NONE and not (again or {}).get("workerPid"): + if handle is not None: + handle.close() + return ( + {"ok": False, "reason": "ownership_unverifiable", + "detail": "the daemon lock is held by a process that has not yet" + " recorded its identity", + "supervisor": "untouched", "worker": "untouched"}, + again, owner, None, fresh, + ) + return (None, again, owner, handle, fresh) + + def _terminate(self, handle, *, timeout, grace) -> str: + if not handle.send(signal.SIGTERM): + return "signal_refused" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if not handle.alive(): + return "exited" + time.sleep(grace) + handle.send(signal.SIGKILL) + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if not handle.alive(): + return "exited" + time.sleep(grace) + return "still_running" + + def _stop_worker(self, record, *, timeout, grace) -> str: + """Reach the worker directly when the supervisor is already gone.""" + pid = (record or {}).get("workerPid") + if not pid: + return "gone" + handle = ProcessHandle(pid) + try: + if handle.already_gone or not handle.usable: + return "gone" if handle.already_gone else "unverifiable" + ticks = (record or {}).get("workerStartTicks") + current = start_ticks(pid) + if ticks is None or current is None: + # The same rule the supervisor gets. Without a recorded start time the worker + # pid is just a number, and a crashed worker whose number was reused would + # put SIGTERM into an unrelated process. + return "unverifiable" + if current != ticks: + return "gone" + return self._terminate(handle, timeout=timeout, grace=grace) + finally: + handle.close() + + def restart(self, *, allow_isolated: bool = False, launcher=None, **kw) -> dict: + """Preserves the recorded intent. A disabled service is refused, never switched on.""" + intent = self.intent.read() + if not intent["enabled"]: + return {"ok": False, "reason": "service_disabled", "intent": intent, + "detail": "restart never enables a service the owner turned off"} + stopped = self.stop(**{k: v for k, v in kw.items() if k in ("actor", "timeout")}) + if not stopped["ok"] and stopped["reason"] not in ("not_running",): + return {"ok": False, "reason": stopped["reason"], "stop": stopped} + # Re-read AFTER stopping: an owner who disabled the service while it was being + # stopped must not get a replacement launched from the value we cached above. + if not self.intent.read()["enabled"]: + return {"ok": False, "reason": "service_disabled", "stop": stopped, + "intent": self.intent.read(), + "detail": "intent changed to disabled while the service was stopping"} + started = self.start(allow_isolated=allow_isolated, launcher=launcher, **kw) + return {"ok": started["ok"], "reason": started["reason"], "stop": stopped, + "start": started} + + def start(self, *, allow_isolated: bool = False, launcher=None, actor: str = "cli", + max_ticks=None, deadline=None, segment_seconds=None, max_segments=None, + timeout: float = 20.0, poll: float = 0.05, takeover: bool = False) -> dict: + """Preconditions here; ownership in the child. + + The daemon lock and the scope claim are taken by the process that will HOLD them, not + by this one: a parent that claimed and then exited would hand the child an unowned + scope. So start checks what it can cheaply, launches, and then reports what the child + actually managed to do. + """ + gate = self.authority_check(allow_isolated=allow_isolated) + if not gate["ok"]: + return gate + intent = self.intent.read() + if not intent["enabled"]: + return {"ok": False, "reason": "service_disabled", "intent": intent, + "detail": "start never enables a service; enable it explicitly first"} + if self.lock_is_held(): + return {"ok": False, "reason": "already_running", "status": self.status()} + live = [conflict for conflict in self.conflicts() if conflict["live"]] + if live: + return {"ok": False, "reason": "scope_owned_by_other_store", "conflicts": live} + # A registration naming a store that no longer exists - a database deleted, lost or + # deliberately replaced - would otherwise block this socket forever, with recovery + # only through finding and removing a registry file by hand. Taking it over is + # explicit, and only ever from a registration nothing is running behind: the live + # check above has already refused if anything holds the scope. + self.takeover = bool(takeover) + + launch = uuid.uuid4().hex + self.launch_id = launch + child = (launcher or self.default_launcher)( + self, allow_isolated=allow_isolated, max_ticks=max_ticks, deadline=deadline, + segment_seconds=segment_seconds, max_segments=max_segments, + ) + deadline_at = time.monotonic() + timeout + try: + return self._await_launch(child, launch, deadline_at, timeout=timeout, poll=poll) + except BaseException: + # Anything that leaves this call without a confirmed result - an exception, a + # KeyboardInterrupt while recovery is still initialising - leaves a child the + # default launcher put in its own session, so no terminal signal reaches it. It + # would finish starting up, take both locks and begin serving a launch the caller + # was never told had succeeded. + self._abandon(child, timeout=timeout) + raise + + def _await_launch(self, child, launch, deadline_at, *, timeout, poll): + """Wait for the child to publish a record this call can recognise as its own launch.""" + while time.monotonic() < deadline_at: + record = self.record() or {} + # Matched on the launch id, not on the pid changing. After a crash the OS can + # hand the replacement the very pid the stale record already names, and waiting + # for a different number would time out on a service that is running fine. + # The pid must still be there: supervision clears it on the way out while the + # daemon lock is not yet released, so a launch that already finished spends a + # moment matching the id and holding the lock with nothing left running. + # readyAt is written only after on_start returns, so recovery that is still + # running - or about to fail on an App Server connection - is not success either. + if (record.get("pid") and record.get("readyAt") + and record.get("launchId") == launch and self.lock_is_held()): + # The child minted or opened the store; this process only probed a path that + # may not have existed yet. Without adopting its identity, conflicts() cannot + # recognise the child's own scope registration as this store and reports the + # service we just started as same_scope_different_store. + if self.store_id is None and record.get("storeId"): + self.store_id = record["storeId"] + return {"ok": True, "reason": None, "pid": record["pid"], + "scopeAuthority": self.scope.authority, + "scopeRoot": str(self.scope.root), "status": self.status()} + if getattr(child, "poll", lambda: None)() is not None: + return {"ok": False, "reason": "child_exited", + "exitCode": child.returncode, "log": self._log_tail()} + time.sleep(poll) + # A launch that never reported is not a launch that can be left alone. It may still + # be initialising and would come up AFTER the caller was told it failed, holding the + # locks against the retry the caller is about to make. + return {"ok": False, "reason": "did_not_report", "log": self._log_tail(), + "child": self._abandon(child, timeout=timeout)} + + def _abandon(self, child, *, timeout: float) -> str: + """Stop a child this call started and could not confirm, and reap it.""" + if getattr(child, "poll", lambda: None)() is not None: + return "exited" + for step in (getattr(child, "terminate", None), getattr(child, "kill", None)): + if step is None: + continue + try: + step() + child.wait(timeout=max(1.0, timeout / 2)) + return "terminated" + except Exception: # noqa: BLE001 - a child we cannot reach is reported, not raised + continue + return "still_running" + + def _log_tail(self, lines: int = 20) -> str: + try: + return "".join( + (self.selection.path / DAEMON_LOG).read_text(encoding="utf-8").splitlines(True) + [-lines:] + ) + except OSError: + return "" + + def default_launcher(self, service, *, allow_isolated, max_ticks=None, deadline=None, + segment_seconds=None, max_segments=None): + import subprocess + import sys + + argv = [sys.executable, "-m", "codex_session_relay.cli", + "--state", str(self.selection.path)] + if self.socket_path: + argv += ["--socket", str(self.socket_path)] + argv += ["service", "run"] + if allow_isolated: + argv.append("--allow-isolated-scope") + if segment_seconds is not None: + argv += ["--segment-seconds", str(segment_seconds)] + if max_segments is not None: + argv += ["--max-segments", str(max_segments)] + if deadline is not None: + argv += ["--deadline", str(deadline)] + environment = dict(os.environ) + if self.scope.authority == ISOLATED: + # The ALREADY RESOLVED absolute root, so a relative override cannot resolve + # differently in the child, and the child cannot land in another registry. + environment[SCOPE_ENV] = str(self.scope.root) + # The transport ledger resolves from the environment, not from --state, so a managed + # launch forwarding only the flag would inherit the store/ledger split. + environment["CODEX_SESSION_RELAY_STATE"] = str(self.selection.path) + if self.launch_id: + argv += ["--launch-id", self.launch_id] + if self.takeover: + # The supervisor is the process that CLAIMS the scope, so the flag has to reach + # it. Set only on this object, it was dropped at the process boundary and the + # takeover silently did nothing. + argv.append("--takeover-scope") + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: + return subprocess.Popen( + argv, stdout=log, stderr=log, stdin=subprocess.DEVNULL, + start_new_session=True, env=environment, + ) + + # ------------------------------------------------------------- supervision + + @property + def stop_request_path(self) -> Path: + return self.selection.path / STOP_REQUEST + + def request_stop(self) -> None: + """Recorded BEFORE any signal, so a graceful stop is never read as a crash.""" + self.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + self.stop_request_path.write_text(_now(), encoding="utf-8") + + def clear_stop_request(self) -> None: + try: + self.stop_request_path.unlink() + except OSError: + pass + + def stop_requested(self) -> bool: + return self.stop_request_path.exists() + + def spawn_worker(self, *, lock_fd, scope_fd, token, segment_seconds, allow_isolated): + """One bounded worker, sharing the descriptors this supervisor already holds.""" + import subprocess + import sys + + argv = [sys.executable, "-m", "codex_session_relay.cli", + "--state", str(self.selection.path)] + if self.socket_path: + argv += ["--socket", str(self.socket_path)] + argv += ["daemon", "--deadline", str(segment_seconds), + "--supervised-token", token, + "--supervised-lock-fd", str(lock_fd), + "--supervised-scope-fd", str(scope_fd if scope_fd is not None else -1)] + if allow_isolated: + argv.append("--allow-isolated-scope") + environment = dict(os.environ) + if self.scope.authority == ISOLATED: + environment[SCOPE_ENV] = str(self.scope.root) + environment["CODEX_SESSION_RELAY_STATE"] = str(self.selection.path) + pass_fds = tuple(fd for fd in (lock_fd, scope_fd) if fd is not None) + with open(self.selection.path / DAEMON_LOG, "a", encoding="utf-8") as log: + return subprocess.Popen( + argv, stdout=log, stderr=log, stdin=subprocess.DEVNULL, + pass_fds=pass_fds, env=environment, + ) + + def supervise(self, *, allow_isolated=False, segment_seconds=None, max_segments=None, + deadline=None, spawn=None, policy=None, sleeper=None, on_start=None) -> dict: + """Replace bounded workers for as long as the owner wants this service running. + + RelayDaemon.run stays bounded by construction; continuation is a supervisor OVER + successive bounded workers, not a longer loop inside one. The locks are acquired once + here and inherited by every worker, so the store, the generations and the scope claim + are untouched across a worker boundary - which is what carries an assignment past any + single process lifetime without asking the parent model anything. + """ + from .daemon import SingleInstance + from .policy import RetryPolicy + + policy = policy or RetryPolicy() + sleeper = sleeper or time.sleep + spawn = spawn or self.spawn_worker + segment_seconds = segment_seconds or policy.segment_seconds + + gate = self.authority_check(allow_isolated=allow_isolated) + if not gate["ok"]: + raise ServiceRefused(gate["reason"], gate["detail"]) + if not self.intent.read()["enabled"]: + raise ServiceRefused( + "service_disabled", + "this service is not enabled; supervising it would ignore the owner's intent", + ) + + self.clear_stop_request() + token = uuid.uuid4().hex + segments, failures, degraded = [], 0, None + outstanding = None + with SingleInstance(self.selection.path, shared=True) as lock: + # Re-read UNDER the lock. The check above ran while nothing was held, so a disable + # landing between them is missed: this supervisor starts, and the disabling caller + # - which classified the PREVIOUS holder and wrote outside the lock - has written + # enabled=false at a supervisor it never examined, which this one then obeys at its + # first worker boundary. Whoever holds this lock is the one whose intent decides. + if not self.intent.read()["enabled"]: + raise ServiceRefused( + "service_disabled", + "this service was disabled while this supervisor was starting", + ) + scope_fd = None + if self.socket_path: + claim = self.scope.claim( + self.socket_path, self.new_record(pid=os.getpid(), token=token), + shared=True, + ) + if not claim["ok"]: + raise ServiceRefused(claim["reason"], json.dumps(claim.get("held_by"))) + scope_fd = claim["lockFd"] + self.write_record(self.new_record(pid=os.getpid(), token=token)) + try: + # Started BEFORE on_start, so the bound covers the whole supervisor run. + # Recovery can make real App Server calls for unresolved attempts, and timing + # only the loop let --deadline N spend an arbitrary startup interval first and + # then run for another N - even spawning a worker past the requested bound. + started = time.monotonic() + if on_start is not None: + on_start() + # Only now is this service serving. Recovery runs before any worker can send, + # and a caller told "started" while it was still in flight would go on to use + # a service that might yet fail to initialise at all. + # + # And not at all if the bound was spent getting ready: the loop below exits + # immediately in that case, so publishing readiness here would let a waiting + # start() report a service that is already on its way out. + expired = deadline is not None and time.monotonic() - started >= deadline + if not expired: + self._note(readyAt=_now()) + while True: + if max_segments is not None and len(segments) >= max_segments: + break + if deadline is not None and time.monotonic() - started >= deadline: + break + if self.stop_requested(): + break + # Re-read every cycle: an owner who disables the service while a worker + # was running gets no replacement, and the supervisor never writes intent. + if not self.intent.read()["enabled"]: + break + child = spawn( + lock_fd=lock.fileno(), scope_fd=scope_fd, token=token, + # Clamped to what remains of the supervisor's own bound, or a worker + # started just before the deadline outlives it by a whole segment. + segment_seconds=( + segment_seconds if deadline is None else + max(0.1, min(segment_seconds, + deadline - (time.monotonic() - started))) + ), + allow_isolated=allow_isolated, + ) + # Marked outstanding BEFORE the bookkeeping that can fail. A _note that + # raises after a successful spawn left a running worker the cleanup then + # treated as never started, clearing the identity a stop needs while the + # worker still holds the inherited locks. + outstanding = child + self._note(workerPid=child.pid, workerStartTicks=start_ticks(child.pid)) + code = child.wait() + outstanding = None + self._note(workerPid=None, workerStartTicks=None, lastExit=code, + restarts=len(segments) + 1) + segments.append(code) + failures = failures + 1 if code != 0 else 0 + if failures >= policy.repeat_failure_threshold: + degraded = (f"{failures} consecutive worker failures, last exit {code}") + self.store_journal_note(degraded) + self._note(consecutiveFailures=failures, degraded=degraded) + if self.stop_requested() or not self.intent.read()["enabled"]: + break + if max_segments is not None and len(segments) >= max_segments: + # The top of the loop stops for this too, but only AFTER the restart + # delay below - so a finite run outran its own bound by up to the + # backoff cap, which after repeated failures is five minutes, before + # returning. There is nothing left to wait for. + break + if deadline is not None and time.monotonic() - started >= deadline: + break + wait = policy.restart_delay_for(failures) + if deadline is not None: + # Clamped the same way a worker segment is. An unclamped delay - up to + # five minutes after repeated failures - outlives the supervisor's own + # bound, so a short deadline took minutes to return. + wait = max(0.0, min(wait, deadline - (time.monotonic() - started))) + self._note(nextRestartAt=time.time() + wait) + sleeper(wait) + finally: + if self.socket_path: + # shared=True: this descriptor is the one every worker inherited, and + # LOCK_UN through it would release the scope for all of them. After an + # exception a worker may still be alive, and unlocking would let another + # state directory claim this socket beside the orphan. + self.scope.release(self.socket_path, shared=True) + current = self.record() or {} + # nextRestartAt with it: it names a restart this supervisor is no longer going + # to make, and leaving it behind let a stopped service report a pending one. + cleared = dict(current, pid=None, stoppedAt=_now(), nextRestartAt=None) + if outstanding is None: + cleared["workerPid"] = None + cleared["workerStartTicks"] = None + # Otherwise the worker was never waited on - an exception between spawn and + # wait - and it still holds the inherited locks. Erasing its pid and start + # time would leave a stop with nothing to aim at, so the orphan would keep + # delivering for the rest of its segment while status reported not_running. + self.write_record(cleared) + return {"ok": True, "reason": None, "segments": segments, + "consecutiveFailures": failures, "degraded": degraded} + + def _note(self, **fields) -> None: + record = self.record() + if record is not None: + self.write_record(dict(record, **fields)) + + def store_journal_note(self, detail: str) -> None: + """Repeated failure is reported, not silently retried forever.""" + path = self.selection.path / DAEMON_LOG + try: + with open(path, "a", encoding="utf-8") as log: + log.write(f"{_now()} service_degraded: {detail}\n") + except OSError: + pass + + +@contextmanager +def owned_service(service: RelayService, *, allow_isolated: bool = False, token=None, + require_intent: bool = True, adopt_lock_fd=None, adopt_scope_fd=None): + """Hold the daemon lock and the scope claim for as long as this process serves. + + Both are released together on the way out, and the scope RECORD is deliberately kept: a + stopped registration on a different store is still a duplicate registration, and a scan + that only looked at live records could not see it. + """ + from .daemon import SingleInstance + + gate = service.authority_check(allow_isolated=allow_isolated) + if not gate["ok"]: + raise ServiceRefused(gate["reason"], gate["detail"]) + if require_intent and not service.intent.read()["enabled"]: + raise ServiceRefused( + "service_disabled", + "this service is not enabled; running it would ignore the owner's intent", + ) + supervised = adopt_lock_fd is not None + with SingleInstance(service.selection.path, adopt_fd=adopt_lock_fd): + claim = {"ok": True, "reason": None} + if service.socket_path and not supervised: + claim = service.scope.claim(service.socket_path, service.new_record( + pid=os.getpid(), token=token, + )) + if not claim["ok"]: + raise ServiceRefused(claim["reason"], json.dumps(claim.get("held_by"))) + if supervised: + # The supervisor owns both the claim and the record. A worker that rewrote them + # would erase the identity a stop needs to reach the supervisor. + record = service.record() or service.new_record(pid=os.getpid(), token=token) + else: + record = service.write_record(service.new_record(pid=os.getpid(), token=token)) + try: + yield record + finally: + if not supervised: + if service.socket_path: + service.scope.release(service.socket_path) + current = service.record() or record + service.write_record( + dict(current, pid=None, workerPid=None, stoppedAt=_now()), + ) + elif adopt_scope_fd is not None and adopt_scope_fd >= 0: + # Close our copy only. The supervisor still holds the description, so the + # scope stays claimed; unlocking would release it for both of us. + try: + os.close(adopt_scope_fd) + except OSError: + pass + + +class ServiceRefused(Exception): + def __init__(self, reason, detail=None): + super().__init__(detail or reason) + self.reason = reason + self.detail = detail diff --git a/packages/codex-session-relay/src/codex_session_relay/store.py b/packages/codex-session-relay/src/codex_session_relay/store.py index ff5d5d2..16d95c7 100644 --- a/packages/codex-session-relay/src/codex_session_relay/store.py +++ b/packages/codex-session-relay/src/codex_session_relay/store.py @@ -13,7 +13,11 @@ import json import os import sqlite3 +import tempfile +import uuid from contextlib import contextmanager +from dataclasses import dataclass +from datetime import datetime, timezone from pathlib import Path SCHEMA_VERSION = 1 @@ -458,6 +462,12 @@ detail TEXT ); +CREATE TABLE IF NOT EXISTS store_challenge ( + nonce TEXT PRIMARY KEY, + written_by TEXT NOT NULL, + written_at TEXT NOT NULL +); + CREATE INDEX IF NOT EXISTS deliveries_state ON deliveries (state, next_eligible_at); CREATE INDEX IF NOT EXISTS attempts_open ON attempts (internal_state); CREATE INDEX IF NOT EXISTS events_relationship ON events (relationship_id, execution_generation); @@ -469,23 +479,237 @@ """ -def state_dir(socket_path: str | None = None) -> Path: - """Runtime state lives outside every repository, mirroring the bridge's convention.""" - override = os.environ.get("CODEX_SESSION_RELAY_STATE") +STATE_ENV = "CODEX_SESSION_RELAY_STATE" +PRECEDENCE = ("flag", "env", "xdg", "home") + + +def _now_iso() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds").replace("+00:00", "Z") + + +def socket_scope(socket_path) -> str: + if not socket_path: + return "default" + # Canonicalised the same way ScopeRegistry.key does. Hashing the spelling as supplied + # gave a relative path and a symlink alias for one socket two different default state + # directories, so the second invocation opened a different store and was then refused + # by the scope registry as a foreign owner instead of joining the service already there. + return hashlib.sha256(canonical_socket(socket_path).encode()).hexdigest()[:16] + + +def canonical_socket(socket_path) -> str: + """One spelling for one socket, the same way ScopeRegistry.key resolves it.""" + return str(Path(socket_path).expanduser().absolute().resolve()) + + +def legacy_socket_scope(socket_path) -> str: + """What socket_scope produced before it canonicalised, for finding an existing store.""" + if not socket_path: + return "default" + return hashlib.sha256(str(Path(socket_path).expanduser()).encode()).hexdigest()[:16] + + +@dataclass(frozen=True) +class StateSelection: + """Which rule chose the state directory, and the exact value that won. + + Four rules can decide where the store lives, and a participant that cannot say which one + applied to it cannot be compared with another participant. The reason travels with the + path for that reason alone. + """ + + path: Path + source: str + detail: str + socket_scope: str | None + # Every store that already records this socket, when there is more than one of them and + # no canonical store exists yet. Empty for every ordinary selection. It travels ON the + # selection because the decision not to create a store here has to reach whoever holds + # the path they would otherwise have created one at. + ambiguous: tuple = () + # Stores that record NO socket at all, when this selection is about to create a new one + # beside them. They predate provenance, their directory hash cannot be inverted, and one + # of them may be this socket's - so creating here may be hiding real assignments. + unidentified: tuple = () + + @property + def db_path(self) -> Path: + return self.path / "relay.sqlite3" + + def to_record(self) -> dict: + return { + "path": str(self.path), "dbPath": str(self.db_path), "source": self.source, + "detail": self.detail, "socketScope": self.socket_scope, + "precedence": list(PRECEDENCE), "ambiguous": list(self.ambiguous), + "unidentified": list(self.unidentified), + } + + +def resolve_state_dir(explicit=None, socket_path=None) -> StateSelection: + """Runtime state lives outside every repository, mirroring the bridge's convention. + + Highest precedence first: an explicit --state, then the environment override, then + XDG_STATE_HOME, then the home default. Only the last two carry a socket scope; an + explicit directory is used exactly as given, because the caller has already decided + which participants share it. + """ + if explicit: + return StateSelection( + Path(explicit).expanduser().absolute(), "flag", f"--state {explicit}", None + ) + override = os.environ.get(STATE_ENV) if override: - return Path(override).expanduser() - base = Path( - os.environ.get("XDG_STATE_HOME") or (Path.home() / ".local" / "state") - ).expanduser() - root = base / "codex-session-relay" - if socket_path: - endpoint = hashlib.sha256(str(Path(socket_path).expanduser()).encode()).hexdigest()[:16] - return root / endpoint - return root / "default" + return StateSelection( + Path(override).expanduser().absolute(), "env", f"{STATE_ENV}={override}", None + ) + xdg = os.environ.get("XDG_STATE_HOME") + if xdg: + base, source, detail = Path(xdg).expanduser(), "xdg", f"XDG_STATE_HOME={xdg}" + else: + base, source = Path.home() / ".local" / "state", "home" + detail = f"default under {Path.home() / '.local' / 'state'}" + scope = socket_scope(socket_path) + chosen = (base / "codex-session-relay" / scope).absolute() + if (chosen / "relay.sqlite3").exists(): + return StateSelection(chosen, source, detail, scope) + # An existing store keeps its directory. Canonicalising the socket changed this hash, so + # a relative or symlinked socket that had been running would otherwise point at a fresh + # empty database while its assignments, generations and pending deliveries sat in the + # old one, invisible. The new name is used for anything new; the old one wins only when + # it actually holds a store and the new one does not. + legacy = legacy_socket_scope(socket_path) + # Whether the canonical DATABASE exists, not whether its directory does. A directory is + # created by any command that writes beside the store - a stop request is enough - and + # testing for the directory let one such command hide a legacy store holding real + # assignments behind an empty folder. + if legacy != scope: + previous = (base / "codex-session-relay" / legacy).absolute() + if (previous / "relay.sqlite3").exists(): + return StateSelection( + previous, source, + f"{detail}; kept the directory this socket was already using", + legacy, + ) + # The legacy hash only helps when THIS invocation used the old spelling. A first + # post-upgrade command that happens to use the absolute path has legacy == scope, so the + # comparison above never looks at the store the relative spelling created - and creating + # a canonical database here would hide it for good, because afterwards even the old + # spelling finds the new one. So before creating anything, ask the stores themselves. + # Only reached when no canonical database exists yet, which is the one moment it matters. + claims = stores_claiming_socket(base / "codex-session-relay", socket_path, skip=scope) + if len(claims) == 1: + adopted = Path(claims[0]) + return StateSelection( + adopted, source, + f"{detail}; adopted the store already recorded for this socket", + adopted.name, + ) + if len(claims) > 1: + # Returning the canonical directory with nothing to say about the conflict is how a + # THIRD store gets made. The first command to write here creates it, and from that + # moment the canonical-exists branch at the top of this function wins every later + # resolution, so both of the real stores - with their assignments, generations and + # pending deliveries - are invisible. Choosing between them would be just as wrong in + # a quieter way. So the ambiguity travels with the path and the caller refuses. + return StateSelection( + chosen, source, + f"{detail}; {len(claims)} stores already record this socket", + scope, tuple(claims), + ) + # Nothing claims this socket, so a store is about to be created here. A store that + # predates provenance records no socket at all and its directory hash cannot be inverted, + # so if one of them IS this socket's - created from a spelling we cannot reconstruct - + # creating a canonical database now hides it permanently, exactly the way a third store + # would. Reporting them through doctor alone was not enough, because ordinary commands do + # not run doctor. Only when we would CREATE: an existing canonical store has already + # answered the question and returned above. + unidentified = stores_without_provenance(base / "codex-session-relay", skip=scope) + if unidentified: + return StateSelection( + chosen, source, + f"{detail}; {len(unidentified)} stores here record no socket", + scope, (), tuple(unidentified), + ) + return StateSelection(chosen, source, detail, scope) + + +def store_socket(db_path) -> str | None: + """The canonical socket a store recorded for itself, or None if it never recorded one.""" + try: + connection = sqlite3.connect(f"{Path(db_path).as_uri()}?mode=ro", uri=True, timeout=5) + except (OSError, sqlite3.Error, ValueError): + return None + try: + row = connection.execute( + "SELECT value FROM schema_meta WHERE key = 'socket_path'" + ).fetchone() + except sqlite3.Error: + return None + finally: + connection.close() + return row[0] if row else None + + +def discover_store_for_socket(root, socket_path, *, skip=None): + """The directory holding the store this socket already has, under any spelling. + + Provenance rather than arithmetic: a hash cannot be inverted, so a store created under a + spelling we cannot guess is only findable if it says which socket it belongs to. Stores + record that from now on; one created before it did says nothing and is reported by doctor + instead of being adopted on a guess. + + Exactly one, because two stores claiming one socket is an ambiguity rather than a choice. + This answers the narrow question "is there a single store to adopt". A caller that has to + ACT on the difference between none and several reads stores_claiming_socket, which is the + one walk both of them share. + """ + claims = stores_claiming_socket(root, socket_path, skip=skip) + return Path(claims[0]) if len(claims) == 1 else None + + +def stores_claiming_socket(root, socket_path, *, skip=None) -> list: + """Every store recording this socket. More than one is an ambiguity, not a choice.""" + if not socket_path: + return [] + try: + candidates = sorted(p for p in Path(root).iterdir() if p.is_dir()) + except OSError: + return [] + wanted = canonical_socket(socket_path) + return [ + str(directory) for directory in candidates + if (skip is None or directory.name != skip) + and (directory / "relay.sqlite3").exists() + and store_socket(directory / "relay.sqlite3") == wanted + ] + + +def stores_without_provenance(root, *, skip=None) -> list: + """Store directories that never recorded which socket they serve. + + They cannot be matched to a socket by anything but their directory hash, so a command + that creates a fresh canonical database beside one of them may be hiding real data. + Reported rather than adopted: adopting on a guess is how the wrong store gets served. + """ + try: + candidates = sorted(p for p in Path(root).iterdir() if p.is_dir()) + except OSError: + return [] + found = [] + for directory in candidates: + if skip is not None and directory.name == skip: + continue + database = directory / "relay.sqlite3" + if database.exists() and store_socket(database) is None: + found.append(str(directory)) + return found +def state_dir(socket_path: str | None = None) -> Path: + """The directory the environment alone would choose. Kept for callers that have no flag.""" + return resolve_state_dir(None, socket_path).path class Store: - def __init__(self, path): + def __init__(self, path, socket_path=None): path = Path(path) path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) descriptor = os.open(path, os.O_CREAT | os.O_RDWR, 0o600) @@ -500,9 +724,79 @@ def __init__(self, path): self.db.execute( "INSERT OR IGNORE INTO schema_meta VALUES ('version', ?)", (str(SCHEMA_VERSION),) ) + # Minted once and never rewritten, so reopening a store - or restarting the daemon on + # it - cannot look like a new store. It is deliberately NOT proof on its own: copying + # the file copies the identifier too, which is what store_challenge exists for. + self.db.execute( + "INSERT OR IGNORE INTO schema_meta VALUES ('store_id', ?)", (uuid.uuid4().hex,) + ) + self.db.execute( + "INSERT OR IGNORE INTO schema_meta VALUES ('store_created_at', ?)", (_now_iso(),) + ) + if socket_path: + # Provenance, so this store is findable by the socket it serves rather than only + # by the hash of whichever spelling created it. INSERT OR IGNORE: the first + # recording wins, so re-opening through a different spelling never rewrites it. + self.db.execute( + "INSERT OR IGNORE INTO schema_meta VALUES ('socket_path', ?)", + (canonical_socket(socket_path),), + ) # Tests set this to prove a transition rolls back; nothing in production assigns it. self.fault_hook = None + # ------------------------------------------------------------------ identity + + def meta(self, key: str): + row = self.one("SELECT value FROM schema_meta WHERE key = ?", (key,)) + return row["value"] if row is not None else None + + @property + def identity(self): + return self.meta("store_id") + + def locate(self) -> dict: + """The physical facts a same-store comparison needs, not just the path we were given. + + The device and inode come from the RESOLVED path, so a symlink or a bind mount that + reaches the same bytes compares equal while two genuinely different files do not. + """ + try: + real = self.path.resolve() + info = os.stat(real) + device, inode, real_path = info.st_dev, info.st_ino, str(real) + except OSError: + device = inode = real_path = None + return { + "exists": True, + "storeId": self.identity, + "createdAt": self.meta("store_created_at"), + "dbPath": str(self.path), + "realPath": real_path, + "device": device, + "inode": inode, + "schemaVersion": self.meta("version"), + } + + def write_challenge(self, *, actor: str) -> dict: + """Leave a value only a participant reading THIS file can find.""" + nonce = uuid.uuid4().hex + written_at = _now_iso() + with self.transaction() as db: + db.execute( + "INSERT INTO store_challenge (nonce, written_by, written_at) VALUES (?,?,?)", + (nonce, actor, written_at), + ) + return {"nonce": nonce, "writtenBy": actor, "writtenAt": written_at} + + def read_challenge(self, nonce: str) -> dict: + row = self.one("SELECT * FROM store_challenge WHERE nonce = ?", (nonce,)) + if row is None: + return {"nonce": nonce, "found": False, "writtenBy": None, "writtenAt": None} + return { + "nonce": nonce, "found": True, "writtenBy": row["written_by"], + "writtenAt": row["written_at"], + } + @contextmanager def transaction(self): """BEGIN IMMEDIATE, then commit or roll back. Never a partial record.""" @@ -534,3 +828,196 @@ def all(self, sql: str, params=()): def close(self) -> None: self.db.close() + + +PROVEN, UNPROVEN, MISMATCH = "proven", "unproven", "mismatch" + + +def probe(selection: StateSelection) -> dict: + """Describe the selected state WITHOUT constructing a Store. + + Store.__init__ creates the directory, opens the file O_RDWR, switches on WAL and runs the + schema script. Constructing one in order to find out whether that works fails before it + can report anything, on exactly the host that needed the report. Every step below owns its + error and becomes a field instead, so this function never raises. + """ + directory, db_path = selection.path, selection.db_path + notes = [] + access = { + "directoryExists": False, "directoryReadable": False, "directoryWritable": False, + "dbExists": False, "dbReadable": False, "dbWritable": False, "detail": None, + } + store = { + "exists": False, "storeId": None, "createdAt": None, "dbPath": str(db_path), + "realPath": None, "device": None, "inode": None, "schemaVersion": None, + } + + try: + access["directoryExists"] = directory.is_dir() + except OSError as error: + notes.append(f"directory stat failed: {type(error).__name__}: {error}") + if access["directoryExists"]: + access["directoryReadable"] = os.access(directory, os.R_OK | os.X_OK) + # os.access answers for the REAL uid and can disagree with the kernel under a + # privileged runner or an unusual mount, so writability is measured by writing. + try: + with tempfile.NamedTemporaryFile(dir=directory, prefix=".probe-"): + pass + access["directoryWritable"] = True + except OSError as error: + notes.append(f"directory write failed: {type(error).__name__}: {error}") + + try: + info = os.stat(db_path) + access["dbExists"] = store["exists"] = True + store["realPath"] = str(db_path.resolve()) + store["device"], store["inode"] = info.st_dev, info.st_ino + except OSError as error: + if access["directoryExists"]: + notes.append(f"database stat failed: {type(error).__name__}: {error}") + + if access["dbExists"]: + try: + connection = sqlite3.connect(f"{db_path.as_uri()}?mode=ro", uri=True, timeout=5) + try: + connection.row_factory = sqlite3.Row + access["dbReadable"] = True + for key, field in ( + ("store_id", "storeId"), ("store_created_at", "createdAt"), + ("version", "schemaVersion"), + ): + row = connection.execute( + "SELECT value FROM schema_meta WHERE key = ?", (key,) + ).fetchone() + # A store written before identity existed has no row here. Absence is + # reported as absence and never defaulted, because a default could later + # compare equal to another store's and be read as proof. + store[field] = row["value"] if row is not None else None + finally: + connection.close() + except (OSError, sqlite3.Error, TypeError, ValueError) as error: + notes.append(f"database read failed: {type(error).__name__}: {error}") + + try: + connection = sqlite3.connect( + f"{db_path.as_uri()}?mode=rw", uri=True, timeout=5, isolation_level=None + ) + try: + connection.execute("BEGIN IMMEDIATE") + connection.execute("ROLLBACK") + # Acquiring a write transaction is evidence that this process can write NOW. + # It is not a promise that a later commit succeeds; a full disk still fails. + access["dbWritable"] = True + finally: + connection.close() + except (OSError, sqlite3.Error) as error: + notes.append(f"database write probe failed: {type(error).__name__}: {error}") + + access["detail"] = "; ".join(notes) or None + return {"stateSelection": selection.to_record(), "store": store, "access": access} + + +def read_only_rows(selection: StateSelection, sql: str, params=()) -> dict: + """Answer a question about the store without creating or migrating one. + + Store.__init__ opens the file O_RDWR, switches on WAL and runs the whole schema script, + so any command that reaches for it to READ leaves a fully formed relay database behind. + For diagnosis that is a side effect the command promised not to have: pointing it at an + empty, legacy or unrelated file would silently adopt it. Every error becomes a field. + """ + db_path = selection.db_path + try: + connection = sqlite3.connect(f"{db_path.as_uri()}?mode=ro", uri=True, timeout=5) + except (OSError, sqlite3.Error, ValueError) as error: + return {"readable": False, "rows": [], + "detail": f"{type(error).__name__}: {error}"} + try: + connection.row_factory = sqlite3.Row + rows = [dict(row) for row in connection.execute(sql, params).fetchall()] + except sqlite3.Error as error: + connection.close() + return {"readable": True, "rows": [], "detail": f"{type(error).__name__}: {error}"} + connection.close() + return {"readable": True, "rows": rows, "detail": None} + + +def nonce_lookup(selection: StateSelection, nonce: str) -> dict: + """Look for a challenge nonce read-only, so a comparison never writes to the store.""" + db_path = selection.db_path + try: + connection = sqlite3.connect(f"{db_path.as_uri()}?mode=ro", uri=True, timeout=5) + except (OSError, sqlite3.Error) as error: + return {"nonce": nonce, "found": False, "readable": False, + "detail": f"{type(error).__name__}: {error}"} + try: + connection.row_factory = sqlite3.Row + row = connection.execute( + "SELECT written_by, written_at FROM store_challenge WHERE nonce = ?", (nonce,) + ).fetchone() + except sqlite3.Error as error: + # NOT readable. A locked, malformed or momentarily unavailable database answers no + # question, and calling it readable turns "we could not look" into "it is not there", + # which compare_store then grades as a definite store mismatch. + return {"nonce": nonce, "found": False, "readable": False, + "detail": f"{type(error).__name__}: {error}"} + finally: + connection.close() + if row is None: + return {"nonce": nonce, "found": False, "readable": True, "detail": None} + return {"nonce": nonce, "found": True, "readable": True, "detail": None, + "writtenBy": row["written_by"], "writtenAt": row["written_at"]} + + +def compare_store(store: dict, *, expect_store=None, expect_inode=None, nonce=None) -> dict: + """Grade the evidence that this participant and another share ONE store. + + Conflicting evidence is decided before agreeing evidence, so an easier comparison that + happened to succeed can never talk a mismatch down. Absence is never agreement: a store + that cannot state its identity is unproven, not proven, because the criterion is that a + different database must never be reported as healthy. + """ + reasons = [] + if expect_store is not None: + if store.get("storeId") is None: + reasons.append((UNPROVEN, "this store states no identity, so it cannot be compared")) + elif store["storeId"] != expect_store: + reasons.append((MISMATCH, f"store id {store['storeId']} is not {expect_store}")) + else: + reasons.append((None, "store id matches")) + if expect_inode is not None: + want = str(expect_inode).split(":") + here = (store.get("device"), store.get("inode")) + if len(want) != 2 or None in here: + reasons.append((UNPROVEN, "physical identity is not comparable here")) + elif (str(here[0]), str(here[1])) != (want[0], want[1]): + reasons.append(( + MISMATCH, f"device:inode {here[0]}:{here[1]} is not {expect_inode}", + )) + else: + reasons.append((PROVEN, "same device and inode")) + if nonce is not None: + if nonce.get("readable") is False: + # Not being able to read is not the same as the nonce being absent. Calling it a + # mismatch would tell an operator two participants use different stores when the + # truth is that this one merely could not look. + reasons.append((UNPROVEN, f"the nonce could not be read here: {nonce.get('detail')}")) + elif nonce.get("found"): + reasons.append((PROVEN, "a nonce written by another participant is readable here")) + else: + reasons.append((MISMATCH, "a nonce written by another participant is not here")) + + if not reasons: + return {"sameStore": UNPROVEN, "detail": "no expectation was supplied to compare against"} + for verdict in (MISMATCH, UNPROVEN, PROVEN): + matched = [detail for grade, detail in reasons if grade == verdict] + if matched: + if verdict is PROVEN and any(g == UNPROVEN for g, _ in reasons): + continue + return {"sameStore": verdict, "detail": "; ".join(matched)} + # Every expectation agreed, but none of them was physical or live evidence: an identical + # store id alone is satisfied by a copy of the file, so this is not proof. + return { + "sameStore": UNPROVEN, + "detail": "only the store id was compared, and copying a database copies it too;" + " supply --expect-inode or a nonce for proof", + } diff --git a/packages/codex-session-relay/tests/test_cli.py b/packages/codex-session-relay/tests/test_cli.py index 45eb4d2..b848267 100644 --- a/packages/codex-session-relay/tests/test_cli.py +++ b/packages/codex-session-relay/tests/test_cli.py @@ -1,9 +1,12 @@ """The command surface, driven end to end with no host and no socket.""" +import argparse import json import os +import shutil import subprocess import sys +import tempfile import unittest from .support import CHILD, DISPATCH_TURN, HOST, ISSUE, PARENT, RelayTestCase @@ -177,6 +180,49 @@ def test_a_staged_claim_is_visible_and_not_deliverable(self): self.assertEqual(self.run_cli("status")["deliveries"], []) +class ServiceExitCodes(CliBase): + """A refusal that exits zero is read by automation as a success.""" + + def test_a_refused_enable_does_not_exit_zero(self): + self.run_cli("service", "enable") + state = os.path.join(self.tmp, "daemon.json") + with open(state, "w", encoding="utf-8") as handle: + json.dump({"pid": os.getpid(), "installationId": "someone-else", + "storeId": "another-store", "bootId": None, + "startTicks": None, "workerPid": None}, handle) + # No lock is held here, so this must still succeed: a stopped foreign registration + # is not a reason to make a state directory unconfigurable. + self.assertTrue(self.run_cli("service", "enable")["ok"]) + + def test_a_refused_enable_is_a_refusal_the_shell_can_see(self): + """Returning the payload directly exits zero, and automation reads that as done.""" + from unittest import mock + + from codex_session_relay import cli + + class Refusing: + def enable(self, *, actor): + return {"ok": False, "reason": "not_ours", "intent": {"enabled": False}} + + args = argparse.Namespace(service_command="enable", actor=None) + with mock.patch.object(cli, "_service_for", lambda _services: Refusing()): + with self.assertRaises(cli.PayloadExit) as caught: + cli.cmd_service(object(), args) + self.assertEqual(caught.exception.code, cli.EXIT_REFUSED) + self.assertEqual(caught.exception.payload["reason"], "not_ours") + + def test_a_refused_disable_does_not_exit_zero(self): + self.run_cli("service", "enable") + state = os.path.join(self.tmp, "daemon.json") + with open(state, "w", encoding="utf-8") as handle: + json.dump({"pid": os.getpid(), "installationId": "someone-else", + "storeId": "another-store", "bootId": "irrelevant", + "startTicks": 1, "workerPid": None}, handle) + refused = self.run_cli("service", "disable", expect=2) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "not_ours") + + class SettingsCommands(CliBase): """The registration interface JUN-92 populates from Run's creation result.""" @@ -233,3 +279,276 @@ def test_an_incomplete_record_is_refused_with_a_machine_readable_reason(self): ) self.assertEqual(refused["reason"], "settings_incomplete") self.assertIn("environments", refused["detail"]) + + +class Diagnosis(unittest.TestCase): + """doctor has to answer ON the host it is describing, including a broken one.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-doctor-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.home = os.path.join(self.tmp, "home") + os.makedirs(self.home) + + def cli(self, *args, state=None, socket=None, expect=0, env=None): + environment = dict( + os.environ, PYTHONPATH=os.path.join(REPO, "src"), HOME=self.home, + ) + environment.pop("CODEX_SESSION_RELAY_STATE", None) + environment.pop("XDG_STATE_HOME", None) + environment.update(env or {}) + completed = subprocess.run( + [sys.executable, "-m", "codex_session_relay.cli", + *(["--state", state] if state else []), + *(["--socket", socket] if socket else []), *args], + capture_output=True, text=True, env=environment, timeout=60, + ) + self.assertEqual( + completed.returncode, expect, + f"exit {completed.returncode}: {completed.stdout}{completed.stderr}", + ) + return json.loads(completed.stdout) + + def test_doctor_answers_for_a_state_directory_that_does_not_exist_yet(self): + absent = os.path.join(self.tmp, "absent") + report = self.cli("doctor", state=absent) + self.assertEqual(report["stateSelection"]["source"], "flag") + self.assertFalse(report["access"]["directoryExists"]) + self.assertFalse(report["contents"]["available"]) + # The old doctor built a Store first, which created the directory it was asked about. + self.assertFalse(os.path.exists(absent)) + + def test_doctor_does_not_turn_an_unrelated_file_into_a_relay_database(self): + """The directory existing was not the whole side effect; counting rows was too. + + A readable relay.sqlite3 sent the contents block through a real Store, and + Store.__init__ opens O_RDWR, switches on WAL and runs the entire schema script. An + empty, legacy or unrelated file was quietly adopted by the command that promised to + do nothing but look. + """ + state = os.path.join(self.tmp, "borrowed") + os.makedirs(state) + target = os.path.join(state, "relay.sqlite3") + open(target, "w").close() + + report = self.cli("doctor", state=state) + + self.assertEqual(os.path.getsize(target), 0, "doctor wrote a schema into it") + self.assertEqual( + sorted(os.listdir(state)), ["relay.sqlite3"], "no WAL or shm sidecar either", + ) + self.assertTrue(report["access"]["dbExists"]) + self.assertFalse(report["contents"]["available"], + "and it says so rather than inventing counts") + self.assertIsNotNone(report["contents"]["detail"]) + + def test_doctor_names_the_rule_that_chose_the_directory(self): + chosen = os.path.join(self.tmp, "chosen") + by_env = self.cli("doctor", env={"CODEX_SESSION_RELAY_STATE": chosen}) + self.assertEqual(by_env["stateSelection"]["source"], "env") + self.assertEqual(by_env["stateSelection"]["path"], chosen) + by_flag = self.cli("doctor", state=chosen, env={ + "CODEX_SESSION_RELAY_STATE": os.path.join(self.tmp, "ignored"), + }) + self.assertEqual(by_flag["stateSelection"]["source"], "flag") + self.assertEqual(by_flag["stateSelection"]["path"], chosen) + + def test_a_different_store_is_refused_rather_than_reported_healthy(self): + a, b = os.path.join(self.tmp, "a"), os.path.join(self.tmp, "b") + mine = self.cli("store-identity", state=a)["store"] + theirs = self.cli("store-identity", state=b)["store"] + self.assertNotEqual(mine["storeId"], theirs["storeId"]) + refused = self.cli( + "doctor", "--expect-store", mine["storeId"], state=b, expect=2, + ) + self.assertEqual(refused["sameStore"], "mismatch") + # The whole diagnosis survives the refusal; it is not replaced by an error envelope. + self.assertIn("stateSelection", refused) + self.assertIn("access", refused) + + def test_a_nonce_proves_one_store_and_disproves_a_copy(self): + a, b = os.path.join(self.tmp, "a"), os.path.join(self.tmp, "b") + mine = self.cli("store-identity", state=a)["store"] + os.makedirs(b, exist_ok=True) + for suffix in ("", "-wal", "-shm"): + source = os.path.join(a, f"relay.sqlite3{suffix}") + if os.path.exists(source): + shutil.copy(source, os.path.join(b, f"relay.sqlite3{suffix}")) + nonce = self.cli("store-challenge", "--write", "--actor", "parent", state=a)["nonce"] + proven = self.cli( + "doctor", "--expect-store", mine["storeId"], "--expect-nonce", nonce, state=a, + ) + self.assertEqual(proven["sameStore"], "proven") + copied = self.cli( + "doctor", "--expect-store", mine["storeId"], "--expect-nonce", nonce, state=b, + expect=2, + ) + self.assertEqual(copied["sameStore"], "mismatch") + # Identifier alone cannot separate them, which is why it is graded unproven. + weak = self.cli("doctor", "--expect-store", mine["storeId"], state=b, expect=2) + self.assertEqual(weak["sameStore"], "unproven") + + def test_doctor_reports_that_state_and_the_transport_ledger_have_split(self): + state = os.path.join(self.tmp, "state") + socket = os.path.join(self.tmp, "app.sock") + split = self.cli("doctor", state=state, socket=socket) + self.assertTrue(split["ledger"]["configured"]) + # --state moved the store; the adapter resolves its ledger from the environment. + self.assertTrue(split["ledger"]["split"]) + together = self.cli( + "doctor", state=state, socket=socket, + env={"CODEX_SESSION_RELAY_STATE": state}, + ) + self.assertFalse(together["ledger"]["split"]) + + +class LazyServices(unittest.TestCase): + """Building dependencies on demand must not drop the wiring __init__ used to do.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-lazy-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + + def services(self): + from codex_session_relay.cli import Services + + built = Services(argparse.Namespace(state=self.tmp, socket=None)) + self.addCleanup(built.close) + return built + + def test_the_first_ack_is_already_wired_to_the_outbox(self): + built = self.services() + # Touching ack FIRST is the regression: record_verdict skips its outbox obligation + # when sync is absent, so an unwired ack would lose it silently. + self.assertIsNotNone(built.ack.sync) + self.assertIs(built.ack.sync, built.sync) + + def test_every_dependency_shares_one_store_and_is_built_once(self): + built = self.services() + self.assertIs(built.registry.store, built.store) + self.assertIs(built.delivery.store, built.store) + self.assertIs(built.ack.store, built.store) + self.assertIs(built.registry, built.registry) + self.assertIs(built.delivery, built.delivery) + + def test_closing_without_ever_using_the_store_creates_nothing(self): + from codex_session_relay.cli import Services + + empty = os.path.join(self.tmp, "untouched") + built = Services(argparse.Namespace(state=empty, socket=None)) + built.close() + self.assertFalse(os.path.exists(empty)) + + +class ContestedSocket(CliBase): + """Two stores recording one socket must not quietly become three. + + These runs deliberately pass no --state: the whole question is what the environment alone + resolves to, and an explicit directory answers it before discovery ever runs. + """ + + def contested(self, name): + """A home holding two stores that both record one socket.""" + from pathlib import Path + + from codex_session_relay.store import Store + + home = os.path.join(self.tmp, name) + root = os.path.join(home, ".local", "state", "codex-session-relay") + socket = os.path.join(self.tmp, f"{name}.sock") + for directory in ("aaaa444444444444", "bbbb444444444444"): + os.makedirs(os.path.join(root, directory)) + Store(Path(root) / directory / "relay.sqlite3", socket_path=socket).close() + return home, root, socket + + def run_in_home(self, home, *args, expect=0): + environment = dict(os.environ, PYTHONPATH=os.path.join(REPO, "src"), HOME=home) + for name in ("CODEX_SESSION_RELAY_STATE", "XDG_STATE_HOME"): + environment.pop(name, None) + completed = subprocess.run( + [sys.executable, "-m", "codex_session_relay.cli", *args], + capture_output=True, text=True, env=environment, timeout=60, + ) + self.assertEqual( + completed.returncode, expect, + f"exit {completed.returncode}: {completed.stdout}{completed.stderr}", + ) + return json.loads(completed.stdout) + + def test_an_ordinary_command_refuses_rather_than_creating_a_third_store(self): + home, root, socket = self.contested("contested-status") + + refused = self.run_in_home(home, "--socket", socket, "status", expect=2) + + self.assertEqual(refused["reason"], "ambiguous_state_directory") + self.assertEqual(len(refused["candidates"]), 2) + self.assertFalse( + os.path.exists(refused["wouldHaveCreated"]), + "the refusal must not leave behind the store it refused to choose", + ) + self.assertEqual(sorted(os.listdir(root)), ["aaaa444444444444", "bbbb444444444444"]) + + def test_doctor_still_describes_a_contested_socket(self): + home, _root, socket = self.contested("contested-doctor") + + report = self.run_in_home(home, "--socket", socket, "doctor") + + self.assertTrue(report["siblingStores"]["ambiguous"]) + self.assertEqual(len(report["siblingStores"]["claimingThisSocket"]), 2) + + def test_a_state_directory_recording_another_socket_is_refused(self): + """An explicit directory reused with a different App Server. + + Choosing a directory is not choosing what is already in it: the service would claim + and serve the new socket while the database went on attributing itself to the old + one, so one installation's assignments could be exposed through another and later + discovery would still match the store to the socket it no longer serves. + """ + from pathlib import Path + + from codex_session_relay.store import Store + + state = os.path.join(self.tmp, "reused-state") + first = os.path.join(self.tmp, "first.sock") + second = os.path.join(self.tmp, "second.sock") + Store(Path(state) / "relay.sqlite3", socket_path=first).close() + + environment = dict(os.environ, PYTHONPATH=os.path.join(REPO, "src")) + completed = subprocess.run( + [sys.executable, "-m", "codex_session_relay.cli", "--state", state, + "--socket", second, "status"], + capture_output=True, text=True, env=environment, timeout=60, + ) + + self.assertEqual(completed.returncode, 2, completed.stdout + completed.stderr) + refused = json.loads(completed.stdout) + self.assertEqual(refused["reason"], "state_directory_serves_another_socket") + self.assertEqual(refused["recordedSocket"], first) + + def test_a_store_recording_no_socket_also_refuses_before_creating_one(self): + """A store older than provenance cannot be matched to a socket by anything but its + directory hash, which cannot be inverted. Reporting it through doctor was not enough, + because an ordinary command does not run doctor and creates the store anyway.""" + from pathlib import Path + + from codex_session_relay.store import Store + + home = os.path.join(self.tmp, "unlabelled-home") + root = os.path.join(home, ".local", "state", "codex-session-relay") + os.makedirs(os.path.join(root, "0123456789abcdef")) + Store(Path(root) / "0123456789abcdef" / "relay.sqlite3").close() + socket = os.path.join(self.tmp, "unlabelled.sock") + + refused = self.run_in_home(home, "--socket", socket, "status", expect=2) + + self.assertEqual(refused["reason"], "unidentified_state_directory") + self.assertFalse(os.path.exists(refused["wouldHaveCreated"])) + self.assertEqual(os.listdir(root), ["0123456789abcdef"], "no store was created") + + def test_an_explicit_state_directory_resolves_the_contest(self): + home, root, socket = self.contested("contested-explicit") + chosen = os.path.join(root, "aaaa444444444444") + + answer = self.run_in_home(home, "--state", chosen, "--socket", socket, "status") + + self.assertEqual(answer["deliveries"], []) diff --git a/packages/codex-session-relay/tests/test_daemon_cadence.py b/packages/codex-session-relay/tests/test_daemon_cadence.py index 648aece..6da997c 100644 --- a/packages/codex-session-relay/tests/test_daemon_cadence.py +++ b/packages/codex-session-relay/tests/test_daemon_cadence.py @@ -5,6 +5,8 @@ deadline-only run busy-spun for its whole duration. These tests pin both halves. """ +import os +import shutil import tempfile import time import unittest @@ -21,6 +23,10 @@ def __init__(self, state, socket=None, max_ticks=None, deadline=None): self.socket = socket self.max_ticks = max_ticks self.deadline = deadline + # A bounded daemon run now takes the same scope claim a managed service does, so a + # test has to say which registry it is claiming in. Without this it would write an + # ownership record under the real home. + self.allow_isolated_scope = True class SchedulerWait(unittest.TestCase): @@ -50,6 +56,16 @@ class CliSuppliesTheCadence(unittest.TestCase): def _services(self, **kwargs): directory = tempfile.mkdtemp(prefix="relay-cadence-") + self.addCleanup(shutil.rmtree, directory, ignore_errors=True) + scopes = tempfile.mkdtemp(prefix="relay-cadence-scopes-") + self.addCleanup(shutil.rmtree, scopes, ignore_errors=True) + previous = os.environ.get("CODEX_SESSION_RELAY_SCOPE_DIR") + os.environ["CODEX_SESSION_RELAY_SCOPE_DIR"] = scopes + self.addCleanup( + lambda: os.environ.__setitem__("CODEX_SESSION_RELAY_SCOPE_DIR", previous) + if previous is not None + else os.environ.pop("CODEX_SESSION_RELAY_SCOPE_DIR", None) + ) args = _Args(directory, socket="/nonexistent-for-this-test", **kwargs) services = Services(args) # The lazy adapter returns whatever is already set, so no bridge is built and no socket diff --git a/packages/codex-session-relay/tests/test_delivery.py b/packages/codex-session-relay/tests/test_delivery.py index 69024da..7e7ff6d 100644 --- a/packages/codex-session-relay/tests/test_delivery.py +++ b/packages/codex-session-relay/tests/test_delivery.py @@ -1,8 +1,9 @@ """JUN-91 delivery: busy handling, dispatch, bounds, and honest reporting.""" +import os import unittest -from codex_session_relay.delivery import COMPLETION, REVISION +from codex_session_relay.delivery import COMPLETION, REVISION, DeliveryService from codex_session_relay.errors import RefusalReason from codex_session_relay.lifecycle import ARCHIVED, BUDGET_LIMITED, CANNOT_ACCEPT, PAUSED, UNKNOWN from codex_session_relay.transport import ( @@ -14,7 +15,7 @@ WITHHELD_PRE_SEND, ) -from .support import CHILD, PARENT, DeliveryTestCase +from .support import CHILD, HOST, PARENT, DeliveryTestCase class Queueing(DeliveryTestCase): @@ -583,3 +584,168 @@ def test_the_instruction_each_side_is_given_is_the_one_that_actually_works(self) # What the message DOES tell the child to do is available: emit under generation 2. relationship = self.registry.get(self._rid) self.assertEqual(relationship["executionGeneration"], 2) + + +class CrossAssignmentDelivery(DeliveryTestCase): + """One shared service carries several assignments; none may answer for another.""" + + def other_assignment(self): + """A second project whose parent is ALSO an authorized recipient of the first.""" + from codex_session_relay.models import Endpoint + + other_root = os.path.join(self.tmp, "other-project") + os.makedirs(other_root, exist_ok=True) + return self.registry.register( + parent=Endpoint("01other-parent", HOST, cwd="/other", cxc_session="cxc-other"), + child=Endpoint("01other-child", HOST, cwd=other_root, cxc_session="cxc-other-c"), + issue_key="REL-2", + artifact_roots=[other_root], + allowed_recipients=["01other-parent"], + dispatch_request_id="dispatch-2", + dispatch_turn_id="turn-dispatch-2", + ) + + def test_a_completion_may_not_be_addressed_to_another_assignments_parent(self): + other = self.other_assignment() + # The first assignment authorizes the other project's parent as a recipient, which is + # legitimate. Membership alone must still not make it a valid destination. + relationship, event_id = self.ready_event( + recipients=[PARENT, other["parent"]["taskId"]], + ) + refused = self.assertRefused( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, + self.delivery.enqueue, event_id, + recipient_task_id=other["parent"]["taskId"], + ) + self.assertIn("its own parent", refused.detail) + self.assertIsNone(self.delivery.find(event_id)) + + def test_a_tampered_delivery_row_is_refused_before_any_transport_call(self): + other = self.other_assignment() + relationship, event_id = self.queued_event( + recipients=[PARENT, other["parent"]["taskId"]], + ) + self.adapter.add_thread(other["parent"]["taskId"]) + with self.store.transaction() as db: + db.execute( + "UPDATE deliveries SET recipient_task_id = ?, recipient_thread_id = ?" + " WHERE event_id = ?", + (other["parent"]["taskId"], other["parent"]["taskId"], event_id), + ) + self.assertRefused( + RefusalReason.RECIPIENT_NOT_AUTHORIZED, self.attempt, event_id, + ) + self.assertEqual(self.adapter.sends, [], "nothing may reach the host") + + def test_the_ordinary_completion_still_goes_to_its_own_parent(self): + self.other_assignment() + relationship, event_id = self.queued_event() + record = self.attempt(event_id) + self.assertEqual(record["deliveryState"], "dispatched") + self.assertEqual(self.delivery.get(event_id)["recipient_task_id"], PARENT) + + def test_projects_are_distinguishable_for_a_shared_service(self): + from codex_session_relay.registry import project_key + + other = self.other_assignment() + mine = self.register(issue_key="REL-3", dispatch_request_id="dispatch-3") + self.assertNotEqual(project_key(mine), project_key(other)) + self.assertEqual(project_key(other), "/other") + + +class RestartPreservation(DeliveryTestCase): + """Everything durable survives a process boundary, and nothing is sent twice for it.""" + + def reopen(self): + """Close the store and open it again: the process boundary, minus the process.""" + from codex_session_relay.ack import AckService + from codex_session_relay.receipts import ReceiptIntake + from codex_session_relay.reconcile import Reconciler + from codex_session_relay.registry import Registry + from codex_session_relay.store import Store + from codex_session_relay.sync import SyncOutbox + + path = self.store.path + self.store.close() + self.store = Store(path) + self.addCleanup(self.store.close) + self.registry = Registry(self.store, self.clock) + self.intake = ReceiptIntake(self.store, self.registry, self.clock) + self.delivery = DeliveryService(self.store, self.registry, self.intake, self.clock) + self.ack = AckService( + self.store, self.registry, self.intake, self.delivery, self.clock, + ) + self.reconciler = Reconciler(self.store, self.registry, self.delivery, self.clock) + self.sync = SyncOutbox(self.store, self.clock) + + def snapshot(self, table, columns): + return [ + tuple(row[column] for column in columns) + for row in self.store.all(f"SELECT * FROM {table} ORDER BY rowid") + ] + + def test_a_restart_keeps_every_durable_record_and_resends_nothing(self): + relationship, dispatched_event = self.queued_event() + rid = relationship["relationshipId"] + record = self.attempt(dispatched_event) + self.assertEqual(record["deliveryState"], DISPATCHED) + + # A second event left genuinely unresolved: a settled attempt never enters + # open_attempts, so without this the recovery assertion would prove nothing. + path = self.artifact("second.txt", "still in flight") + payload = self.ready_payload(relationship, [path], attempt=2) + self.accept(payload) + in_flight = payload["eventId"] + self.delivery.enqueue(in_flight) + self.adapter.script("transport_unknown") + # Past the minimum send interval, or the claim never happens and there is no + # unresolved attempt for recovery to find. + later = self.clock.now() + 3600 + self.attempt(in_flight, now=later) + unresolved = self.store.one( + "SELECT request_id FROM attempts WHERE event_id = ? AND state = ?", + (in_flight, HELD_UNCERTAIN), + )["request_id"] + + from codex_session_relay.sync import SyncOutbox + + SyncOutbox(self.store, self.clock).set_target(rid, "coordination_document", "DOC-1") + before = { + "relationships": self.snapshot("relationships", ("relationship_id", "status", + "execution_generation")), + "generations": self.snapshot("generations", ("relationship_id", + "execution_generation", + "anchor_state", "dispatch_turn_id")), + "events": self.snapshot("events", ("event_id", "stage", "revision_hash")), + "deliveries": self.snapshot("deliveries", ("event_id", "state", "attempt_count")), + "sync_targets": self.snapshot("sync_targets", ("relationship_id", "target_ref")), + } + sends_before = len(self.adapter.sends) + + self.reopen() + + for table, columns in ( + ("relationships", ("relationship_id", "status", "execution_generation")), + ("generations", ("relationship_id", "execution_generation", "anchor_state", + "dispatch_turn_id")), + ("events", ("event_id", "stage", "revision_hash")), + ("deliveries", ("event_id", "state", "attempt_count")), + ("sync_targets", ("relationship_id", "target_ref")), + ): + self.assertEqual(self.snapshot(table, columns), before[table], table) + + # Recovery establishes what happened. It sends nothing. + outcome = self.reconciler.recover_on_start(self.adapter) + self.assertIn(unresolved, [entry["requestId"] for entry in outcome["reconciled"]]) + self.assertEqual(outcome["resent"], []) + self.assertIn(dispatched_event, outcome["awaitingAck"]) + self.assertEqual(len(self.adapter.sends), sends_before, "recovery must not send") + + # And a replay of the already dispatched event is refused by the claim itself, + # rather than merely being left out of the schedule. + attempts_before = self.delivery.get(dispatched_event)["attempt_count"] + self.assertIsNone(self.attempt(dispatched_event, now=later + 3600)) + self.assertEqual( + self.delivery.get(dispatched_event)["attempt_count"], attempts_before, + ) + self.assertEqual(len(self.adapter.sends), sends_before) diff --git a/packages/codex-session-relay/tests/test_service.py b/packages/codex-session-relay/tests/test_service.py new file mode 100644 index 0000000..9106c8f --- /dev/null +++ b/packages/codex-session-relay/tests/test_service.py @@ -0,0 +1,1838 @@ +"""Who owns the daemon, and who is allowed to stop it. + +These tests use real child processes on purpose. A mocked flock proves that the code called +flock; only a second process trying to start proves that the first one is actually excluded. +""" + +import contextlib +import errno +import json +import os +import shutil +import subprocess +import sys +import tempfile +import time +import unittest +from pathlib import Path +from unittest import mock + +from codex_session_relay import service as service_module +from codex_session_relay.policy import RetryPolicy +from codex_session_relay.service import ( + ISOLATED, PRODUCTION, ProcessHandle, RelayService, ScopeRegistry, ServiceRefused, + installation_id, owned_service, production_scope_root, resolve_scope_root, +) +from codex_session_relay.store import Store, resolve_state_dir + +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +SOCKET = "/nonexistent-app-server.sock" + +# A child that does nothing but stay alive, for tests that only need a live pid. +IDLE_CHILD = "import time\nwhile True:\n time.sleep(0.05)\n" + +# A child that takes the claim and holds it until told to stop, so exclusion is observed +# rather than asserted against a double. +HOLDER = """ +import os, sys, time +sys.path.insert(0, {src!r}) +from codex_session_relay.service import RelayService, ScopeRegistry, owned_service +from codex_session_relay.store import resolve_state_dir +service = RelayService( + resolve_state_dir({state!r}), socket_path={socket!r}, + scope=ScopeRegistry(__import__("pathlib").Path({scopes!r}), "isolated"), + store_id={store_id!r}, +) +service.launch_id = {launch!r} +with owned_service(service, allow_isolated=True, require_intent=False) as record: + # What supervise() writes once on_start has returned: this child is serving. + service._note(readyAt="2026-01-01T00:00:00Z") + # The record file IS the handshake. A pipe would add buffering and lifetime questions + # that have nothing to do with what this test is about. + while True: + time.sleep(0.05) +""" + +# A child parked in the window supervision leaves behind on its way out: the launch is +# recorded, the pid has been cleared, and the daemon lock is not released until the context +# exits. The record is written once, already cleared, so the parent cannot observe a live +# pid first and pass for the wrong reason. +FINISHING = """ +import os, sys, time +sys.path.insert(0, {src!r}) +from codex_session_relay.daemon import SingleInstance +from codex_session_relay.service import RelayService, ScopeRegistry +from codex_session_relay.store import resolve_state_dir +service = RelayService( + resolve_state_dir({state!r}), socket_path={socket!r}, + scope=ScopeRegistry(__import__("pathlib").Path({scopes!r}), "isolated"), + store_id={store_id!r}, +) +service.launch_id = {launch!r} +with SingleInstance(service.selection.path): + service.write_record(dict( + service.new_record(pid=os.getpid()), + readyAt="2026-01-01T00:00:00Z", pid=None, workerPid=None, stoppedAt="cleanup", + )) + while True: + time.sleep(0.05) +""" + + +class _Captured(Exception): + """Stops a start before it launches anything.""" + + +class ServiceTestCase(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-service-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.scopes = os.path.join(self.tmp, "scopes") + self.children = [] + self.addCleanup(self._reap) + + def _reap(self): + for child in self.children: + if child.poll() is None: + child.kill() + try: + child.wait(timeout=10) + except subprocess.TimeoutExpired: # pragma: no cover - a hung child is a failure + self.fail(f"child {child.pid} did not exit") + + def service(self, name="a", *, socket=SOCKET, scopes=None): + state = os.path.join(self.tmp, name) + os.makedirs(state, exist_ok=True) + store = Store(Path(state) / "relay.sqlite3") + store_id = store.identity + store.close() + return RelayService( + resolve_state_dir(state), socket_path=socket, + scope=ScopeRegistry(Path(scopes or self.scopes), ISOLATED), store_id=store_id, + ) + + def holder(self, service, *, launch=None): + """Start a child that really holds the lock and the claim, and wait until it does.""" + program = HOLDER.format( + src=os.path.join(REPO, "src"), state=str(service.selection.path), + socket=service.socket_path, scopes=str(service.scope.root), + store_id=service.store_id, launch=launch, + ) + child = subprocess.Popen( + [sys.executable, "-c", program], stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, text=True, + ) + self.children.append(child) + deadline = time.monotonic() + 20.0 + while time.monotonic() < deadline: + record = service.record() + if record and record.get("pid") and service.lock_is_held(): + return child, record["pid"] + if child.poll() is not None: + self.fail(f"holder exited {child.returncode}: {child.stderr.read()}") + time.sleep(0.02) + self.fail("the holder never took the claim") + + +class Ownership(ServiceTestCase): + def test_a_second_start_is_refused_and_the_first_is_untouched(self): + service = self.service("a") + child, pid = self.holder(service) + service.intent.write(enabled=True, actor="test") + refused = service.start(allow_isolated=True, launcher=self._never_launch) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "already_running") + self.assertIsNone(child.poll(), "the running service must not be disturbed") + + def _never_launch(self, *_args, **_kwargs): # pragma: no cover - reaching it is the failure + self.fail("start launched a second process while one was already running") + + def test_a_different_state_directory_cannot_serve_the_same_socket(self): + """The hole a per-state-directory lock cannot close.""" + first = self.service("a") + self.holder(first) + second = self.service("b") + self.assertNotEqual(second.store_id, first.store_id) + second.intent.write(enabled=True, actor="test") + refused = second.start(allow_isolated=True, launcher=self._never_launch) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "scope_owned_by_other_store") + self.assertEqual(refused["conflicts"][0]["storeId"], first.store_id) + + def test_a_stopped_registration_on_another_store_is_still_a_conflict(self): + """Live-only scanning would miss this one entirely.""" + first = self.service("a") + child, _pid = self.holder(first) + child.terminate() + child.wait(timeout=10) + second = self.service("b") + conflicts = second.conflicts() + self.assertEqual([c["storeId"] for c in conflicts], [first.store_id]) + self.assertFalse(conflicts[0]["live"]) + + def test_a_stopped_registration_is_not_silently_overwritten(self): + """Overwriting it would erase the only evidence two stores served one socket.""" + first = self.service("a") + child, _pid = self.holder(first) + child.terminate() + child.wait(timeout=10) + second = self.service("b") + refused = second.scope.claim( + second.socket_path, second.new_record(pid=os.getpid()), + ) + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "scope_registered_to_other_store") + self.assertEqual(refused["held_by"]["storeId"], first.store_id) + + def test_a_definite_mismatch_beats_a_missing_start_time(self): + """Installation identity already proves foreign; unverifiable would discard that.""" + service = self.service("a") + child, _pid = self.holder(service) + record = service.record() + service.write_record( + dict(record, startTicks=None, installationId="someone-else"), + ) + refused = service.stop() + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("another installation", refused["detail"]) + self.assertIsNone(child.poll()) + + def test_stop_refuses_a_process_another_installation_owns(self): + service = self.service("a") + child, pid = self.holder(service) + record = service.record() + service.write_record(dict(record, installationId="someone-else")) + refused = service.stop() + self.assertFalse(refused["ok"]) + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("another installation", refused["detail"]) + self.assertIsNone(child.poll(), "a foreign process must not be signalled") + + def test_disable_refuses_a_foreign_service_without_touching_shared_intent(self): + """service.json is shared by everything pointed at this state directory. + + Writing enabled=false and only then discovering the supervisor is foreign refused + the stop while still shutting that supervisor down at its next worker boundary, + because it re-reads intent there. A refusal that still has an effect is not one. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), installationId="someone-else")) + + refused = service.disable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertTrue( + service.intent.read()["enabled"], + "the owner's intent must survive a refused disable", + ) + self.assertIsNone(child.poll()) + self.assertFalse(service.stop_request_path.exists()) + + def test_a_record_with_no_boot_id_is_unverifiable(self): + """A reboot resets the pid space AND the start-tick counter together. + + Without a recorded boot, a matching pid with a matching tick count proves nothing: + an unrelated process holding the old number passes both checks and would be signalled. + """ + service = self.service("a") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), bootId=None)) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIn("boot id", refused["detail"]) + self.assertIsNone(child.poll(), "refusing is the point") + + def test_enable_refuses_to_reverse_a_foreign_owners_intent(self): + """service.json is shared, and disable already asks this question. + + An owner who has just disabled a service whose supervisor is still exiting must not + have that reversed by another installation, leaving it eligible to restart. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.intent.write(enabled=False, actor="owner") + service.write_record(dict(service.record(), installationId="someone-else")) + + refused = service.enable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertFalse( + service.intent.read()["enabled"], + "the owner's disable must not be reversed by another installation", + ) + self.assertIsNone(child.poll()) + + def test_enable_is_refused_while_a_foreign_supervisor_is_shutting_down(self): + """A supervisor clears its pids BEFORE releasing the lock. + + ownership answers none once both are absent, which is exactly the interval in which + a foreign shutdown could have its owner's disable reversed. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.intent.write(enabled=False, actor="owner") + service.write_record(dict( + service.record(), pid=None, workerPid=None, installationId="someone-else", + )) + self.assertTrue(service.lock_is_held(), "the fixture needs the lock still held") + + refused = service.enable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertFalse(service.intent.read()["enabled"]) + self.assertIsNone(child.poll()) + + def test_a_host_with_no_boot_id_can_still_stop_its_own_service(self): + """Refusing every record without one traded a narrow risk for a certain failure. + + Where no boot id is available at all, every record lacks one - including the record + this installation just wrote - so a blanket refusal makes the service unstoppable by + its own owner. + """ + service = self.service("a") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), bootId=None)) + + with mock.patch.object(service_module, "boot_id", lambda: None): + stopped = service.stop() + + self.assertTrue(stopped["ok"], stopped) + child.wait(timeout=10) + + def test_an_orphan_worker_with_no_boot_id_is_unverifiable_too(self): + """_stop_worker checks start ticks only, and a reboot resets those with the pids. + + The live-supervisor path already refused this; the orphan branch did not, so an + unrelated process holding the old worker number would have been signalled. + """ + service = self.service("a") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, bootId=None, workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIsNone(child.poll(), "an unidentifiable orphan must not be signalled") + + def test_disable_is_refused_while_a_foreign_supervisor_is_shutting_down(self): + """The same cleanup window enable guards, in the command that writes the same file.""" + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, workerPid=None, installationId="someone-else", + )) + self.assertTrue(service.lock_is_held(), "the fixture needs the lock still held") + + refused = service.disable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertTrue( + service.intent.read()["enabled"], + "a foreign owner's service must not be left disabled by a refused command", + ) + self.assertIsNone(child.poll()) + + def test_the_takeover_flag_reaches_the_process_that_claims_the_scope(self): + """It was set on the parent object and dropped at the process boundary. + + The supervisor is the process that claims the scope, so a flag the parent keeps to + itself leaves the takeover silently doing nothing. + """ + service = self.service("a") + service.enable(actor="test") + + self.assertNotIn("--takeover-scope", self._launch_argv(service)) + + service.takeover = True + + self.assertIn("--takeover-scope", self._launch_argv(service)) + + def _launch_argv(self, service): + """The argv default_launcher would build, without starting anything.""" + import subprocess + from unittest import mock + + captured = {} + + class Fake: + pid = os.getpid() + returncode = None + + def poll(self): + return None + + def popen(argv, **_kwargs): + captured["argv"] = argv + return Fake() + + with mock.patch.object(subprocess, "Popen", popen): + service.default_launcher(service, allow_isolated=True) + return captured["argv"] + + def test_a_stopped_registration_can_be_taken_over_deliberately(self): + """Otherwise a replaced database blocks its socket forever. + + Recovery was finding and deleting an internal registry file by hand, which is not an + operation anyone should have to discover. + """ + first = self.service("a") + child, _pid = self.holder(first) + child.terminate() + child.wait(timeout=10) + second = self.service("b") + self.assertNotEqual(second.store_id, first.store_id) + + refused = second.scope.claim(second.socket_path, second.new_record(pid=os.getpid())) + self.assertEqual(refused["reason"], "scope_registered_to_other_store") + second.scope.release(second.socket_path) + + second.takeover = True + taken = second.scope.claim(second.socket_path, second.new_record(pid=os.getpid())) + + self.assertTrue(taken["ok"], taken) + self.assertEqual(second.scope.read(second.socket_path)["storeId"], second.store_id) + second.scope.release(second.socket_path) + + def test_a_takeover_cannot_displace_a_live_owner(self): + """It only ever replaces a registration nothing is running behind.""" + first = self.service("a") + self.holder(first) + second = self.service("b") + second.intent.write(enabled=True, actor="test") + + refused = second.start( + allow_isolated=True, launcher=self._never_launch_here, takeover=True, + ) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "scope_owned_by_other_store") + + def _never_launch_here(self, *_a, **_k): # pragma: no cover - reaching it is the failure + self.fail("a takeover must not launch past a live owner") + + def test_disable_decides_and_writes_under_the_lock(self): + """A foreign supervisor starting between the check and the write read enabled=true. + + It took the lock, then saw the intent we changed afterwards and exited at its next + boundary - a refused disable that still stopped it. The classification and the write + happen under the lock now, so no supervisor can start between them. + """ + service = self.service("a") + service.enable(actor="owner") + child, _pid = self.holder(service) + service.write_record(dict(service.record(), installationId="someone-else")) + + refused = service.disable(actor="intruder") + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertTrue( + service.intent.read()["enabled"], + "a refused disable must not change the intent a foreign supervisor reads", + ) + self.assertIsNone(child.poll()) + + def test_disable_takes_the_lock_before_touching_shared_intent(self): + """The ordering itself, so a later edit cannot quietly put the write back first.""" + service = self.service("b") + service.enable(actor="owner") + order = [] + original_lock = service.daemon_lock_if_free + original_write = service.intent.write + + @contextlib.contextmanager + def watched_lock(): + order.append("lock") + with original_lock() as held: + yield held + + def watched_write(**kwargs): + order.append("write") + return original_write(**kwargs) + + service.daemon_lock_if_free = watched_lock + service.intent.write = watched_write + try: + service.disable(actor="owner") + finally: + service.daemon_lock_if_free = original_lock + service.intent.write = original_write + + self.assertEqual(order[:2], ["lock", "write"]) + self.assertFalse(service.intent.read()["enabled"]) + + def test_enable_still_works_when_nothing_is_running_here(self): + """A stopped registration is not a reason to make a state directory unusable.""" + service = self.service("b") + service.write_record(dict( + service.new_record(pid=os.getpid()), installationId="someone-else", + )) + + enabled = service.enable(actor="owner") + + self.assertTrue(enabled["ok"], enabled) + self.assertTrue(service.intent.read()["enabled"]) + + def test_disable_still_works_on_a_service_this_installation_owns(self): + service = self.service("b") + service.enable(actor="owner") + child, _pid = self.holder(service) + + disabled = service.disable(actor="owner") + + self.assertTrue(disabled["ok"], disabled) + self.assertFalse(service.intent.read()["enabled"]) + child.wait(timeout=10) + + def test_enable_takes_the_lock_before_touching_shared_intent(self): + """The ordering itself, so a later edit cannot quietly put the write back first. + + Classifying and then writing are two operations. A foreign supervisor that passes its + own enabled-intent check and takes the lock in between would have enabled=true written + over a disable that happened during the handoff, leaving it running and eligible to + restart. disable already decides under the lock; enable did not. + """ + service = self.service("f") + order = [] + original_lock = service.daemon_lock_if_free + original_write = service.intent.write + + @contextlib.contextmanager + def watched_lock(): + order.append("lock") + with original_lock() as held: + yield held + + def watched_write(**kwargs): + order.append("write") + return original_write(**kwargs) + + service.daemon_lock_if_free = watched_lock + service.intent.write = watched_write + try: + service.enable(actor="owner") + finally: + service.daemon_lock_if_free = original_lock + service.intent.write = original_write + + self.assertEqual(order[:2], ["lock", "write"]) + self.assertTrue(service.intent.read()["enabled"]) + + def test_enable_refuses_a_lock_holder_nothing_here_can_identify(self): + """The same hole disable had: a supervisor that has the lock and has published + nothing is not a supervisor whose owner asked for this.""" + import fcntl + + service = self.service("g") + service.disable(actor="owner") + service.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(service.selection.path / service_module.DAEMON_LOCK, "a+") + self.addCleanup(handle.close) + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + refused = service.enable(actor="owner") + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertFalse( + service.intent.read()["enabled"], + "a refused enable must not reverse the owner's disable", + ) + + def test_an_interrupted_start_does_not_leave_its_child_running(self): + """The default launcher puts the supervisor in its own session, so a terminal signal + never reaches it. Anything that leaves start() without a confirmed result - an + exception, a Ctrl-C while recovery is still initialising - has to stop it, or it comes + up afterwards holding both locks for a launch the caller was told nothing about.""" + service = self.service("h") + service.enable(actor="owner") + + class Interrupted: + returncode = None + + def __init__(self): + self.stopped = [] + self.polls = 0 + + def poll(self): + self.polls += 1 + if self.polls == 1: + raise KeyboardInterrupt + return self.returncode + + def terminate(self): + self.stopped.append("terminate") + self.returncode = -15 + + def kill(self): # pragma: no cover - terminate already settled it + self.stopped.append("kill") + + def wait(self, timeout=None): + return self.returncode + + child = Interrupted() + with self.assertRaises(KeyboardInterrupt): + service.start(allow_isolated=True, launcher=lambda *a, **k: child, timeout=5.0) + + self.assertIn( + "terminate", child.stopped, + "an unconfirmed child was left running after the start was interrupted", + ) + + def test_a_scope_claim_that_cannot_publish_its_record_releases_the_lock(self): + """A lock with no record is a claim on behalf of a registration that does not exist, + and it goes on refusing every later start in this process until the object dies.""" + registry = ScopeRegistry(Path(self.scopes), ISOLATED) + registry.prepare() + with mock.patch.object(service_module.Path, "write_text", + side_effect=OSError("no space left on device")): + with self.assertRaises(OSError): + registry.claim(SOCKET, {"storeId": "a-store"}) + + self.assertIsNone(registry._handle, "the lock outlived the claim that failed") + self.assertTrue( + registry.claim(SOCKET, {"storeId": "a-store"})["ok"], + "a later claim in the same process was refused by the abandoned lock", + ) + + def test_stop_finalises_the_record_under_the_lock(self): + """The re-read that detects a replacement is a snapshot. + + The daemon lock is free the moment the old supervisor and worker are gone, so a + replacement can acquire it after that read and publish its own record - and the + clearing write would erase the identity of a launch that is running, leaving every + later status and stop with no handle on it. + """ + service = self.service("i") + service.enable(actor="owner") + child, _pid = self.holder(service) + order = [] + original_lock = service.daemon_lock_if_free + original_write = service.write_record + + @contextlib.contextmanager + def watched_lock(): + order.append("lock") + with original_lock() as held: + yield held + + def watched_write(payload): + order.append("write") + return original_write(payload) + + service.daemon_lock_if_free = watched_lock + service.write_record = watched_write + try: + stopped = service.stop(actor="owner") + finally: + service.daemon_lock_if_free = original_lock + service.write_record = original_write + + child.wait(timeout=10) + self.assertTrue(stopped["ok"], stopped) + self.assertEqual( + order[-2:], ["lock", "write"], + "the stopped record was published without holding the lock that keeps a" + " replacement out", + ) + + def test_a_gone_supervisor_with_no_boot_id_leaves_its_worker_unverifiable(self): + """stop() falls through to _stop_worker when ownership answers none, and that + validates start ticks only. A reboot resets those along with the pid space, so an + unrelated process holding the old worker number would be signalled. The cleared-pid + path already refused this; the already-gone path is just as stale and did not. + """ + if service_module.boot_id() is None: + self.skipTest("this host records no boot id, so the rule cannot apply") + service = self.service("j") + finished = subprocess.Popen([sys.executable, "-c", "pass"]) + finished.wait(timeout=10) + service.write_record(dict( + service.new_record(pid=os.getpid()), pid=finished.pid, workerPid=os.getpid(), + workerStartTicks=service_module.start_ticks(os.getpid()), bootId=None, + )) + + owner, handle, detail = service.ownership() + if handle is not None: + handle.close() + + self.assertEqual(owner, service_module.UNVERIFIABLE) + self.assertIn("boot id", detail) + + def test_stop_reads_a_lock_it_cannot_take_after_both_exits_as_a_replacement(self): + """A replacement usually has not published yet, so latest still reads the OLD record + and the identity comparison cannot see it. Both recorded processes are confirmed gone + by this point, so nothing this stop was acting on can be holding the lock, and the + failed acquisition is the only evidence available in that window. + """ + service = self.service("k") + service.enable(actor="owner") + child, _pid = self.holder(service) + + @contextlib.contextmanager + def never_free(): + yield None + + service.daemon_lock_if_free = never_free + stopped = service.stop(actor="owner") + + child.wait(timeout=10) + self.assertFalse(stopped["ok"], stopped) + self.assertEqual(stopped["reason"], "replaced_by_new_launch") + + def test_stop_writes_nothing_when_it_cannot_hold_the_lock(self): + """One rule, because three narrower ones each left a window. + + A replacement that has published, one that has not, and an absent record are all the + same situation: this process does not hold the state directory, so it does not get to + write the record. Holding the lock is the only thing that excludes a replacement for + the duration of the write rather than at one instant. + """ + service = self.service("m") + service.enable(actor="owner") + child, _pid = self.holder(service) + before = service.record() + + @contextlib.contextmanager + def never_free(): + yield None + + writes = [] + original_write = service.write_record + service.daemon_lock_if_free = never_free + service.write_record = lambda payload: (writes.append(payload), + original_write(payload))[1] + try: + stopped = service.stop(actor="owner") + finally: + service.write_record = original_write + + child.wait(timeout=10) + self.assertFalse(stopped["ok"], stopped) + self.assertEqual(stopped["reason"], "replaced_by_new_launch") + self.assertEqual(writes, [], "a stop that holds nothing wrote the shared record") + self.assertEqual(service.record()["launchId"], before["launchId"]) + + def test_a_lock_that_fails_operationally_is_not_reported_as_a_replacement(self): + """flock reports contention with EACCES or EAGAIN. Every other OSError is an + operational failure - an unreadable directory, no descriptors left - and mapping it + to "someone holds it" made stop announce a replacement that does not exist, which a + restart then refuses to work around.""" + service = self.service("n") + service.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + + def refuse_to_open(*_args, **_kwargs): + raise OSError(errno.EMFILE, "too many open files") + + with mock.patch("builtins.open", side_effect=refuse_to_open): + with self.assertRaises(OSError) as caught: + with service.daemon_lock_if_free(): + pass + + self.assertEqual(caught.exception.errno, errno.EMFILE) + + def test_disable_refuses_a_lock_holder_nothing_here_can_identify(self): + """A supervisor that has taken the lock and not yet published its record. + + ownership() answers none, _foreign_markers has nothing to read from, and the old + condition asked only about foreign and unverifiable - so this wrote the shared + enabled=false anyway. The starting supervisor reads that at its first worker boundary + and exits, which is a refused disable that still shut another installation down. + """ + import fcntl + + service = self.service("c") + service.enable(actor="owner") + service.selection.path.mkdir(mode=0o700, parents=True, exist_ok=True) + handle = open(service.selection.path / service_module.DAEMON_LOCK, "a+") + self.addCleanup(handle.close) + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + refused = service.disable(actor="owner") + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertTrue( + service.intent.read()["enabled"], + "a refused disable must not change the intent the starting supervisor reads", + ) + + def test_a_stale_worker_number_does_not_make_a_held_lock_ours(self): + """A recorded worker pid is not ownership until its start time still matches. + + Once that process is gone the number proves nothing, and something else is holding + this directory - which is the case the guard exists for. + """ + import fcntl + + service = self.service("d") + service.enable(actor="owner") + service.write_record(dict( + service.new_record(pid=os.getpid()), pid=None, workerPid=os.getpid(), + workerStartTicks=(service_module.start_ticks(os.getpid()) or 0) + 1, + )) + handle = open(service.selection.path / service_module.DAEMON_LOCK, "a+") + self.addCleanup(handle.close) + fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + refused = service.disable(actor="owner") + finally: + fcntl.flock(handle, fcntl.LOCK_UN) + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertTrue(service.intent.read()["enabled"]) + + def test_disable_still_reaches_an_orphaned_worker_of_our_own(self): + """The guard must not refuse the case stop() deliberately still supports. + + A supervisor that died leaving our own worker alive answers none, and the worker holds + the lock through the descriptor it inherited. Refusing here would leave an owner unable + to disable their own orphan. + """ + service = self.service("e") + service.enable(actor="owner") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + disabled = service.disable(actor="owner") + + child.wait(timeout=10) + self.assertTrue(disabled["ok"], disabled) + self.assertFalse(service.intent.read()["enabled"]) + def test_stop_refuses_a_recycled_pid(self): + service = self.service("a") + child, pid = self.holder(service) + record = service.record() + service.write_record(dict(record, startTicks=(record["startTicks"] or 0) + 1)) + refused = service.stop() + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("reused", refused["detail"]) + self.assertIsNone(child.poll()) + + def test_stop_leaves_no_request_when_a_supervisor_starts_in_the_window(self): + """The probe and the write were two operations, with a startup in between. + + supervise() clears pending requests BEFORE it acquires the lock, so a request left + by a stop that saw the lock free is consumed by the supervisor that took it - which + then exits, although the stop reported not_running. + """ + service = self.service("a") + service.enable(actor="test") + original = service.daemon_lock_if_free + calls = {"n": 0} + + @contextlib.contextmanager + def taken_by_a_starting_supervisor(): + calls["n"] += 1 + # The lock is free when the old code probed it and held by the time the decision + # is actually made. Holding it across the decision is what closes the window. + yield None + + service.daemon_lock_if_free = taken_by_a_starting_supervisor + try: + refused = service.stop() + finally: + service.daemon_lock_if_free = original + + self.assertEqual(calls["n"], 1, "the decision must be taken under the lock") + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertFalse( + service.stop_request_path.exists(), + "a stop that could not establish ownership must leave nothing behind", + ) + + def test_stop_reports_not_running_without_a_request_when_the_lock_is_free(self): + """Holding the lock IS the proof that nothing is running and nothing can start.""" + service = self.service("a") + service.enable(actor="test") + + outcome = service.stop() + + self.assertFalse(outcome["ok"]) + self.assertEqual(outcome["reason"], "not_running") + self.assertFalse( + service.stop_request_path.exists(), + "nothing was running, so nothing needs to be told to stop", + ) + + def test_stop_writes_no_request_for_a_lock_held_by_an_unidentified_process(self): + """A supervisor between taking the lock and writing its record has no identity yet. + + The stop request is not harmless there: a supervisor clears pending requests only + BEFORE taking the lock, so a request written after that is consumed by the very + service the caller was told it had not stopped. + """ + service = self.service("a") + child, _pid = self.holder(service) + service.record_path.unlink() + self.assertTrue(service.lock_is_held()) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertFalse( + service.stop_request_path.exists(), + "a refused stop must not leave a request the service will act on", + ) + self.assertIsNone(child.poll()) + + def test_stop_refuses_when_the_process_cannot_be_aimed_at(self): + service = self.service("a") + child, pid = self.holder(service) + with mock.patch.object(os, "pidfd_open", side_effect=OSError("no pidfd here")): + refused = service.stop() + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIsNone(child.poll(), "refusing is the point; never fall back to os.kill") + self.assertFalse( + service.stop_request_path.exists(), + "a refused stop must leave no request behind; it would halt someone else", + ) + + def test_a_record_without_a_start_time_is_unverifiable_not_ours(self): + """A pid with no start time is just a number, and anything holding it would pass.""" + service = self.service("a") + child, _pid = self.holder(service) + record = service.record() + service.write_record(dict(record, startTicks=None)) + refused = service.stop() + self.assertEqual(refused["reason"], "ownership_unverifiable") + self.assertIn("start time", refused["detail"]) + self.assertIsNone(child.poll()) + self.assertFalse(service.stop_request_path.exists()) + + def test_stop_terminates_a_process_this_installation_owns(self): + service = self.service("a") + child, pid = self.holder(service) + stopped = service.stop() + self.assertTrue(stopped["ok"], stopped) + self.assertEqual(stopped["supervisor"], "exited") + child.wait(timeout=10) + self.assertIsNone(service.record()["pid"]) + self.assertFalse(service.lock_is_held()) + + def test_a_supervisor_finishing_normally_is_not_a_replacement(self): + """Cleanup clears the supervisor's own pid while keeping its launch id. + + A launch identity that included the pid turned that ordinary exit into a phantom + replacement, so a stop that genuinely stopped the service reported failure. + """ + service = self.service("a") + live = dict(service.new_record(pid=os.getpid()), launchId="launch-7") + cleaned = dict(live, pid=None) + + self.assertEqual( + service._launch_identity(live), service._launch_identity(cleaned), + "the same launch clearing its own pid is still the same launch", + ) + self.assertNotEqual( + service._launch_identity(live), + service._launch_identity(dict(live, launchId="launch-8")), + ) + # And anonymous runs still separate on when they started. + anon = dict(live, launchId=None, startedAt="2026-01-01T00:00:00Z") + self.assertNotEqual( + service._launch_identity(anon), + service._launch_identity(dict(anon, startedAt="2026-01-02T00:00:00Z")), + ) + + def test_two_anonymous_launches_are_still_distinguishable(self): + """A direct 'service run' carries no launch id, so comparing that field alone made + two different launches compare equal - and handed the replacement straight back.""" + service = self.service("a") + child, _pid = self.holder(service) + ours = dict(service.record(), launchId=None) + service.write_record(ours) + replacement = subprocess.Popen( + [sys.executable, "-c", IDLE_CHILD], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(replacement) + original = service._terminate + + def publish_anonymous_replacement(handle, **kwargs): + outcome = original(handle, **kwargs) + service.write_record(dict( + ours, launchId=None, pid=os.getpid(), startedAt="2099-01-01T00:00:00Z", + workerPid=replacement.pid, + workerStartTicks=service_module.start_ticks(replacement.pid), + )) + return outcome + + with mock.patch.object(service, "_terminate", publish_anonymous_replacement): + outcome = service.stop() + + child.wait(timeout=10) + self.assertIsNone(replacement.poll(), "the replacement's worker is not ours") + self.assertFalse( + outcome["ok"], + "a stop that left a replacement running has not stopped the service", + ) + self.assertEqual(outcome["reason"], "replaced_by_new_launch") + + def test_stop_does_not_reach_into_a_replacement_launch(self): + """Re-reading after the supervisor exits can pick up a NEW launch's record. + + Acting on that stops the replacement's worker and clears the replacement's pid using + the outcome of the supervisor we actually stopped - reporting success while the new + service is still alive. + """ + service = self.service("a") + child, _pid = self.holder(service) + ours = dict(service.record(), launchId="launch-ours") + service.write_record(ours) + replacement = subprocess.Popen( + [sys.executable, "-c", IDLE_CHILD], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(replacement) + original = service._terminate + + def publish_a_new_launch(handle, **kwargs): + outcome = original(handle, **kwargs) + # A start that acquired the lock the moment ours let go. + service.write_record(dict( + ours, launchId="launch-theirs", pid=os.getpid(), + workerPid=replacement.pid, + workerStartTicks=service_module.start_ticks(replacement.pid), + )) + return outcome + + with mock.patch.object(service, "_terminate", publish_a_new_launch): + service.stop() + + child.wait(timeout=10) + self.assertIsNone( + replacement.poll(), + "the replacement launch's worker is not ours to signal", + ) + self.assertEqual( + service.record()["launchId"], "launch-theirs", + "and its record must not be overwritten with our outcome", + ) + self.assertEqual(service.record()["workerPid"], replacement.pid) + + def test_stop_reaches_the_worker_the_record_names_now(self): + """A supervisor replacing a worker while stop runs left the first read stale. + + Stopping the worker that already exited reports success while its replacement, + which holds the inherited locks, is still delivering. + """ + service = self.service("a") + child, _pid = self.holder(service) + first = subprocess.Popen( + [sys.executable, "-c", "import time\nwhile True:\n time.sleep(0.05)\n"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(first) + first.terminate() + first.wait(timeout=10) + service.write_record(dict( + service.record(), workerPid=first.pid, workerStartTicks=1, + )) + replacement = subprocess.Popen( + [sys.executable, "-c", "import time\nwhile True:\n time.sleep(0.05)\n"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ) + self.children.append(replacement) + original = service._terminate + + def replace_then_terminate(handle, **kwargs): + # The supervisor swapping workers in the window stop() used to read across. + service.write_record(dict( + service.record(), workerPid=replacement.pid, + workerStartTicks=service_module.start_ticks(replacement.pid), + )) + return original(handle, **kwargs) + + with mock.patch.object(service, "_terminate", replace_then_terminate): + stopped = service.stop() + + child.wait(timeout=10) + self.assertIsNotNone( + replacement.poll(), "the worker the record names NOW is the one stop must reach", + ) + self.assertEqual(stopped["worker"], "exited") + def test_a_stale_record_does_not_block_a_fresh_start(self): + service = self.service("a") + child, pid = self.holder(service) + child.terminate() + child.wait(timeout=10) + status = service.status() + self.assertFalse(status["running"]) + self.assertEqual(status["ownership"], "none") + self.assertFalse(service.lock_is_held()) + + +class LaunchReporting(ServiceTestCase): + """start reports what the child managed to do, and a finished launch did nothing.""" + + def finishing_launcher(self): + """A launcher whose child reaches supervision's cleanup window and stays there.""" + def launcher(service, **_kw): + program = FINISHING.format( + src=os.path.join(REPO, "src"), state=str(service.selection.path), + socket=service.socket_path, scopes=str(service.scope.root), + store_id=service.store_id, launch=service.launch_id, + ) + child = subprocess.Popen( + [sys.executable, "-c", program], stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE, text=True, + ) + self.children.append(child) + return child + return launcher + + def test_a_launch_that_already_finished_is_not_reported_as_running(self): + """A bound small enough to finish during startup - or an expired deadline. + + The launch id still matches and the daemon lock is still held, so matching on + those two alone hands the caller success for a service that is on its way out. + """ + service = self.service("a") + service.enable(actor="test") + + outcome = service.start( + allow_isolated=True, launcher=self.finishing_launcher(), timeout=2.0, + ) + + self.assertFalse(outcome["ok"], f"a cleared pid is not a running service: {outcome}") + self.assertIsNone(outcome.get("pid")) + self.assertEqual(outcome["reason"], "did_not_report") + + def test_a_live_launch_is_still_reported_as_running(self): + """The same path with a pid in the record, so the guard is not refusing everything.""" + service = self.service("b") + service.enable(actor="test") + started = {} + + def launcher(svc, **_kw): + child, pid = self.holder(svc, launch=svc.launch_id) + started["pid"] = pid + return child + + outcome = service.start(allow_isolated=True, launcher=launcher, timeout=10.0) + + self.assertTrue(outcome["ok"], outcome) + self.assertEqual(outcome["pid"], started["pid"]) + + def test_a_launch_that_never_reports_is_not_left_running(self): + """Reporting failure and walking away leaves the locks held against the retry. + + A child that is merely slow can come up after the caller was told the start failed, + and it holds the daemon lock and the scope claim while it does. + """ + service = self.service("c") + service.enable(actor="test") + program = "import time\nwhile True:\n time.sleep(0.05)\n" + + def launcher(_svc, **_kw): + child = subprocess.Popen( + [sys.executable, "-c", program], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + self.children.append(child) + return child + + outcome = service.start(allow_isolated=True, launcher=launcher, timeout=0.5) + + self.assertFalse(outcome["ok"], outcome) + self.assertEqual(outcome["reason"], "did_not_report") + self.assertEqual(outcome["child"], "terminated") + self.assertIsNotNone(self.children[-1].poll(), "the child this call started is gone") + +class ForeignWorkers(ServiceTestCase): + """A dead supervisor does not make its installation's worker ours to signal.""" + + def test_a_foreign_record_whose_supervisor_died_still_protects_its_worker(self): + """The gone check answered before the foreign markers were ever read. + + stop() deliberately falls through to the worker when the supervisor is gone, because + that orphan is exactly what a stop has to reach. Classifying a foreign record as + none first aimed that fall-through at another installation's worker. + """ + service = self.service("a") + child, worker_pid = self.holder(service) + record = service.record() + service.write_record(dict( + record, installationId="someone-else", workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + real = os.pidfd_open + + def supervisor_is_gone(pid, *args, **kwargs): + if pid == record["pid"]: + raise ProcessLookupError(f"supervisor {pid} is gone") + return real(pid, *args, **kwargs) + + with mock.patch.object(os, "pidfd_open", supervisor_is_gone): + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertIn("another installation", refused["detail"]) + self.assertEqual(refused["worker"], "untouched") + self.assertIsNone(child.poll(), "a foreign worker must not be signalled") + self.assertFalse( + service.stop_request_path.exists(), + "a refused stop must not leave a request that halts another installation", + ) + + def test_an_orphaned_foreign_worker_is_still_not_ours_to_signal(self): + """The record supervise() now leaves behind: pid cleared, worker identity kept. + + Keeping the worker identity is what lets a stop reach an orphan, and the ownership + check returned none as soon as the supervisor pid was gone - so that orphan path + pointed straight at another installation's worker. + """ + service = self.service("a") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, installationId="someone-else", + workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + refused = service.stop() + + self.assertFalse(refused["ok"], refused) + self.assertEqual(refused["reason"], "not_ours") + self.assertEqual(refused["worker"], "untouched") + self.assertIsNone(child.poll(), "a foreign orphan must not be signalled") + self.assertFalse(service.stop_request_path.exists()) + + def test_an_orphaned_worker_of_our_own_is_still_reachable(self): + """The guard must not refuse the case it was added to support.""" + service = self.service("b") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), pid=None, workerPid=worker_pid, + workerStartTicks=service_module.start_ticks(worker_pid), + )) + + stopped = service.stop() + + child.wait(timeout=10) + self.assertEqual(stopped["worker"], "exited") + + def test_a_worker_with_no_recorded_start_time_is_unverifiable_not_killable(self): + """The rule the supervisor already had. A reused worker pid is an unrelated process.""" + service = self.service("a") + child, worker_pid = self.holder(service) + service.write_record(dict( + service.record(), workerPid=worker_pid, workerStartTicks=None, + )) + + self.assertEqual( + service._stop_worker(service.record(), timeout=1.0, grace=0.05), "unverifiable", + ) + self.assertIsNone(child.poll(), "an unverifiable worker must not be signalled") + + +class RecordDurability(ServiceTestCase): + def test_a_record_is_replaced_atomically_rather_than_truncated(self): + """_note rewrites this file at every worker boundary; a reader must never see half. + + A concurrent stop reading the truncated middle would call a running supervisor + absent and return not_running without ever signalling its worker. + """ + service = self.service("a") + service.write_record(service.new_record(pid=os.getpid())) + before = service.record_path.stat() + + service._note(restarts=7) + + self.assertEqual(service.record()["restarts"], 7) + self.assertNotEqual( + service.record_path.stat().st_ino, before.st_ino, + "an in-place rewrite is the truncation window; the file must be replaced", + ) + self.assertEqual( + [p.name for p in service.selection.path.glob(".daemon.json.*")], [], + "no temporary file is left behind", + ) + + def test_the_intent_file_is_replaced_atomically_too(self): + """read() calls an unreadable document not configured, which reads as disabled. + + The supervisor re-reads intent at every worker boundary, so a reader landing in the + truncated middle of this write stops a service its owner left enabled. + """ + service = self.service("a") + service.enable(actor="test") + before = service.intent.path.stat() + + service.intent.write(enabled=True, actor="owner") + + self.assertTrue(service.intent.read()["enabled"]) + self.assertNotEqual(service.intent.path.stat().st_ino, before.st_ino) + self.assertEqual( + [p.name for p in service.selection.path.glob(".service.json.*")], [], + ) + + +class SupervisorCleanup(ServiceTestCase): + """What the supervisor leaves behind when it does not get to finish.""" + + def test_a_worker_that_was_never_waited_on_keeps_its_identity(self): + """The cleanup cleared workerPid unconditionally, including after an exception. + + The worker still holds the daemon lock and the scope claim it inherited, so erasing + the only identity a stop can aim at leaves it delivering for the rest of its segment + while status reports not_running. + """ + service = self.service("a") + service.enable(actor="test") + worker = FakeWorker(0, pid=os.getpid()) + worker.wait = _explode + + with self.assertRaises(RuntimeError): + service.supervise(allow_isolated=True, spawn=lambda **_k: worker, + sleeper=lambda _s: None, max_segments=1) + + record = service.record() + self.assertIsNone(record["pid"], "the supervisor itself is gone") + self.assertEqual(record["workerPid"], worker.pid, + "but the orphan it left behind is still reachable") + self.assertIsNotNone(record["workerStartTicks"]) + + def test_a_clean_run_still_clears_the_worker(self): + service = self.service("b") + service.enable(actor="test") + service.supervise(allow_isolated=True, spawn=lambda **_k: FakeWorker(0), + sleeper=lambda _s: None, max_segments=1) + record = service.record() + self.assertIsNone(record["workerPid"]) + self.assertIsNone(record["workerStartTicks"]) + + def test_the_restart_delay_never_outlives_the_supervisors_own_bound(self): + """An unclamped delay made a 0.01-second deadline take seconds.""" + service = self.service("c") + service.enable(actor="test") + slept = [] + service.supervise( + allow_isolated=True, spawn=lambda **_k: FakeWorker(1), + sleeper=slept.append, max_segments=4, deadline=0.01, + policy=RetryPolicy(restart_base_seconds=30.0, restart_backoff_max_seconds=300.0), + ) + self.assertTrue(slept, "a failed segment does wait before its replacement") + self.assertLess( + max(slept), 30.0, + "the unclamped delay is the policy interval, which outlives the whole bound", + ) + self.assertTrue(all(value <= 0.01 for value in slept), slept) + + def test_a_long_failure_streak_keeps_retrying_at_the_cap(self): + """The backoff built the product and clamped after, so it overflowed and killed the + supervisor it was meant to pace. + """ + policy = RetryPolicy() + ceiling = policy.restart_backoff_max_seconds + for failures in (1, 2, 10, 1024, 1025, 10 ** 6): + delay = policy.restart_delay_for(failures) + self.assertIsInstance(delay, (int, float)) + self.assertLessEqual(delay, ceiling) + self.assertEqual(policy.restart_delay_for(10 ** 6), ceiling) + self.assertLess(policy.restart_delay_for(1), policy.restart_delay_for(4)) + + def test_the_bound_follows_the_policy_rather_than_a_fixed_step_count(self): + """A small base needs more doublings, and a constant bound truncated its backoff.""" + # Deliberately extreme: this ratio needs about 70 doublings, so a fixed 64-step bound + # returns the ceiling while the policy's own backoff still has room. + fine = RetryPolicy(restart_base_seconds=1e-12, restart_backoff_max_seconds=1e9) + self.assertLess( + fine.restart_delay_for(65), fine.restart_backoff_max_seconds, + "this policy has not reached its own ceiling yet", + ) + self.assertLess(fine.restart_delay_for(65), fine.restart_delay_for(80)) + self.assertEqual(fine.restart_delay_for(10 ** 6), fine.restart_backoff_max_seconds) + for failures in (1, 65, 1025, 10 ** 6): + self.assertIsInstance(fine.restart_delay_for(failures), (int, float)) + + +class Intent(ServiceTestCase): + def test_start_never_enables_a_service_that_was_never_configured(self): + service = self.service("a") + refused = service.start(allow_isolated=True, launcher=self._fail) + self.assertEqual(refused["reason"], "service_disabled") + self.assertFalse(service.intent.read()["enabled"]) + self.assertFalse(service.intent.read()["configured"]) + self.assertFalse((service.selection.path / "service.json").exists()) + + def _fail(self, *_a, **_k): # pragma: no cover + self.fail("a disabled service must not be launched") + + def test_restart_refuses_a_disabled_service_and_leaves_it_disabled(self): + service = self.service("a") + service.enable(actor="test") + service.disable(actor="test") + refused = service.restart(allow_isolated=True, launcher=self._fail) + self.assertEqual(refused["reason"], "service_disabled") + self.assertFalse(service.intent.read()["enabled"]) + + def test_a_disable_during_restart_stops_the_replacement(self): + """The window between restart stopping the old process and launching the new one.""" + service = self.service("a") + service.enable(actor="test") + original = service.stop + + def stop_then_disable(**kwargs): + outcome = original(**kwargs) + service.intent.write(enabled=False, actor="owner") + return outcome + + with mock.patch.object(service, "stop", stop_then_disable): + refused = service.restart(allow_isolated=True, launcher=self._fail) + self.assertEqual(refused["reason"], "service_disabled") + self.assertIn("while the service was stopping", refused["detail"]) + self.assertFalse(service.intent.read()["enabled"]) + + +class ScopeAuthority(ServiceTestCase): + def test_the_production_root_does_not_move_with_the_environment(self): + fake = tempfile.mkdtemp(prefix="relay-passwd-home-") + self.addCleanup(shutil.rmtree, fake, ignore_errors=True) + entry = mock.Mock(pw_dir=fake) + with mock.patch.object(service_module.pwd, "getpwuid", return_value=entry): + with mock.patch.dict(os.environ, {"HOME": "/somewhere/else", + "XDG_STATE_HOME": "/elsewhere"}, clear=False): + first = production_scope_root() + with mock.patch.dict(os.environ, {"HOME": "/a/third/place"}, clear=False): + os.environ.pop("XDG_STATE_HOME", None) + second = production_scope_root() + self.assertEqual(first, second) + self.assertTrue(str(first).startswith(fake)) + + def test_an_override_is_isolated_and_refused_without_an_explicit_opt_in(self): + """Two launches with DIFFERENT overrides would otherwise both own the same socket.""" + for name, root in (("a", "scopes-a"), ("b", "scopes-b")): + service = self.service(name, scopes=os.path.join(self.tmp, root)) + service.intent.write(enabled=True, actor="test") + refused = service.start(allow_isolated=False, launcher=self._fail) + self.assertEqual(refused["reason"], "isolated_scope_not_allowed") + self.assertIn("CODEX_SESSION_RELAY_SCOPE_DIR", refused["detail"]) + self.assertEqual(service.status()["scopeAuthority"], ISOLATED) + + def _fail(self, *_a, **_k): # pragma: no cover + self.fail("an isolated authority must not start without an explicit opt-in") + + def test_resolve_reports_production_when_nothing_is_overridden(self): + with mock.patch.dict(os.environ, {}, clear=False): + os.environ.pop("CODEX_SESSION_RELAY_SCOPE_DIR", None) + _root, authority = resolve_scope_root() + self.assertEqual(authority, PRODUCTION) + + def test_an_installation_is_this_checkout_and_this_state_directory(self): + self.assertNotEqual( + installation_id(os.path.join(self.tmp, "a")), + installation_id(os.path.join(self.tmp, "b")), + ) + + def test_two_names_for_one_socket_are_one_operating_scope(self): + """A symlinked socket is the same App Server, so it must be the same lock.""" + real = os.path.join(self.tmp, "real.sock") + open(real, "w", encoding="utf-8").close() + alias = os.path.join(self.tmp, "alias.sock") + os.symlink(real, alias) + registry = ScopeRegistry(Path(self.scopes), ISOLATED) + self.assertEqual( + registry.key(real), registry.key(alias), + "two spellings of one socket must not give two owners", + ) + + +if __name__ == "__main__": + unittest.main() + + +class FakeWorker: + """Stands in for a bounded worker process, so cadence is tested without real time.""" + + def __init__(self, code=0, pid=None): + self.returncode = code + self.pid = pid or os.getpid() + + def wait(self): + return self.returncode + + +def _explode(): + """A worker the supervisor never gets an exit code from.""" + raise RuntimeError("the supervisor died between spawn and wait") + + + +class Supervision(ServiceTestCase): + def supervised(self, service, *, codes, **kwargs): + """Run the supervisor over a scripted sequence of worker exits.""" + launches, slept = [], [] + queue = list(codes) + + def spawn(**call): + launches.append(call) + return FakeWorker(queue.pop(0) if queue else 0) + + outcome = service.supervise( + allow_isolated=True, spawn=spawn, sleeper=slept.append, + max_segments=len(codes), **kwargs, + ) + return outcome, launches, slept + + def test_a_worker_that_exits_is_replaced_on_the_same_store(self): + service = self.service("a") + service.enable(actor="test") + before = service.store_id + outcome, launches, _slept = self.supervised(service, codes=[0, 0, 0]) + self.assertEqual(outcome["segments"], [0, 0, 0]) + self.assertEqual(len(launches), 3, "each bounded worker is replaced by a new one") + # One lock, one claim, one store across every replacement. + self.assertEqual(service.store_id, before) + self.assertEqual(service.record()["restarts"], 3) + self.assertFalse(service.lock_is_held(), "the supervisor released on the way out") + + def test_every_worker_inherits_the_descriptors_the_supervisor_holds(self): + service = self.service("a") + service.enable(actor="test") + _outcome, launches, _slept = self.supervised(service, codes=[0, 0]) + self.assertEqual(len({call["lock_fd"] for call in launches}), 1) + self.assertEqual(len({call["scope_fd"] for call in launches}), 1) + self.assertEqual(len({call["token"] for call in launches}), 1) + self.assertIsNotNone(launches[0]["lock_fd"]) + self.assertIsNotNone(launches[0]["scope_fd"]) + + def test_supervision_refuses_a_disable_that_landed_while_it_was_starting(self): + """The first intent check runs while nothing is held. + + A disable that lands between it and the instance lock would otherwise start a + supervisor its owner had already turned off - and one the disabling caller never + examined, because what it classified was the PREVIOUS holder. Whoever holds the lock + is the one whose intent decides, so the answer is re-read under it. + """ + service = self.service("a") + service.enable(actor="test") + enabled = service.intent.read() + reads = [] + + def once_then_disabled(): + reads.append(1) + return enabled if len(reads) == 1 else dict(enabled, enabled=False) + + service.intent.read = once_then_disabled + launches = [] + + def spawn(**call): # pragma: no cover - reaching it is the failure + launches.append(call) + return FakeWorker(0) + + with self.assertRaises(ServiceRefused) as caught: + service.supervise( + allow_isolated=True, spawn=spawn, sleeper=lambda _s: None, max_segments=1, + ) + + self.assertEqual(caught.exception.reason, "service_disabled") + self.assertEqual(launches, [], "a disabled service was supervised anyway") + + def test_a_stopped_supervisor_reports_no_pending_restart(self): + """nextRestartAt named a restart this supervisor is no longer going to make, so a + stopped service read as though one were still scheduled.""" + service = self.service("a") + service.enable(actor="test") + self.supervised(service, codes=[1, 1]) + self.assertIsNone(service.record()["nextRestartAt"]) + + def test_repeated_failure_backs_off_and_is_reported(self): + service = self.service("a") + service.enable(actor="test") + outcome, _launches, slept = self.supervised(service, codes=[1, 1, 1]) + self.assertEqual(outcome["consecutiveFailures"], 3) + self.assertIsNotNone(outcome["degraded"], "repeated failure is reported, not hidden") + self.assertIn("3 consecutive worker failures", outcome["degraded"]) + # Capped exponential, not a fixed interval and not a tight loop. + self.assertEqual(slept[:2], [2.0, 4.0]) + log = (service.selection.path / "daemon.log").read_text(encoding="utf-8") + self.assertIn("service_degraded", log) + + def test_a_clean_segment_resets_the_failure_count(self): + service = self.service("a") + service.enable(actor="test") + # A fourth segment so the reset is observable as a delay. The delay after the LAST + # segment no longer exists - there is nothing left to wait for - so a three-code run + # would show the backoff climbing and never show it come back down. + outcome, _launches, slept = self.supervised(service, codes=[1, 1, 0, 1]) + self.assertEqual(outcome["consecutiveFailures"], 1) + self.assertEqual(slept[:3], [2.0, 4.0, 2.0]) + + def test_the_last_allowed_segment_does_not_wait_to_restart_nothing(self): + """The segment bound was only checked at the top of the loop, so every finite run + slept one restart delay it had no use for - up to the backoff cap after repeated + failures - before returning.""" + service = self.service("a") + service.enable(actor="test") + _outcome, launches, slept = self.supervised(service, codes=[1, 1]) + self.assertEqual(len(launches), 2) + self.assertEqual(slept, [2.0], "the run waited after the segment it was never going" + " to replace") + + def test_disabling_the_service_ends_supervision_at_the_boundary(self): + service = self.service("a") + service.enable(actor="test") + launches = [] + + def spawn(**call): + launches.append(call) + service.intent.write(enabled=False, actor="owner") + return FakeWorker(0) + + outcome = service.supervise( + allow_isolated=True, spawn=spawn, sleeper=lambda _s: None, max_segments=5, + ) + self.assertEqual(len(launches), 1, "no replacement after the owner disabled it") + self.assertEqual(outcome["segments"], [0]) + self.assertFalse(service.intent.read()["enabled"]) + + def test_a_stop_request_ends_supervision_without_a_replacement(self): + service = self.service("a") + service.enable(actor="test") + launches = [] + + def spawn(**call): + launches.append(call) + service.request_stop() + return FakeWorker(0) + + service.supervise( + allow_isolated=True, spawn=spawn, sleeper=lambda _s: None, max_segments=5, + ) + self.assertEqual(len(launches), 1) + + def test_supervising_a_disabled_service_is_refused(self): + service = self.service("a") + with self.assertRaises(ServiceRefused) as caught: + service.supervise(allow_isolated=True, spawn=lambda **_k: FakeWorker(0)) + self.assertEqual(caught.exception.reason, "service_disabled") + + def test_a_launch_is_not_ready_until_initialisation_has_returned(self): + """start matched a record written before on_start ran. + + Recovery happens in on_start, and it can take longer than the poll interval or fail + outright on the App Server connection. Everything start matched on - the pid, the + launch id, the daemon lock - is already true while that is still in flight. + """ + service = self.service("a") + service.enable(actor="test") + service.launch_id = "launch-under-test" + seen = {} + + def on_start(): + seen["record"] = service.record() + + outcome, _launches, _slept = self.supervised(service, codes=[0], on_start=on_start) + + self.assertTrue(outcome["ok"], outcome) + self.assertEqual(seen["record"]["launchId"], "launch-under-test") + self.assertIsNotNone(seen["record"]["pid"], "the record start polls was already there") + self.assertIsNone( + seen["record"]["readyAt"], + "initialisation had not returned, so this is not a started service", + ) + self.assertIsNotNone(service.record()["readyAt"], "and it is ready afterwards") + + def test_a_failed_initialisation_never_becomes_ready(self): + service = self.service("a") + service.enable(actor="test") + + def on_start(): + raise RuntimeError("the App Server connection failed") + + with self.assertRaises(RuntimeError): + self.supervised(service, codes=[0], on_start=on_start) + self.assertIsNone(service.record()["readyAt"]) + + +class Projects(ServiceTestCase): + """The operations contract says status groups by project; it has to actually do it.""" + + def assignment(self, service, name, *, cwd, issue, status="active"): + from codex_session_relay.store import Store + + store = Store(service.selection.db_path) + try: + store.db.execute( + "INSERT INTO relationships (relationship_id, issue_key, status," + " parent_task_id, parent_host_id, parent_cwd, child_task_id, child_host_id," + " execution_generation, artifact_roots, allowed_recipients, created_at," + " updated_at) VALUES (?,?,?,?,?,?,?,?,1,'[]','[]','now','now')", + (f"rel-{name}", issue, status, f"01parent-{name}", "host", + cwd, f"01child-{name}", "host"), + ) + finally: + store.close() + + def test_status_groups_a_shared_service_by_project(self): + service = self.service("a") + self.assignment(service, "one", cwd="/code/alpha", issue="ALPHA-1") + self.assignment(service, "two", cwd="/code/alpha", issue="ALPHA-2", status="paused") + self.assignment(service, "three", cwd="/code/beta", issue="BETA-1") + + projects = service.status()["projects"] + + self.assertTrue(projects["available"], projects) + self.assertEqual([p["project"] for p in projects["projects"]], + ["/code/alpha", "/code/beta"]) + alpha = projects["projects"][0] + self.assertEqual(alpha["assignments"], 2) + self.assertEqual(alpha["active"], 1, "a paused assignment is carried but not active") + self.assertEqual(alpha["issues"], ["ALPHA-1", "ALPHA-2"]) + self.assertEqual(len(alpha["parents"]), 2) + + def test_a_store_that_does_not_exist_is_reported_rather_than_created(self): + """status is an offline command; it must not bring a store into being to answer.""" + service = self.service("a") + service.selection.db_path.unlink() + + projects = service.status()["projects"] + + self.assertFalse(projects["available"]) + self.assertEqual(projects["projects"], []) + self.assertFalse(service.selection.db_path.exists(), "asking must not create it") + + def test_an_inventory_that_cannot_be_queried_is_not_an_empty_one(self): + """The file opened and the query did not. Reporting available says there are none.""" + service = self.service("a") + service.selection.db_path.write_bytes(b"") + + projects = service.status()["projects"] + + self.assertFalse(projects["available"]) + self.assertIsNotNone(projects["detail"]) + self.assertEqual(projects["projects"], []) + + +class SupervisedWorker(ServiceTestCase): + """A worker adopts what its supervisor holds, or it is refused outright.""" + + def adopt(self, service, **overrides): + from codex_session_relay.cli import _adopt_supervised + + class Args: + pass + + args = Args() + args.supervised_token = overrides.get("token", "tok") + args.supervised_lock_fd = overrides.get("lock_fd", 0) + args.supervised_scope_fd = overrides.get("scope_fd", -1) + return _adopt_supervised(service, args) + + def prepared(self, **record): + service = self.service("a") + service.selection.path.mkdir(parents=True, exist_ok=True) + (service.selection.path / "daemon.lock").write_text("", encoding="utf-8") + base = service.new_record(pid=os.getppid(), token="tok") + service.write_record(dict(base, **record)) + return service + + def test_an_unsupervised_run_is_untouched(self): + from codex_session_relay.cli import _adopt_supervised + + class Args: + supervised_token = None + supervised_lock_fd = None + supervised_scope_fd = None + + self.assertEqual(_adopt_supervised(self.service("a"), Args()), {}) + + def test_a_partial_supervised_invocation_is_refused(self): + from codex_session_relay.cli import PayloadExit + + service = self.prepared() + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, scope_fd=None) + self.assertEqual(caught.exception.payload["reason"], "supervised_invocation_incomplete") + + def test_a_record_naming_another_store_is_refused(self): + """The supervisor recorded which store it registered the scope for. + + This worker opened whatever relay.sqlite3 the path resolves to now, and a database + deleted or atomically replaced between worker segments is a different one. Without + this the worker serves an empty or unrelated store while the supervisor and the scope + registration still name the original - and every participant comparing identities is + told they agree. + """ + from codex_session_relay.cli import PayloadExit + + service = self.prepared(storeId="the-supervisors-store") + service.store_id = "a-replacement-store" + with self.assertRaises(PayloadExit) as caught: + self.adopt(service) + self.assertEqual(caught.exception.payload["reason"], "supervised_store_mismatch") + + def test_a_record_naming_the_same_store_is_not_stopped_by_the_store_check(self): + """The refusal must not swallow the ordinary supervised run. + + This gets as far as the descriptor check, which is where a test with no genuinely + inherited descriptor has to stop. What it establishes is that the store check was not + what stopped it. + """ + from codex_session_relay.cli import PayloadExit + + service = self.prepared(storeId="one-store") + service.store_id = "one-store" + with self.assertRaises(PayloadExit) as caught: + self.adopt(service) + self.assertEqual(caught.exception.payload["reason"], "supervised_fd_mismatch") + + def test_a_wrong_token_is_refused(self): + + from codex_session_relay.cli import PayloadExit + + service = self.prepared() + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, token="not-the-token") + self.assertEqual(caught.exception.payload["reason"], "supervised_token_mismatch") + + def test_a_descriptor_for_another_file_is_refused(self): + from codex_session_relay.cli import PayloadExit + + service = self.prepared() + stranger = os.open(os.path.join(self.tmp, "not-a-lock"), os.O_CREAT | os.O_RDWR, 0o600) + self.addCleanup(os.close, stranger) + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, lock_fd=stranger) + self.assertEqual(caught.exception.payload["reason"], "supervised_fd_mismatch") + + def test_a_worker_whose_supervisor_is_already_gone_refuses_to_serve(self): + from codex_session_relay.cli import PayloadExit + + # The record names a supervisor that is not this process's parent, which is exactly + # what an orphan looks like when PDEATHSIG arrived too late to help. + service = self.prepared(pid=999999) + lock = os.open(service.selection.path / "daemon.lock", os.O_RDWR) + self.addCleanup(os.close, lock) + with self.assertRaises(PayloadExit) as caught: + self.adopt(service, lock_fd=lock) + self.assertEqual(caught.exception.payload["reason"], "supervisor_already_gone") diff --git a/packages/codex-session-relay/tests/test_store.py b/packages/codex-session-relay/tests/test_store.py index 34c9519..66c5550 100644 --- a/packages/codex-session-relay/tests/test_store.py +++ b/packages/codex-session-relay/tests/test_store.py @@ -3,7 +3,14 @@ import os import unittest -from codex_session_relay.store import SCHEMA_VERSION, Store, state_dir +import shutil +import tempfile +from pathlib import Path +from unittest import mock + +from codex_session_relay.store import ( + SCHEMA_VERSION, Store, compare_store, nonce_lookup, probe, resolve_state_dir, state_dir, +) from .support import RelayTestCase @@ -105,5 +112,452 @@ def test_a_socket_gets_its_own_endpoint_directory(self): self.assertIn("codex-session-relay", str(first)) +class Precedence(unittest.TestCase): + """Four rules can choose the directory, and a participant has to be able to say which one.""" + + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-precedence-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + # Never read the real home or the real XDG state during a test. + self.home = os.path.join(self.tmp, "home") + os.makedirs(self.home) + patch = mock.patch.dict(os.environ, {"HOME": self.home}, clear=False) + patch.start() + self.addCleanup(patch.stop) + for name in ("CODEX_SESSION_RELAY_STATE", "XDG_STATE_HOME"): + os.environ.pop(name, None) + + def test_each_rule_wins_in_order_and_says_so(self): + home = resolve_state_dir(None, "/run/x.sock") + self.assertEqual(home.source, "home") + self.assertTrue(str(home.path).startswith(self.home)) + + os.environ["XDG_STATE_HOME"] = os.path.join(self.tmp, "xdg") + xdg = resolve_state_dir(None, "/run/x.sock") + self.assertEqual(xdg.source, "xdg") + self.assertIn("XDG_STATE_HOME=", xdg.detail) + self.assertEqual(xdg.socket_scope, home.socket_scope) + + os.environ["CODEX_SESSION_RELAY_STATE"] = os.path.join(self.tmp, "env") + env = resolve_state_dir(None, "/run/x.sock") + self.assertEqual(env.source, "env") + self.assertIn("CODEX_SESSION_RELAY_STATE=", env.detail) + + flag = resolve_state_dir(os.path.join(self.tmp, "flag"), "/run/x.sock") + self.assertEqual(flag.source, "flag") + self.assertIn("--state ", flag.detail) + self.assertEqual(flag.db_path.name, "relay.sqlite3") + + def test_a_relative_flag_resolves_to_the_same_store_as_its_absolute_form(self): + target = os.path.join(self.tmp, "rel") + os.makedirs(target) + Store(Path(target) / "relay.sqlite3").close() + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = probe(resolve_state_dir("rel")) + finally: + os.chdir(here) + absolute = probe(resolve_state_dir(target)) + self.assertEqual(relative["store"]["storeId"], absolute["store"]["storeId"]) + self.assertEqual(relative["store"]["inode"], absolute["store"]["inode"]) + + def test_a_store_already_in_use_keeps_its_directory_after_the_hash_changed(self): + """Canonicalising the socket moved this hash, which is an upgrade hazard. + + A relative or symlinked socket that had been running would otherwise open a fresh + empty database while its assignments, generations and pending deliveries sat in the + old directory, invisible. + """ + from codex_session_relay.store import legacy_socket_scope, socket_scope + + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = os.path.join("run", "app-server.sock") + os.makedirs(os.path.join(self.tmp, "run"), exist_ok=True) + open(os.path.join(self.tmp, "run", "app-server.sock"), "w").close() + legacy = os.path.join( + self.home, ".local", "state", "codex-session-relay", + legacy_socket_scope(relative), + ) + os.makedirs(legacy) + Store(Path(legacy) / "relay.sqlite3").close() + + chosen = resolve_state_dir(None, relative) + + self.assertEqual(str(chosen.path), legacy, + "the store already in use keeps its directory") + self.assertIn("already using", chosen.detail) + + # An empty canonical DIRECTORY is not a canonical store. Any command that writes + # beside the database creates one - a stop request is enough - and testing for + # the directory let that hide the legacy store behind an empty folder. + os.makedirs( + os.path.join(self.home, ".local", "state", "codex-session-relay", + socket_scope(relative)), + exist_ok=True, + ) + self.assertEqual( + str(resolve_state_dir(None, relative).path), legacy, + "an empty canonical directory must not hide a real store", + ) + finally: + os.chdir(here) + + def test_a_store_is_found_by_the_socket_it_recorded_not_by_its_hash(self): + """The legacy comparison only helps when THIS invocation uses the old spelling. + + A first post-upgrade command that happens to use the absolute path has + legacy == scope, so the comparison never looks at the store the relative spelling + created - and creating a canonical database then hides it for good, because + afterwards even the old spelling finds the new one. + """ + from codex_session_relay.store import legacy_socket_scope, socket_scope + + os.makedirs(os.path.join(self.tmp, "run"), exist_ok=True) + absolute = os.path.join(self.tmp, "run", "app-server.sock") + open(absolute, "w").close() + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = os.path.join("run", "app-server.sock") + # The store the pre-upgrade installation left behind, under the RELATIVE hash. + previous = os.path.join( + self.home, ".local", "state", "codex-session-relay", + legacy_socket_scope(relative), + ) + os.makedirs(previous) + Store(Path(previous) / "relay.sqlite3", socket_path=relative).close() + + # The first post-upgrade command uses the ABSOLUTE spelling, so the legacy + # comparison is a no-op: legacy == scope. + self.assertEqual( + legacy_socket_scope(absolute), socket_scope(absolute), + "the fixture needs the case the hash comparison cannot see", + ) + chosen = resolve_state_dir(None, absolute) + finally: + os.chdir(here) + + self.assertEqual(str(chosen.path), previous, + "the store recorded for this socket must be adopted, not hidden") + self.assertIn("adopted", chosen.detail) + + def test_two_stores_claiming_one_socket_are_not_silently_chosen_between(self): + """Picking whichever sorts first operates on one set of assignments today and the + other after a rename. Adopting nothing is wrong too, but it is visible.""" + from codex_session_relay.store import ( + discover_store_for_socket, stores_claiming_socket, + ) + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + socket = "/run/contested.sock" + both = [] + for name in ("aaaa000000000000", "bbbb000000000000"): + directory = os.path.join(root, name) + os.makedirs(directory) + Store(Path(directory) / "relay.sqlite3", socket_path=socket).close() + both.append(directory) + + self.assertIsNone( + discover_store_for_socket(Path(root), socket), + "an ambiguity is not a choice to make silently", + ) + self.assertEqual(sorted(stores_claiming_socket(Path(root), socket)), sorted(both)) + + def test_one_store_claiming_a_socket_is_still_adopted(self): + """The refusal must not swallow the case discovery exists for.""" + from codex_session_relay.store import discover_store_for_socket + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + socket = "/run/sole.sock" + directory = os.path.join(root, "cccc000000000000") + os.makedirs(directory) + Store(Path(directory) / "relay.sqlite3", socket_path=socket).close() + + self.assertEqual( + str(discover_store_for_socket(Path(root), socket)), directory, + ) + + def test_two_stores_claiming_one_socket_do_not_produce_a_third(self): + """Adopting neither is not the visible failure it was argued to be. + + The caller lands on the canonical directory, the first write there creates a THIRD + empty database, and from that moment the canonical-exists branch wins every later + resolution - so both real stores are hidden for good. The selection has to carry the + conflict instead, which is what lets the command line refuse. + """ + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + socket = "/run/contested-selection.sock" + both = [] + for name in ("aaaa333333333333", "bbbb333333333333"): + directory = os.path.join(root, name) + os.makedirs(directory) + Store(Path(directory) / "relay.sqlite3", socket_path=socket).close() + both.append(directory) + + chosen = resolve_state_dir(None, socket) + + self.assertEqual(sorted(chosen.ambiguous), sorted(both)) + self.assertEqual(sorted(chosen.to_record()["ambiguous"]), sorted(both)) + self.assertNotIn(str(chosen.path), both, "neither is adopted on a guess") + + def test_an_ordinary_selection_carries_no_ambiguity(self): + """The field is a conflict report, not a list of neighbours.""" + self.assertEqual(resolve_state_dir(None, "/run/quiet.sock").ambiguous, ()) + self.assertEqual(resolve_state_dir(os.path.join(self.tmp, "flag")).ambiguous, ()) + + def test_a_store_with_no_recorded_socket_is_never_adopted_on_a_guess(self): + """Provenance or nothing. Adopting an unlabelled store is how the wrong one is served.""" + from codex_session_relay.store import socket_scope, stores_without_provenance + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + stranger = os.path.join(root, "0123456789abcdef") + os.makedirs(stranger) + Store(Path(stranger) / "relay.sqlite3").close() + + chosen = resolve_state_dir(None, "/run/brand-new.sock") + + self.assertEqual(chosen.socket_scope, socket_scope("/run/brand-new.sock")) + self.assertNotEqual(str(chosen.path), stranger) + # But it IS visible, so an operator is not left guessing why a store looks empty. + self.assertIn(stranger, stores_without_provenance(root, skip=chosen.path.name)) + + def test_a_store_recording_no_socket_blocks_creating_one_beside_it(self): + """Its directory hash cannot be inverted, so it cannot be ruled out as this socket's. + + Creating a canonical database next to it hides it exactly the way a third store hides + two contested ones, and reporting it through doctor alone did not help: ordinary + commands do not run doctor. + """ + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + stranger = os.path.join(root, "fedcba9876543210") + os.makedirs(stranger) + Store(Path(stranger) / "relay.sqlite3").close() + + chosen = resolve_state_dir(None, "/run/first-after-upgrade.sock") + + self.assertEqual(list(chosen.unidentified), [stranger]) + self.assertEqual(chosen.ambiguous, ()) + self.assertEqual(list(chosen.to_record()["unidentified"]), [stranger]) + + def test_an_existing_canonical_store_settles_the_question_already(self): + """The refusal is about CREATING one. A store that is already here has answered.""" + from codex_session_relay.store import socket_scope + + root = os.path.join(self.home, ".local", "state", "codex-session-relay") + stranger = os.path.join(root, "fedcba9876543211") + os.makedirs(stranger) + Store(Path(stranger) / "relay.sqlite3").close() + socket = "/run/already-here.sock" + mine = os.path.join(root, socket_scope(socket)) + os.makedirs(mine) + Store(Path(mine) / "relay.sqlite3", socket_path=socket).close() + + chosen = resolve_state_dir(None, socket) + + self.assertEqual(str(chosen.path), mine) + self.assertEqual(chosen.unidentified, ()) + + def test_a_fresh_socket_uses_the_canonical_directory(self): + """The fallback is for an existing store only; nothing new lands in the old name.""" + from codex_session_relay.store import socket_scope + + chosen = resolve_state_dir(None, "/run/brand-new.sock") + + self.assertEqual(chosen.socket_scope, socket_scope("/run/brand-new.sock")) + self.assertTrue(str(chosen.path).endswith(socket_scope("/run/brand-new.sock"))) + + def test_two_spellings_of_one_socket_choose_the_same_default_store(self): + """The scope registry canonicalises the socket; this hash used to take it verbatim. + + With no --state and no environment override, an alias for the socket produced a + different default state directory and therefore a different store. The registry then + refused the second invocation as a foreign owner rather than letting it join the + service already running on that socket. + """ + real = os.path.join(self.tmp, "run", "app-server.sock") + os.makedirs(os.path.dirname(real)) + open(real, "w").close() + alias_dir = os.path.join(self.tmp, "alias") + os.symlink(os.path.join(self.tmp, "run"), alias_dir) + + canonical = resolve_state_dir(None, real) + through_symlink = resolve_state_dir(None, os.path.join(alias_dir, "app-server.sock")) + + here = os.getcwd() + os.chdir(self.tmp) + try: + relative = resolve_state_dir(None, os.path.join("run", "app-server.sock")) + finally: + os.chdir(here) + + self.assertEqual(through_symlink.socket_scope, canonical.socket_scope) + self.assertEqual(relative.socket_scope, canonical.socket_scope) + self.assertEqual(through_symlink.path, canonical.path) + self.assertEqual(relative.path, canonical.path) + +class Identity(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-identity-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + self.a = os.path.join(self.tmp, "a") + os.makedirs(self.a) + self.store = Store(Path(self.a) / "relay.sqlite3") + self.addCleanup(self.store.close) + + def copy_store(self, into): + """Copy the store the way a person copying a state directory would. + + The sidecars matter: in WAL mode a recent write still lives in relay.sqlite3-wal, so + copying the main file alone produces a store that has lost it. That would make this + test pass for the wrong reason - an empty copy has no identifier to collide with. + """ + os.makedirs(into, exist_ok=True) + for suffix in ("", "-wal", "-shm"): + source = os.path.join(self.a, f"relay.sqlite3{suffix}") + if os.path.exists(source): + shutil.copy(source, os.path.join(into, f"relay.sqlite3{suffix}")) + return into + + def test_identity_is_minted_once_and_survives_reopening(self): + first = self.store.locate() + self.assertTrue(first["storeId"]) + self.store.close() + again = Store(Path(self.a) / "relay.sqlite3") + self.addCleanup(again.close) + self.assertEqual(again.identity, first["storeId"]) + self.assertEqual(again.meta("store_created_at"), first["createdAt"]) + + def test_a_copy_keeps_the_identifier_and_is_not_the_same_store(self): + """The case a stored uuid gets wrong: copying the file copies the identifier.""" + mine = self.store.locate() + b = self.copy_store(os.path.join(self.tmp, "b")) + theirs = probe(resolve_state_dir(b))["store"] + self.assertEqual(theirs["storeId"], mine["storeId"]) + self.assertNotEqual(theirs["inode"], mine["inode"]) + + # The identifier alone is satisfied by the copy, so it is never proof on its own. + self.assertEqual( + compare_store(theirs, expect_store=mine["storeId"])["sameStore"], "unproven", + ) + self.assertEqual( + compare_store( + theirs, expect_store=mine["storeId"], + expect_inode=f"{mine['device']}:{mine['inode']}", + )["sameStore"], + "mismatch", + ) + + def test_a_nonce_written_after_the_copy_separates_them(self): + b = self.copy_store(os.path.join(self.tmp, "b")) + self.assertEqual( + probe(resolve_state_dir(b))["store"]["storeId"], self.store.identity, + "the copy must be a real copy, or the nonce result proves nothing", + ) + # Written AFTER the copy on purpose: a nonce that already existed would have been + # copied too, and finding it would prove nothing. + written = self.store.write_challenge(actor="parent") + here = nonce_lookup(resolve_state_dir(self.a), written["nonce"]) + there = nonce_lookup(resolve_state_dir(b), written["nonce"]) + self.assertTrue(here["found"]) + self.assertFalse(there["found"]) + self.assertEqual(compare_store(self.store.locate(), nonce=here)["sameStore"], "proven") + self.assertEqual( + compare_store(probe(resolve_state_dir(b))["store"], nonce=there)["sameStore"], + "mismatch", + ) + + def test_a_nonce_query_that_fails_is_unreadable_rather_than_absent(self): + """Could not look is not is not there, and compare_store grades the difference. + + A locked, malformed or momentarily unavailable database opens and then fails the + query. Calling that readable turned it into a definite store mismatch, so + doctor --expect-nonce would claim two participants use different stores on the + strength of a transient read failure. + """ + broken = os.path.join(self.tmp, "broken") + os.makedirs(broken) + with open(os.path.join(broken, "relay.sqlite3"), "wb"): + pass + + answer = nonce_lookup(resolve_state_dir(broken), "any-nonce") + + self.assertFalse(answer["found"]) + self.assertFalse(answer["readable"], "the query failed; nothing was learned") + self.assertIsNotNone(answer["detail"]) + self.assertNotEqual( + compare_store(probe(resolve_state_dir(broken))["store"], nonce=answer)["sameStore"], + "mismatch", + ) + + def test_a_symlinked_directory_is_the_same_store(self): + alias = os.path.join(self.tmp, "alias") + os.symlink(self.a, alias) + mine = self.store.locate() + through_alias = probe(resolve_state_dir(alias))["store"] + self.assertNotEqual(through_alias["dbPath"], mine["dbPath"]) + self.assertEqual( + compare_store( + through_alias, expect_store=mine["storeId"], + expect_inode=f"{mine['device']}:{mine['inode']}", + )["sameStore"], + "proven", + ) + + def test_a_store_with_no_identity_is_never_proven_equal(self): + """Absence must not become agreement; an old store predates the identity rows.""" + self.assertEqual( + compare_store({"storeId": None}, expect_store="whatever")["sameStore"], "unproven", + ) + + def test_an_unreadable_nonce_is_unproven_rather_than_a_mismatch(self): + """Not being able to look is not the same as looking and finding nothing.""" + unreadable = {"nonce": "abc", "found": False, "readable": False, + "detail": "OperationalError: unable to open database file"} + graded = compare_store({"storeId": "x"}, nonce=unreadable) + self.assertEqual(graded["sameStore"], "unproven") + self.assertIn("could not be read", graded["detail"]) + absent = {"nonce": "abc", "found": False, "readable": True, "detail": None} + self.assertEqual( + compare_store({"storeId": "x"}, nonce=absent)["sameStore"], "mismatch", + ) + + +class Probe(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.mkdtemp(prefix="relay-probe-") + self.addCleanup(shutil.rmtree, self.tmp, ignore_errors=True) + + def test_a_missing_state_directory_is_an_answer_and_is_not_created(self): + absent = os.path.join(self.tmp, "nothing-here") + report = probe(resolve_state_dir(absent)) + self.assertFalse(report["access"]["directoryExists"]) + self.assertFalse(report["store"]["exists"]) + self.assertFalse(os.path.exists(absent), "probe must not create what it describes") + + def test_reported_writability_matches_what_this_process_can_really_do(self): + """Asserted against a measured attempt, because a privileged runner ignores mode bits.""" + locked = os.path.join(self.tmp, "locked") + os.makedirs(locked) + Store(Path(locked) / "relay.sqlite3").close() + os.chmod(locked, 0o500) + self.addCleanup(os.chmod, locked, 0o700) + try: + probe_file = os.path.join(locked, ".really-writable") + with open(probe_file, "w", encoding="utf-8"): + pass + os.unlink(probe_file) + really_writable = True + except OSError: + really_writable = False + report = probe(resolve_state_dir(locked)) + self.assertEqual(report["access"]["directoryWritable"], really_writable) + self.assertTrue(report["access"]["dbReadable"]) + if not really_writable: + self.assertIsNotNone(report["access"]["detail"]) + + if __name__ == "__main__": unittest.main()