From 22d6bfa0fdd5498196add6d6de0f61ed3a1445e4 Mon Sep 17 00:00:00 2001 From: James Ross Date: Fri, 13 Nov 2026 19:41:49 -0800 Subject: [PATCH 1/2] Keep devrun evidence independent of other runs and of dead runners Four findings from the review of #11 (issue #12). - Pinned runs (--commit, --history) build into their own target per worktree, -target-pinned-, and hold that target's lock from before their command starts until their container has stopped. Two pinned runs of one worktree no longer share a build, and a --worktree run can no longer land a build between a pinned run's mtime refresh and its build. --worktree runs no longer see a commit run's output either (issue #9, case 1). - A container whose runner was killed counts as a held slot, and --exclusive waits for it, found by its labels and its dead liveness record. Once its own timeout (new org.flyingrobots.timeout label, from State.StartedAt, by the daemon's clock) has passed, a waiting run or --gc stops it; nothing stops it earlier. - Slots, the exclusive gate, the fetch lock, and the pinned lock stay held until Docker confirms the container stopped or it was removed. When cleanup cannot remove it, the locks go only after the liveness record is unlocked. - Volume and container creates are recorded before Docker is asked; cleanup resolves them by listing the run's labelled resources and keeps the record when Docker cannot answer. Signals during cleanup are ignored. - _inspect accepts a non-zero exit only when every missing resource is absent from a fresh listing; otherwise --gc stops and forgets nothing. Verified: 111 unit tests in a container through the installed runner; tests/integration-devrun PASS on the host. Against the old runner the new integration sections fail (B printed built-from-P1, C printed built-from-W, the worktree run printed built-from-P2, an ordinary and an exclusive run started beside a SIGKILL survivor), and the new unit tests for the interrupted create, the failed inspect, and SIGTERM ordering fail. A mutation per fix turns a test red, including a lock without the separate pinned target (the commit run then runs the worktree's build). Agent-Tool: Claude Code --- CHANGELOG.md | 33 ++ README.md | 54 ++-- bin/devrun | 413 +++++++++++++++++++++---- docs/hygiene.md | 117 ++++++-- tests/integration-devrun | 123 ++++++++ tests/test_devrun.py | 629 ++++++++++++++++++++++++++++++++++++++- 6 files changed, 1255 insertions(+), 114 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 32b14f0..6df41b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,35 @@ version numbers yet, so changes are listed under Unreleased. ## [Unreleased] +### Fixed + +- Two `--commit` runs of different commits of one worktree, started close + together, could share a build: both copies got their time before either build + started, so cargo served the first run's binary to the second (issue #12). A + `--worktree` run between a `--commit` run's copy and its build had the same + effect. Pinned runs (`--commit`, `--history`) now build into their own target + per worktree, `-target-pinned-`, and hold that target's lock from + before their command starts until their container has stopped. Pinned runs of + one worktree now run their commands one at a time. +- A `--worktree` run after a `--commit` run of the same worktree could reuse the + commit's build for an edit older than that build (part of issue #9). The two + modes no longer share a target. +- A container whose runner was killed with SIGKILL no longer escapes the run cap: + it counts as a held slot, and `--exclusive` runs wait for it, until it stops. + Once its own timeout has passed, a waiting run or `--gc` stops it. Run + containers carry two new labels for this, `org.flyingrobots.phase` and + `org.flyingrobots.timeout`. +- A run keeps its slot, the exclusive gate, the fetch lock, and the pinned lock + until Docker confirms that its container has stopped or it has been removed; + on SIGTERM they were released before cleanup had removed the container. +- A signal between Docker creating a volume or container and devrun recording it + no longer leaves an untracked resource: devrun records the intent first, and + cleanup lists the run's labelled resources to resolve it, keeping the run's + liveness record when Docker cannot answer. A signal during cleanup is ignored. +- `--gc` no longer forgets a dead run when `docker inspect` fails: a failed + inspect is accepted only when each missing resource is also missing from a + fresh listing. + ### Changed - `bin/devrun` no longer takes the machine-wide `host/heavy-work` lock or a @@ -21,6 +50,10 @@ version numbers yet, so changes are listed under Unreleased. (`-target-evidence-`) and is removed with it. - `--commit` runs give the copied files the current time, so an older commit cannot reuse output built from a newer one (part of issue #9). +- Existing `-target-` volumes stay the targets of `--worktree` + runs. They may still hold output of earlier `--commit` runs, which the first + `--worktree` run after the upgrade can reuse for files older than that output; + remove such a target with `docker volume rm` if that matters. - Each run registers `host/docker/-dev-run-/` in the workstation git-locks store for visibility; a missing or broken git-locks no longer stops a run. diff --git a/README.md b/README.md index 9f03420..8faaa17 100644 --- a/README.md +++ b/README.md @@ -70,33 +70,44 @@ Runs do not wait for each other unless the machine is full. The rules are in every exit path, including SIGTERM and SIGINT. - At most 4 command phases run at once on the machine. `--max-runs N` or `DEVRUN_MAX_RUNS=N` sets the cap; a waiting run prints how long it has waited, - and gives up after `--wait SECONDS` (default 3600). + and gives up after `--wait SECONDS` (default 3600). A run keeps its slot until + its container has stopped. +- A command container whose runner was killed (SIGKILL) keeps counting against + the cap, and `--exclusive` runs wait for it, until it ends. Once its own + `--timeout` has passed, the next run that waits for it (or `--gc`) stops it; + nothing stops it earlier. - `--exclusive` waits until no other command phase runs and keeps them all out until it finishes. Use it for every performance measurement: `bin/devrun echo ~/git/echo --commit HEAD --exclusive -- cargo bench --locked`. - Fetch phases of one project take turns (cargo's own cache lock cannot work on the read-only root). +- `--commit` and `--history` runs of one worktree take turns on that worktree's + pinned build target (below). - With `CI=true` none of this coordination applies; CI machines run one job. - Runs register `host/docker/-dev-run-/` in the workstation git-locks store for visibility; without git-locks they run unregistered. ### Build output and disk -The iteration build output lives in one volume per worktree, -`-target-`, where the key is a hash of the worktree's real path. -Each worktree keeps its own warm cache and never sees another worktree's -artifacts. Two runs from the same worktree share it safely: cargo locks its build -directory. The cost is disk: each target can grow to a full build of the -workspace, so eight worktrees of a large workspace can hold several times the -20 GiB per-project budget. `--evidence` builds into a new, empty volume that -belongs to the one run and is removed with it. - -`--commit` runs give the copied files the current time, so an older commit never -reuses output built from a newer one in the same worktree's target. `--worktree` -runs keep the files' own times, like cargo on the host. A `--worktree` run after a -`--commit` run of the same worktree can still reuse the commit's output for files -that are older than that run (flyingrobots/devcontainers#9); touch the changed -files, or use `--evidence`, when that matters. +Build output lives in kept volumes per worktree, keyed by a hash of the +worktree's real path: `-target-` for `--worktree` runs and +`-target-pinned-` for `--commit` and `--history` runs. Each +worktree keeps its own warm caches and never sees another worktree's artifacts. +Two `--worktree` runs from the same worktree share their target safely: cargo +locks its build directory. The cost is disk: each target can grow to a full +build of the workspace, so eight worktrees of a large workspace can hold several +times the 20 GiB per-project budget. `--evidence` builds into a new, empty volume +that belongs to the one run and is removed with it. + +A pinned run's result does not depend on what else runs: its copy gets the +current time (`--commit`) or is checked out in the container (`--history`), and +no other run can build in the pinned target until its command has ended, so +cargo rebuilds the workspace's crates from this run's source and reuses only +registry and git dependencies. Pinned runs of one worktree therefore run their +commands one at a time. `--worktree` runs keep the files' own times, like cargo +on the host, and never see a pinned run's output; a tool that restores old +modification times can still make them reuse stale output +(flyingrobots/devcontainers#9). ### Leftovers: `bin/devrun --gc` @@ -108,10 +119,13 @@ bin/devrun --gc bunny # remove it `--gc` removes the stopped containers and the volumes of runs that died without cleaning up (killed with SIGKILL, or a crash), and the targets of worktrees that no longer exist. A run counts as dead only when its liveness lock in the devrun -state directory is free, which proves its process exited. `--gc` never removes a -running container, a volume in use, the cargo caches, or anything without a -valid run id or source label, such as the fixed-name volumes of older runner -versions (remove those by hand once nothing uses them). +state directory is free, which proves its process exited. A dead run's container +that is still running is stopped first if its own timeout has passed, and kept +otherwise. `--gc` never removes a running container, a volume in use, the cargo +caches, or anything without a valid run id or source label, such as the +fixed-name volumes of older runner versions (remove those by hand once nothing +uses them). If Docker cannot inspect a resource that still exists, `--gc` stops +with an error and forgets no run. Test it: diff --git a/bin/devrun b/bin/devrun index 81ff613..cb2a76e 100755 --- a/bin/devrun +++ b/bin/devrun @@ -19,6 +19,7 @@ Usage: from __future__ import annotations import argparse +import datetime import fcntl import hashlib import io @@ -48,6 +49,8 @@ SHA_RE = re.compile(r"[0-9a-f]{7,40}") RUN_ID_LABEL = "org.flyingrobots.run-id" SOURCE_PATH_LABEL = "org.flyingrobots.source-path" SOURCE_KEY_LABEL = "org.flyingrobots.source-key" +PHASE_LABEL = "org.flyingrobots.phase" +TIMEOUT_LABEL = "org.flyingrobots.timeout" RUN_ID_RE = re.compile(r"[0-9a-f]{12}") SOURCE_KEY_RE = re.compile(r"[0-9a-f]{16}") LOCK_STORE = "/Users/Shared/git-locks/workstation.git" @@ -59,6 +62,8 @@ MAX_RUNS_LIMIT = 64 DEFAULT_WAIT = 3600 # Roles of the volumes that belong to one run and are removed by it (docs/hygiene.md section 5). PER_RUN_ROLES = ("work", "scratch", "target-evidence") +# Roles of the build targets kept per worktree: one for --worktree runs, one for pinned runs. +TARGET_ROLES = ("target", "target-pinned") class DevrunError(Exception): @@ -414,8 +419,11 @@ def source_key(top: str) -> str: return hashlib.sha256(os.fsencode(top)).hexdigest()[:16] -def target_volume_name(project: str, key: str) -> str: - """The iteration target of one source tree (one git worktree).""" +def target_volume_name(project: str, key: str, role: str = "target") -> str: + """A kept build target of one source tree (one git worktree): role "target" for --worktree + runs, "target-pinned" for --commit and --history runs (docs/hygiene.md section 5).""" + if role == "target-pinned": + return f"{project}-target-pinned-{key}" return f"{project}-target-{key}" @@ -489,6 +497,12 @@ class Waiter: self.start = clock() self.next_report = self.start + def because(self, what: str) -> None: + """Change what the wait is for; a new reason is reported at the next poll.""" + if what != self.what: + self.what = what + self.next_report = self.clock() + def wait(self) -> None: now = self.clock() waited = now - self.start @@ -521,6 +535,16 @@ class Admission: self.release() +def survivor_names(survivors: List[dict]) -> str: + return ", ".join(sorted(s["name"] for s in survivors)) + + +def counts_as_command(survivor: dict) -> bool: + """A survivor's container runs a command phase, or is of an older runner that did not label + its phase (then it is treated as one).""" + return survivor.get("phase") in ("command", None) + + class Gate: """At most `max_runs` command phases at once; an exclusive run has the machine to itself. @@ -533,16 +557,23 @@ class Gate: slot- an ordinary run holds one of slot-0 .. slot-. An ordinary run takes the three in one attempt or none, and retries; it never holds the shared lock while it waits for a slot, so it cannot delay an exclusive run that is already waiting. + + A command container can outlive its runner (SIGKILL): its locks are gone but it still runs. + `survivors()` lists such containers (Survivors); each one whose phase is the command + counts as a held slot, and an exclusive run waits until none is left. It is called only once + the locks would admit the run, so a waiting run does not ask Docker on every poll. """ def __init__(self, directory: Path, max_runs: int, clock: Callable[[], float] = time.monotonic, sleep: Callable[[float], None] = time.sleep, report: Callable[[str], None] = _report_stderr, - poll: float = 0.5, report_every: float = 30.0): + poll: float = 0.5, report_every: float = 30.0, + survivors: Optional[Callable[[], List[dict]]] = None): if not 1 <= max_runs <= MAX_RUNS_LIMIT: raise DevrunError(f"the run cap must be between 1 and {MAX_RUNS_LIMIT}") self.directory, self.max_runs = directory, max_runs self.clock, self.sleep, self.report = clock, sleep, report self.poll, self.report_every = poll, report_every + self.survivors = survivors or (lambda: []) def waiter(self, what: str, limit: Optional[float], poll: Optional[float] = None) -> Waiter: return Waiter(what, limit, self.clock, self.sleep, self.report, poll or self.poll, self.report_every) @@ -566,29 +597,49 @@ class Gate: w = self.waiter("exclusive access (another exclusive run is queued or running)", limit) while not _try_lock(turnstile, fcntl.LOCK_EX): w.wait() - w.what = "exclusive access (ordinary runs are finishing; new ones wait behind this run)" + w.because("exclusive access (ordinary runs are finishing; new ones wait behind this run)") while not _try_lock(shared, fcntl.LOCK_EX): w.wait() - return Admission(opened) + # Every admitted run has finished; a container whose runner died may still be running. + while True: + left = [s for s in self.survivors() if counts_as_command(s)] + if not left: + return Admission(opened) + w.because(f"exclusive access (the runner of {survivor_names(left)} is dead but its command " + f"is still running; new runs wait behind this run)") + w.wait() def _admit_ordinary(self, turnstile: int, shared: int, opened: List[int], limit: Optional[float]) -> Admission: slots = [] for i in range(self.max_runs): slots.append(_open_lock(self.directory / f"slot-{i}")) opened.append(slots[-1]) - w = self.waiter(f"a run slot (at most {self.max_runs} at once, or an exclusive run is queued)", limit) + plain = f"a run slot (at most {self.max_runs} at once, or an exclusive run is queued)" + w = self.waiter(plain, limit) while True: - got = None + got: List[int] = [] if _try_lock(turnstile, fcntl.LOCK_EX): try: if _try_lock(shared, fcntl.LOCK_SH): - got = next((fd for fd in slots if _try_lock(fd, fcntl.LOCK_EX)), None) - if got is None: + # Take every free slot for a moment, to count them: ordinary attempts pass + # the turnstile one at a time, so no other run takes a slot meanwhile. + got = [fd for fd in slots if _try_lock(fd, fcntl.LOCK_EX)] + left = [s for s in self.survivors() if counts_as_command(s)] if got else [] + if len(got) <= len(left): + w.because(f"a run slot (at most {self.max_runs} at once; the runner of " + f"{survivor_names(left)} is dead but its command still runs)" if left else plain) + for fd in got: + fcntl.flock(fd, fcntl.LOCK_UN) + got = [] fcntl.flock(shared, fcntl.LOCK_UN) + else: + for fd in got[1:]: + fcntl.flock(fd, fcntl.LOCK_UN) + got = got[:1] finally: fcntl.flock(turnstile, fcntl.LOCK_UN) - if got is not None: - keep = [shared, got] + if got: + keep = [shared, got[0]] for fd in opened: if fd not in keep: os.close(fd) @@ -596,13 +647,22 @@ class Gate: return Admission(list(keep)) w.wait() - def hold(self, name: str, what: str, limit: Optional[float]) -> Admission: - """An exclusive lock on one named file, such as a project's fetch lock.""" + def hold(self, name: str, what: str, limit: Optional[float], + blocking: Optional[Callable[[dict], bool]] = None) -> Admission: + """An exclusive lock on one named file, such as a project's fetch lock. With `blocking`, + the lock is also not taken while a survivor for which blocking(survivor) is true runs.""" fd = _open_lock(self.directory / name) try: w = self.waiter(what, limit) while not _try_lock(fd, fcntl.LOCK_EX): w.wait() + while blocking: + left = [s for s in self.survivors() if blocking(s)] + if not left: + break + w.because(f"{what.split(' (')[0]} (the runner of {survivor_names(left)} is dead " + f"but its container still runs)") + w.wait() except BaseException: os.close(fd) raise @@ -676,6 +736,98 @@ def live_run_facts(directory: Path) -> Optional[List[dict]]: return facts +def parse_docker_time(text: str) -> float: + """Seconds since the epoch for a time as Docker prints it (RFC 3339 with up to nanoseconds, + such as 2026-11-14T03:19:46.51321438Z or 2026-11-13T19:21:36.244827538-08:00).""" + m = re.fullmatch(r"(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d+))?(Z|[+-]\d\d:\d\d)", text.strip()) + if not m: + raise DevrunError(f"cannot read the Docker time {text!r}") + base, frac, zone = m.groups() + stamp = datetime.datetime.fromisoformat(base + ("+00:00" if zone == "Z" else zone)).timestamp() + return stamp + (float("0." + frac) if frac else 0.0) + + +def docker_now() -> float: + """The Docker daemon's current time, so that a container's start time is compared with the + clock that wrote it.""" + p = run(["docker", "info", "--format", "{{json .SystemTime}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker info failed: {p.stderr.decode(errors='replace').strip()}") + return parse_docker_time(json_loads(p.stdout.decode())) + + +class Survivors: + """Running containers of this tool whose runner is dead (docs/hygiene.md section 10). + + A container counts only when it carries the role dev-run, a valid owner and run id label, a + name that matches them, and a liveness record in this state directory that --gc's test proves + dead. A container of a run in another state directory is of unknown state and never counts. + With `stop_overdue`, a survivor whose own timeout (its timeout label, from its start, by the + daemon's clock) has passed is killed, and is no longer listed once Docker confirms it stopped: + that is the timeout its runner would have enforced. A survivor without a timeout label (from + an older runner) is never stopped, only waited for. + """ + + def __init__(self, directory: Path, project: Optional[str] = None, stop_overdue: bool = True, + report: Callable[[str], None] = _report_stderr, now: Callable[[], float] = docker_now): + self.directory, self.project, self.stop_overdue = directory, project, stop_overdue + self.report, self.now = report, now + + def scan(self) -> List[dict]: + filters = ["--filter", f"label={ROLE_LABEL}=dev-run", "--filter", f"label={RUN_ID_LABEL}"] + if self.project: + filters += ["--filter", f"label={OWNER_LABEL}={self.project}"] + p = run(["docker", "ps", "-q", "--no-trunc", *filters], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") + found = [] + for d in _inspect(["container"], p.stdout.decode().split(), all_container_ids): + labels = d["Config"].get("Labels") or {} + project, rid = labels.get(OWNER_LABEL) or "", labels.get(RUN_ID_LABEL) or "" + name = d["Name"].lstrip("/") + if (labels.get(ROLE_LABEL) != "dev-run" or not PROJECT_RE.fullmatch(project) or not RUN_ID_RE.fullmatch(rid) + or name != run_resource_names(project, rid)["container"] or not d["State"]["Running"]): + continue + if run_liveness(self.directory, rid) != "dead": + continue + found.append({"id": d["Id"], "name": name, "project": project, "run_id": rid, + "phase": labels.get(PHASE_LABEL), "timeout": labels.get(TIMEOUT_LABEL), + "started": d["State"].get("StartedAt") or "", + "mounts": [m.get("Name") for m in d.get("Mounts") or [] if m.get("Type") == "volume"]}) + return found + + def deadline(self, survivor: dict) -> Optional[float]: + """When the survivor's own timeout ends, by the daemon's clock; None without a timeout label.""" + timeout = survivor.get("timeout") or "" + if not timeout.isdigit() or not survivor.get("started"): + return None + return parse_docker_time(survivor["started"]) + int(timeout) + + def stop(self, survivor: dict) -> bool: + """Kill one overdue survivor by its id; True once Docker confirms that it stopped.""" + self.report(f"devrun: stopping {survivor['name']}: its runner is dead and its " + f"{survivor['timeout']} s timeout has passed") + run(["docker", "kill", survivor["id"]], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + try: + p = run(["docker", "wait", survivor["id"]], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=60) + except subprocess.TimeoutExpired: + return False + return p.returncode == 0 + + def __call__(self) -> List[dict]: + found = self.scan() + if not self.stop_overdue or not any(self.deadline(s) is not None for s in found): + return found + now = self.now() + left = [] + for s in found: + end = self.deadline(s) + if end is not None and now >= end and self.stop(s): + continue + left.append(s) + return left + + # --------------------------------------------------------------------------- # # git-locks visibility (docs/hygiene.md section 10) # --------------------------------------------------------------------------- # @@ -743,14 +895,21 @@ class Docker: self.memory = memory self.names = run_resource_names(project, run_id) self.name = self.names["container"] - # True only between a successful `docker create` by this run and its removal, so cleanup - # never touches a container that this run did not create. - self.created = False - # Per-run volumes this run created and has not removed yet; cleanup removes only these. + # The run container: "absent", "uncertain" (a create was sent and its outcome is not + # known, for example because a signal interrupted it), or "created". Cleanup removes it + # only when this run created it, resolving "uncertain" by listing this run's labels. + self.state = "absent" + # Per-run volumes that may exist because this run asked Docker to create them, and have + # not been removed yet; cleanup removes only these. A name is added before the create is + # sent, and stays in `unconfirmed` until the create is confirmed. self.volumes: List[str] = [] + self.unconfirmed: Set[str] = set() + # Locks (a run slot, the exclusive gate, the fetch or pinned-build lock) that must stay held + # until the running container is confirmed stopped or removed (docs/hygiene.md section 10). + self.holds: List[Admission] = [] - def labels(self, role: str) -> List[str]: - return label_args(resource_labels(self.project, role, self.run_id)) + def labels(self, role: str, extra: Optional[Dict[str, str]] = None) -> List[str]: + return label_args(resource_labels(self.project, role, self.run_id, extra)) def _volume_labels(self, name: str) -> Optional[dict]: p = run(["docker", "volume", "inspect", name, "--format", "{{json .Labels}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) @@ -758,6 +917,17 @@ class Docker: return None return json_loads(p.stdout.decode() or "null") or {} + def _listed(self, kind: str) -> Optional[Set[str]]: + """The names of this run's own containers ("container") or volumes ("volume") that exist, + selected by this run's owner and run id labels; None when Docker cannot list them.""" + cmd = (["docker", "ps", "-a", "--format", "{{.Names}}"] if kind == "container" + else ["docker", "volume", "ls", "--format", "{{.Name}}"]) + p = run([*cmd, "--filter", f"label={OWNER_LABEL}={self.project}", "--filter", f"label={RUN_ID_LABEL}={self.run_id}"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + return None + return set(p.stdout.decode().split()) + def ensure_volume(self, name: str, role: str, extra: Optional[Dict[str, str]] = None) -> None: """Create a shared volume, or check that the existing one is ours (and, for a target, that it is the target of this source tree). Two runs may create it at the same moment: the @@ -778,17 +948,23 @@ class Docker: name = self.names[role] if self._volume_labels(name) is not None: raise DevrunError(f"refused: volume {name} already exists (run id collision, or litter)") + # From here on the volume may exist, even if the create below is interrupted. + self.volumes.append(name) + self.unconfirmed.add(name) p = run(["docker", "volume", "create", *self.labels(role), name], stdout=subprocess.PIPE, stderr=subprocess.PIPE) if p.returncode != 0: raise DevrunError(f"docker volume create {name} failed: {p.stderr.decode(errors='replace').strip()}") - self.volumes.append(name) + self.unconfirmed.discard(name) return name def create(self, cmd: List[str], network: bool, target_volume: str, env: Optional[dict] = None, - caches_writable: bool = False) -> None: - """Create the run container. The cargo caches are writable only in the fetch phase.""" + caches_writable: bool = False, phase: str = "command", timeout: int = 300) -> None: + """Create the run container. The cargo caches are writable only in the fetch phase. The + phase and the timeout are labels, so that another run can recognize a container whose + runner died and enforce its timeout (Survivors).""" ro = "" if caches_writable else ",readonly" - args = ["docker", "create", "--name", self.name, *self.labels("dev-run"), + args = ["docker", "create", "--name", self.name, + *self.labels("dev-run", {PHASE_LABEL: phase, TIMEOUT_LABEL: str(timeout)}), "--user", f"{UID}:{GID}", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--read-only", "--tmpfs", "/tmp:rw,nosuid,nodev,size=1g,mode=1777", "--pids-limit", "4096", "--memory", self.memory, "--memory-swap", self.memory, "--cpus", self.cpus, @@ -811,10 +987,12 @@ class Docker: for k, v in (env or {}).items(): args += ["--env", f"{k}={v}"] args += [self.image, *cmd] + # From here on the container may exist, even if the create below is interrupted. + self.state = "uncertain" p = run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) if p.returncode != 0: raise DevrunError(f"docker create failed: {p.stderr.decode(errors='replace').strip()}") - self.created = True + self.state = "created" def copy_in(self, tar_path: Path, dest: str) -> None: with open(tar_path, "rb") as fh: @@ -822,14 +1000,30 @@ class Docker: if p.returncode != 0: raise DevrunError(f"docker cp into {dest} failed: {p.stderr.decode(errors='replace').strip()}") - def start(self, timeout: int) -> int: + def start(self, timeout: int, holds: Iterable[Admission] = ()) -> int: + """Run the created container to its end. The `holds` stay held until the container is + confirmed stopped here, or removed by remove() or cleanup(): a timeout, a signal, or an + error in between never releases them while the container may still run.""" + self.holds.extend(holds) try: p = run(["docker", "start", "-a", self.name], timeout=timeout) except subprocess.TimeoutExpired: run(["docker", "kill", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) raise DevrunError(f"timeout after {timeout}s; container killed") + if self.holds and self.stopped(): + self.release_holds() return p.returncode + def stopped(self) -> bool: + """True when Docker confirms that the run container exists and is not running.""" + p = run(["docker", "container", "inspect", "--format", "{{.State.Running}}", self.name], + stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) + return p.returncode == 0 and p.stdout.strip() == b"false" + + def release_holds(self) -> None: + while self.holds: + self.holds.pop().release() + def export(self, path: str, dest: Path) -> List[str]: p = subprocess.Popen(["docker", "cp", f"{self.name}:/scratch/out/{path}", "-"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) try: @@ -842,23 +1036,47 @@ class Docker: return written def remove(self) -> None: - """Remove the run container, only if this run created it.""" - if not self.created: - return - p = run(["docker", "rm", "-f", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - if p.returncode == 0: - self.created = False + """Remove the run container, only if this run created it, and then release the holds.""" + if self.state == "uncertain": + listed = self._listed("container") + if listed is None: + return # cannot tell; keep the holds, and cleanup reports the run as not clean + self.state = "created" if self.name in listed else "absent" + if self.state == "created": + p = run(["docker", "rm", "-f", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + if p.returncode == 0: + self.state = "absent" + else: + listed = self._listed("container") + if listed is not None and self.name not in listed: + self.state = "absent" + if self.state == "absent": + self.release_holds() def cleanup(self) -> bool: - """Remove this run's container and volumes. True when nothing this run created is left.""" + """Remove this run's container and volumes. True when nothing this run created is left. + The holds are released only once the container is gone.""" self.remove() + listed: Optional[Set[str]] = None for name in list(self.volumes): + if name in self.unconfirmed: + # The create may or may not have happened: a fresh listing of this run's labelled + # volumes decides. A volume that is not listed was never created by this run. + listed = self._listed("volume") if listed is None else listed + if listed is None: + print(f"devrun: cannot tell whether volume {name} was created; keeping the run record", file=sys.stderr) + continue + if name not in listed: + self.volumes.remove(name) + self.unconfirmed.discard(name) + continue p = run(["docker", "volume", "rm", name], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE) if p.returncode == 0: self.volumes.remove(name) + self.unconfirmed.discard(name) else: print(f"devrun: could not remove volume {name}: {p.stderr.decode(errors='replace').strip()}", file=sys.stderr) - return not self.created and not self.volumes + return self.state == "absent" and not self.volumes def exists(self) -> bool: p = run(["docker", "ps", "-a", "--filter", f"name=^{self.name}$", "--format", "{{.Names}}"], stdout=subprocess.PIPE) @@ -883,8 +1101,8 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: liveness(run_id) -> 'alive' | 'dead' | 'unknown'; only 'dead' allows removal. protected_keys: source keys of live runs' targets, or None when that is unknown (then no target volume is removed). - Selected: stopped containers and per-run volumes of dead runs, and target volumes whose source - path no longer exists. Never selected: anything without this project's owner label, anything + Selected: stopped containers and per-run volumes of dead runs, and target volumes (worktree or + pinned) whose source path no longer exists. Never selected: anything without this project's owner label, anything whose name does not match its labels, anything attached to a running container or to a container that is not itself selected, the cargo caches, and legacy fixed-name volumes. """ @@ -908,8 +1126,9 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: if state != "dead": kept.append((c["name"], f"run {rid} is {state}")) elif c.get("running"): - kept.append((c["name"], f"run {rid} is dead but its container is still running; " - f"stop it with docker stop {c['name']} and run --gc again")) + kept.append((c["name"], f"run {rid} is dead but its container is still running, within its timeout; " + f"devrun stops it when that has passed, or stop it with docker stop {c['name']}, " + f"then run --gc again")) else: remove_containers.append(c["name"]) @@ -940,12 +1159,12 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: kept.append((name, problem)) else: remove_volumes.append(name) - elif role == "target": + elif role in TARGET_ROLES: path, key = labels.get(SOURCE_PATH_LABEL), labels.get(SOURCE_KEY_LABEL) or "" if not path: kept.append((name, "no source path label (legacy per-project target); not touched")) continue - if not SOURCE_KEY_RE.fullmatch(key) or key != source_key(path) or name != target_volume_name(project, key): + if not SOURCE_KEY_RE.fullmatch(key) or key != source_key(path) or name != target_volume_name(project, key, role): kept.append((name, "its name and source labels do not match; not touched")) continue if path_exists(path): @@ -962,27 +1181,58 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: return {"containers": remove_containers, "volumes": remove_volumes, "kept": kept} -def _inspect(kind: List[str], names: List[str]) -> List[dict]: +def all_container_ids() -> Set[str]: + p = run(["docker", "ps", "-aq", "--no-trunc"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") + return set(p.stdout.decode().split()) + + +def project_volume_names(project: str) -> Set[str]: + p = run(["docker", "volume", "ls", "-q", "--filter", f"label={OWNER_LABEL}={project}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker volume ls failed: {p.stderr.decode(errors='replace').strip()}") + return set(p.stdout.decode().split()) + + +def _inspect(kind: List[str], names: List[str], listing: Callable[[], Set[str]]) -> List[dict]: + """docker inspect of the named resources (container ids, or volume names). + + A resource removed between the listing and the inspect makes the inspect exit 1 while it still + prints the others. That is accepted only when every requested name missing from the output is + also missing from a fresh `listing()`. Any other failure raises, so that a resource that still + exists is never taken for an absent one (--gc would then forget a dead run that has one). + """ if not names: return [] p = run(["docker", *kind, "inspect", *names], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - # A resource removed between the listing and the inspect makes the inspect exit 1 but still - # prints the others. - return json_loads(p.stdout.decode() or "[]") if p.stdout.strip() else [] + what = f"docker {' '.join(kind)} inspect" + err = p.stderr.decode(errors="replace").strip() + try: + found = json_loads(p.stdout.decode()) if p.stdout.strip() else [] + except ValueError: + raise DevrunError(f"{what} printed something that is not JSON: {err}") + if not isinstance(found, list): + raise DevrunError(f"{what} printed something that is not a list: {err}") + key = "Id" if kind == ["container"] else "Name" + printed = {d.get(key) for d in found if isinstance(d, dict)} + missing = [n for n in names if n not in printed] + if p.returncode != 0 or missing: + if not missing: + raise DevrunError(f"{what} failed: {err}") + still = listing() & set(missing) + if still: + raise DevrunError(f"{what} failed for {', '.join(sorted(still))}, which still exist: {err}") + return found def docker_inventory(project: str) -> tuple: - p = run(["docker", "ps", "-aq", "--no-trunc"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - if p.returncode != 0: - raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") containers = [{"name": d["Name"].lstrip("/"), "running": bool(d["State"]["Running"]), "labels": d["Config"].get("Labels") or {}, "mounts": [m.get("Name") for m in d.get("Mounts") or [] if m.get("Type") == "volume"]} - for d in _inspect(["container"], p.stdout.decode().split())] - p = run(["docker", "volume", "ls", "-q", "--filter", f"label={OWNER_LABEL}={project}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - if p.returncode != 0: - raise DevrunError(f"docker volume ls failed: {p.stderr.decode(errors='replace').strip()}") - volumes = [{"name": d["Name"], "labels": d.get("Labels") or {}} for d in _inspect(["volume"], p.stdout.decode().split())] + for d in _inspect(["container"], sorted(all_container_ids()), all_container_ids)] + volumes = [{"name": d["Name"], "labels": d.get("Labels") or {}} + for d in _inspect(["volume"], sorted(project_volume_names(project)), lambda: project_volume_names(project))] return containers, volumes @@ -996,6 +1246,17 @@ def gc_main(argv: List[str]) -> int: directory = state_dir() live = live_run_facts(directory) protected = None if live is None else {f.get("source_key") for f in live if f.get("project") == ns.project} + # A dead run's container that is past its own timeout is stopped first (dry run: only said). + survivors = Survivors(directory, project=ns.project, stop_overdue=not ns.dry_run) + if ns.dry_run: + found = survivors() + now = docker_now() if any(survivors.deadline(s) is not None for s in found) else 0.0 + for s in found: + end = survivors.deadline(s) + if end is not None and now >= end: + print(f"devrun gc: would stop container {s['name']}: its runner is dead and its {s['timeout']} s timeout has passed") + else: + survivors() containers, volumes = docker_inventory(ns.project) plan = plan_gc(ns.project, containers, volumes, lambda rid: run_liveness(directory, rid), os.path.isdir, protected) verb = "would remove" if ns.dry_run else "remove" @@ -1116,7 +1377,7 @@ def fetch_environment(env: dict, rust_version: str) -> dict: def prepare(docker: "Docker", target_volume: str, empty_target: bool) -> None: """Empty the copy and scratch volumes (and the evidence target when asked), offline.""" - docker.create(["bash", "-c", PREP_SCRIPT, "prep", "1" if empty_target else "0"], False, target_volume) + docker.create(["bash", "-c", PREP_SCRIPT, "prep", "1" if empty_target else "0"], False, target_volume, phase="prep") if docker.start(300) != 0: raise DevrunError("preparing the work and scratch volumes failed") docker.remove() @@ -1124,7 +1385,7 @@ def prepare(docker: "Docker", target_volume: str, empty_target: bool) -> None: def copy_in(docker: "Docker", target_volume: str, tar_path: Path, dest: str, env: dict) -> None: """Extract a sanitized tar into the volumes through a short-lived, offline container.""" - docker.create(["true"], False, target_volume, env) + docker.create(["true"], False, target_volume, env, phase="copy") docker.copy_in(tar_path, dest) if docker.start(300) != 0: raise DevrunError("copy-in container failed") @@ -1142,7 +1403,7 @@ def main_inner(ns: argparse.Namespace) -> int: # CI machines run one job at a time and have no lock store: no slots and no registration there. coordinated = os.environ.get("CI") != "true" directory = state_dir() - gate = Gate(directory, ns.max_runs) + gate = Gate(directory, ns.max_runs, survivors=Survivors(directory)) record = RunRecord(directory, run_id, {"project": ns.project, "source_key": key, "pid": str(os.getpid())}) lease = Lease(f"devrun-{ns.project}-{run_id}", f"devrun-{run_id}", f"host/docker/{docker.name}/", 2 * ns.timeout + 2 * ns.wait + 1500, f"devrun {ns.project} run {run_id}") @@ -1158,6 +1419,7 @@ def main_inner(ns: argparse.Namespace) -> int: signal.signal(signal.SIGTERM, on_signal) signal.signal(signal.SIGINT, on_signal) clean = False + no_hold = Admission([]) try: if coordinated: lease.claim(gate.waiter("host/docker/ in the lock store (daemon maintenance?)", ns.wait, poll=2.0)) @@ -1173,11 +1435,16 @@ def main_inner(ns: argparse.Namespace) -> int: raise DevrunError(f"refused: container {docker.name} already exists (run id collision, or litter)") docker.ensure_volume(f"{ns.project}-cargo-registry", "cargo-registry") docker.ensure_volume(f"{ns.project}-cargo-git", "cargo-git") + # A pinned run (--commit, --history) builds into the worktree's pinned target, never into + # the one that --worktree runs use, and holds that target's lock from before its command + # starts until its container has stopped (docs/hygiene.md section 5). + pinned = mode in ("commit", "history") and not ns.evidence if ns.evidence: target_volume = docker.create_run_volume("target-evidence") else: - target_volume = target_volume_name(ns.project, key) - docker.ensure_volume(target_volume, "target", {SOURCE_PATH_LABEL: str(top), SOURCE_KEY_LABEL: key}) + target_role = "target-pinned" if pinned else "target" + target_volume = target_volume_name(ns.project, key, target_role) + docker.ensure_volume(target_volume, target_role, {SOURCE_PATH_LABEL: str(top), SOURCE_KEY_LABEL: key}) docker.create_run_volume("work") docker.create_run_volume("scratch") env = {"DEVRUN_HEAD": facts.get("sha") or "", "DEVRUN_BASE": facts.get("base") or ""} @@ -1190,15 +1457,17 @@ def main_inner(ns: argparse.Namespace) -> int: fetch_copy(fin, fout) prepare(docker, target_volume, ns.evidence) copy_in(docker, target_volume, fetch_tar, "/work", env) - docker.create(FETCH_COMMAND, True, target_volume, fetch_environment(env, ns.rust), caches_writable=True) + docker.create(FETCH_COMMAND, True, target_volume, fetch_environment(env, ns.rust), caches_writable=True, + phase="fetch", timeout=ns.timeout) # Cargo's own package-cache lock is $CARGO_HOME/.package-cache, on the read-only root: # cargo warns that it cannot open it and fetches without any lock. Two fetches into the # project's shared caches would then write the same files at once, so fetch phases of # one project take turns on a host file lock (docs/hygiene.md section 10). - fetch_lock = (gate.hold(f"fetch-{ns.project}.lock", f"the {ns.project} fetch lock (another run is fetching)", ns.wait) - if coordinated else Admission([])) - with fetch_lock: - fetched = docker.start(ns.timeout) + # A fetch container whose runner died still writes the caches, so it holds the lock too. + fetch_lock = (gate.hold(f"fetch-{ns.project}.lock", f"the {ns.project} fetch lock (another run is fetching)", ns.wait, + blocking=lambda s: s["project"] == ns.project and s.get("phase") in ("fetch", None)) + if coordinated else no_hold) + fetched = docker.start(ns.timeout, [fetch_lock]) if fetched != 0: raise DevrunError("cargo fetch --locked failed") docker.remove() @@ -1209,10 +1478,17 @@ def main_inner(ns: argparse.Namespace) -> int: cmd = ["bash", "-c", INNER_SCRIPT, "run", mode, "1" if ns.export_patch else "0", *ns.command] env["CARGO_NET_OFFLINE"] = "false" if ns.network else "true" env["DEVRUN_FRESH_MTIMES"] = "1" if mode == "commit" else "0" - docker.create(cmd, ns.network, target_volume, env) - admission = gate.admit(ns.exclusive, ns.wait) if coordinated else Admission([]) - with admission: - status = docker.start(ns.timeout) + docker.create(cmd, ns.network, target_volume, env, timeout=ns.timeout) + # The pinned-target lock comes first and admission second: a run never holds a slot or the + # exclusive gate while it waits for another run of its worktree. Both stay held until the + # container is confirmed stopped or removed (Docker.start, Docker.remove). + build_lock = (gate.hold(f"pinned-{ns.project}-{key}.lock", + "the pinned build target of this worktree (another --commit or --history run of it is running)", + ns.wait, blocking=lambda s: target_volume in s["mounts"] and counts_as_command(s)) + if coordinated and pinned else no_hold) + docker.holds.append(build_lock) + admission = gate.admit(ns.exclusive, ns.wait) if coordinated else no_hold + status = docker.start(ns.timeout, [admission]) exported: List[str] = [] for path in ns.export: exported += docker.export(path, out) @@ -1224,9 +1500,16 @@ def main_inner(ns: argparse.Namespace) -> int: print(f"devrun: exit {status}; exports in {out}: {', '.join(exported) or 'none'}", file=sys.stderr) return status finally: + # A signal that arrives now must not cut the cleanup short either. + signal.signal(signal.SIGTERM, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.SIG_IGN) clean = docker.cleanup() lease.release() record.finish(clean) + # Locks still held here belong to a container that cleanup could not confirm gone. They + # are released only after the record is unlocked, which marks the run dead, so that other + # runs count the container as a survivor from the moment the locks are free. + docker.release_holds() if not clean: print(f"devrun: run {run_id} left resources behind; remove them with: devrun --gc {ns.project}", file=sys.stderr) for p in work.glob("*"): diff --git a/docs/hygiene.md b/docs/hygiene.md index df5fdb0..7ec9a34 100644 --- a/docs/hygiene.md +++ b/docs/hygiene.md @@ -134,10 +134,11 @@ symlinks resolved), so each git worktree has its own key. | --- | --- | --- | --- | | Base image | `flyingrobots/rust-dev:` | `base-image` | Until a new toolchain replaces it | | Upstream base, the build input | `rust@sha256:` from `images/rust-dev/base-digests`, pulled by digest. It carries only upstream labels. Keep it while its digest line exists; after removing the line, remove it with `docker image rm`. | none (upstream image) | As left | -| Run container | `-dev-run-`, also labelled with the run id | `dev-run` | One run; removed by the run | +| Run container | `-dev-run-`, also labelled with the run id, `org.flyingrobots.phase=`, and `org.flyingrobots.timeout=` (section 10 uses both when its runner has died) | `dev-run` | One run; removed by the run | | Cargo registry cache, at `/usr/local/cargo/registry` | `-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` | Kept | | Cargo git cache, at `/usr/local/cargo/git` | `-cargo-git` | `cargo-git` | Kept | -| Build output, iteration | `-target-`, also labelled `org.flyingrobots.source-path=` and `org.flyingrobots.source-key=` (one per worktree) | `target` | Kept; `devrun --gc` removes it once the worktree path no longer exists | +| Build output, `--worktree` runs | `-target-`, also labelled `org.flyingrobots.source-path=` and `org.flyingrobots.source-key=` (one per worktree) | `target` | Kept; `devrun --gc` removes it once the worktree path no longer exists | +| Build output, pinned runs (`--commit`, `--history`) | `-target-pinned-`, with the same source labels (one per worktree) | `target-pinned` | Kept; `devrun --gc` removes it once the worktree path no longer exists | | Build output, evidence | `-target-evidence-`, new and empty for each `--evidence` run, also labelled with the run id | `target-evidence` | One run; removed by the run | | The copy, at `/work` | `-work-`, also labelled with the run id | `work` | One run; removed by the run | | Inputs, logs, test data, exports, at `/scratch` | `-scratch-`, also labelled with the run id | `scratch` | One run; removed by the run | @@ -149,6 +150,23 @@ symlinks resolved), so each git worktree has its own key. - **Evidence runs use a fresh target.** A merge-gate run never reuses build output, so a stale cache cannot fabricate a passing result. The evidence target belongs to the one run, so concurrent evidence runs never share one. +- **What a pinned run guarantees about its build.** A `--commit` or `--history` + run without `--evidence` builds into its worktree's pinned target, which + `--worktree` runs never use. It takes that target's lock (section 10) before its + command container starts and keeps it until the container has stopped. Inside + the container the copy gets the current time (`--commit`) or is checked out + then (`--history`), so every source file is newer than any build output already + in the target: cargo rebuilds the workspace's own crates from this run's copy + and reuses only registry and git dependencies, which it identifies by version + and checksum rather than by time. No other run can write the target between the + time refresh and the end of the command, so the result does not depend on what + else runs. Limits: the command itself can still change the copy before it + builds, a build script can leave files in the target that a later pinned run + reads (the target is still not evidence), and with `CI=true` there is no lock, + which is safe only because a CI machine runs one job. The cost is a second + target per worktree that runs both modes, and pinned runs of one worktree run + their command phases one at a time; runs from different worktrees, and + `--worktree` runs, are not delayed. - A run refuses a shared volume (a cache or a target) that exists with other labels: another owner, another role, or, for a target, another source path or key. It refuses a per-run volume or a container that already exists under its @@ -173,14 +191,20 @@ project share only these, and each sharing has a reason: write. This is cargo's design and is not tested here. The writer still runs no repository-chosen code, so the poisoning rule above holds as before; what is new is only that a read and a write can happen at the same moment. -- **The iteration target of one worktree**, read-write, when two runs come from - the same worktree at once. Cargo's build-directory lock (`.cargo-lock` in the - target, a file lock that works across containers on one Docker volume) - serializes cargo's own builds in it; cargo prints "Blocking waiting for file - lock on build directory". A command that writes `/target` without cargo is not - serialized. One run's build scripts can change what the other run builds from, - which was already true of consecutive runs: the iteration target is never - evidence. Runs from different worktrees never share a target. +- **The `--worktree` target of one worktree**, read-write, when two + `--worktree` runs come from the same worktree at once. Cargo's build-directory + lock (`.cargo-lock` in the target, a file lock that works across containers on + one Docker volume) serializes cargo's own builds in it; cargo prints "Blocking + waiting for file lock on build directory". A command that writes `/target` + without cargo is not serialized. One run's build scripts can change what the + other run builds from, which was already true of consecutive runs: the + iteration target is never evidence. `--worktree` runs keep the files' own + times, so if a file is edited between the copies of two such runs, the run with + the older copy can reuse the other's build of the newer content; the same is + true of cargo on the host. +- **The pinned target of one worktree**, never at the same time: pinned runs of + one worktree take turns on its lock for their whole command phase (above). + Runs from different worktrees never share a target. ## 6. Outputs @@ -199,19 +223,26 @@ project share only these, and each sharing has a reason: timeout, and SIGTERM or SIGINT. After the first signal, further signals are ignored, so they cannot cut cleanup short. Cleanup removes only what this run created; the run's own records say what that is. +- A run records that a container or volume may exist before it asks Docker to + create it. When a signal interrupts a create, cleanup lists this run's labelled + resources to decide: one that exists is removed, one that does not was never + created. When Docker cannot answer, the run keeps its liveness record, so + `--gc` can still find the run. A signal that arrives during cleanup is ignored. - A run that is killed outright (SIGKILL, a host crash) cannot clean up. Its resources stay, labelled with its run id, and `devrun --gc ` removes - them (section 10). Its container may keep running until its command ends, - because the timeout lives in the killed process; `--gc` leaves a running - container alone and says so. + them (section 10). Its container may keep running, because the timeout lived + in the killed process. Until it stops it still counts against the run cap and + keeps exclusive runs out, and once its own timeout has passed a waiting run or + `--gc` stops it (section 10, "Survivors"). - The command runs under a timeout that is shorter than the `git-locks` lease. ## 8. Budgets and free space - Build output: at most 20 GiB per project. Logs and test data: at most 4 GiB, including at most 128 MiB of logs. -- The build output of a project is the sum over its iteration targets, one per - worktree, plus the evidence targets of the runs in progress. Each target can +- The build output of a project is the sum over its kept targets, up to two per + worktree (one for `--worktree` runs, one for pinned runs), plus the evidence + targets of the runs in progress. Each target can reach the size of a full debug build of the workspace, so a handful of worktrees, or a few concurrent evidence runs, can exceed 20 GiB, which a single per-project target could not. Until the measurement exists, remove the @@ -250,8 +281,12 @@ to themselves. boundary. No container can reach the state directory. - **A cap on command phases.** At most N command phases run at once (default 4; `--max-runs N` or `DEVRUN_MAX_RUNS`, 1 to 64). A run holds one of the slot - files `slot-0` to `slot-` only while its command runs; preparation, copy- - in, the fetch phase, and exports do not take a slot. A run admitted with a + files `slot-0` to `slot-` from just before its command container starts + until Docker confirms that the container has stopped, or until the run has + removed it; a timeout, a signal, or an error in between does not release the + slot earlier. Preparation, copy-in, the fetch phase, and exports do not take a + slot. A survivor (below) in its command phase counts as a held slot: a run is + admitted only when the free slots outnumber the survivors. A run admitted with a larger N can use slots that a run with a smaller N never looks at, so the cap in force is the one each run was given. A waiting run prints one line when it starts waiting and one every 30 seconds with the time waited so far, and gives @@ -271,12 +306,43 @@ to themselves. ordinary run is admitted only by passing the `turnstile` file for an instant, and a waiting exclusive run holds the turnstile until it finishes. Ordinary runs hold `runs.lock` shared and an exclusive run holds it exclusively, so the - exclusive run starts only after every admitted ordinary run has finished. + exclusive run starts only after every admitted ordinary run has finished, and + then waits until no survivor is in its command phase, saying which container + it waits for; `--wait` bounds the whole wait. - **Fetch phases of a project take turns.** Cargo's package-cache lock is the file `$CARGO_HOME/.package-cache`, and `CARGO_HOME` is on the read-only root: cargo cannot open it, warns, and fetches without a lock. Two concurrent fetches would write the same cache files, so the fetch phase holds the per-project - file lock `fetch-.lock` while it runs. + file lock `fetch-.lock` until its container has stopped or been + removed, and does not take it while a survivor of the same project is fetching. +- **Pinned runs of a worktree take turns.** A `--commit` or `--history` run + without `--evidence` holds `pinned--.lock` from before its + command container starts until that container has stopped or been removed, and + does not take it while a survivor's command container uses the same pinned + target. It takes this lock before it asks for a slot, so it never holds a slot + or the exclusive gate while it waits for another run of its worktree. Section 5 + says what this guarantees. +- **Survivors.** A survivor is a running container that carries the role + `dev-run`, a valid owner and run id, and a name that matches them, and whose + run's liveness file in this state directory is free: its runner is dead. + Admission, the fetch lock, and the pinned lock find survivors by these labels + with `docker ps` and `docker inspect`, but only once the file locks would admit + the run. A container of a run in another state directory, or of a live run, is + never a survivor. A survivor's timeout died with its runner, so nothing bounds it any + more. The rule: a waiting run, or `--gc`, stops (`docker kill`, then `docker + wait`) a survivor once its own timeout has passed, which is what its runner would + have done. The deadline is the container's `State.StartedAt` plus its + `org.flyingrobots.timeout` label, compared with the daemon's current time + (`docker info`), so host clock skew plays no part. Nothing stops a survivor + before its timeout, or a survivor without the timeout label (from an older + runner): those are waited for, and can be stopped by hand with `docker stop`. + Nothing else is ever stopped. A run whose cleanup cannot remove its container + unlocks its liveness file first and its slot and other locks after, so the + container counts as a survivor from the moment the locks are free. One gap is + left: when a runner is killed, the kernel releases its slot and its liveness + file during the same exit, and a run that passes the turnstile in that instant + can see the slot free and the runner still alive, and so start one run over + the cap. - **Run liveness.** Each run locks `runs/.lock` before it creates any Docker resource and keeps it locked until all of them are gone. The file also records the project and the source key. @@ -288,9 +354,16 @@ to themselves. about use, so neither is used; `--gc` runs only when someone asks for it. A run whose liveness file is missing (another user or another state directory) is of unknown state and is left alone; - - iteration targets whose recorded worktree path no longer exists, when the - name, the source key, and the path labels agree and no live run uses that - source key. + - worktree and pinned targets whose recorded worktree path no longer exists, + when the role, the name, the source key, and the path labels agree and no + live run uses that source key. + Before it decides, it stops the project's survivors whose own timeout has + passed (with `--dry-run` it only names them); a survivor within its timeout is + kept and named. After removing, it forgets the liveness file of a dead run only + when nothing labelled with that run id is left. When `docker inspect` fails, it + accepts the failure only if every resource missing from the output is also + missing from a fresh listing; otherwise it stops with an error and forgets + nothing, so a resource that still exists is never taken for an absent one. It never removes a resource without this project's owner label or without a valid run id or source label, a resource whose name does not match its labels, a running container, a volume attached to any container that it is not diff --git a/tests/integration-devrun b/tests/integration-devrun index c956500..d3ab057 100755 --- a/tests/integration-devrun +++ b/tests/integration-devrun @@ -162,6 +162,65 @@ commit_b="$(git -C "$bin" rev-parse HEAD)" got="$("$devrun" "$project" "$bin" --commit "$commit_b" "${small[@]}" -- cargo run -q --locked 2>/dev/null)" [[ "$got" == built-from-B ]] || fail "commit B ran the stale build of commit A: $got" +echo "== two --commit runs of one worktree at once: the second never reuses the first one's build (issue #12, 1)" +# A refreshes its copy, then waits until B has refreshed its own copy too, then builds. B waits for A's +# build, then builds. Without the pinned-target lock B's copy predates A's build, and cargo reuses A's +# binary. With the lock B cannot start until A has finished, so A's wait for B ends at its bound. +pin="$work/pin"; mkdir -p "$pin/src" +printf '[package]\nname = "pin"\nversion = "0.1.0"\nedition = "2021"\n' > "$pin/Cargo.toml" +printf 'version = 4\n\n[[package]]\nname = "pin"\nversion = "0.1.0"\n' > "$pin/Cargo.lock" +printf 'fn main() { println!("built-from-P1"); }\n' > "$pin/src/main.rs" +git -C "$pin" init -q -b main && git -C "$pin" add -A && git -C "$pin" -c user.name=t -c user.email=t@invalid commit -qm P1 +commit_p1="$(git -C "$pin" rev-parse HEAD)" +printf 'fn main() { println!("built-from-P2"); }\n' > "$pin/src/main.rs" +git -C "$pin" -c user.name=t -c user.email=t@invalid commit -qam P2 +commit_p2="$(git -C "$pin" rev-parse HEAD)" +# await FILE TENTHS: wait inside the container until FILE exists in the shared target, at most TENTHS/10 s. +# shellcheck disable=SC2016 # expands inside the container. +await='await() { i=0; while [ ! -e "$1" ] && [ "$i" -lt "$2" ]; do sleep 0.1; i=$((i + 1)); done; }' +# shellcheck disable=SC2016 +pin_a="$await"'; echo "start $(date +%s.%N)"; await /target/sync-b 150; echo "build $(date +%s.%N)"; cargo run -q --locked; touch /target/sync-a; echo "end $(date +%s.%N)"' +# shellcheck disable=SC2016 +pin_b="$await"'; echo "start $(date +%s.%N)"; touch /target/sync-b; await /target/sync-a 600; cargo run -q --locked' +spawn "$project" "$pin" --commit "$commit_p1" "${small[@]}" -- bash -c "$pin_a" >"$work/pa.out" 2>"$work/pa.err" & p1=$! +pids=("$p1") +wait_for 120 "commit run A's command to start" has_line '^start ' "$work/pa.out" +spawn "$project" "$pin" --commit "$commit_p2" "${small[@]}" -- bash -c "$pin_b" >"$work/pb.out" 2>"$work/pb.err" & p2=$! +pids+=("$p2") +wait "$p1" || fail "commit run A failed: $(cat "$work/pa.err")" +wait "$p2" || fail "commit run B failed: $(cat "$work/pb.err")" +pids=() +grep -qx built-from-P1 "$work/pa.out" || fail "commit run A did not run its own build: $(cat "$work/pa.out")" +grep -qx built-from-P2 "$work/pb.out" || fail "commit run B ran another commit's build: $(grep built-from "$work/pb.out")" +grep -q 'waiting for the pinned build target of this worktree' "$work/pb.err" || fail "commit run B did not wait for run A" +before "$(field end "$work/pa.out")" "$(field start "$work/pb.out")" || fail "two pinned runs of one worktree overlapped" + +echo "== a --worktree run during a --commit run of the same worktree: the commit run builds its own source" +# C refreshes its copy, waits for W's build, then builds; W builds once C has refreshed. In one shared +# target C would reuse W's build. Pinned runs have their own target, so neither sees the other's marker +# files and both waits end at their bound. +printf 'fn main() { println!("built-from-W"); }\n' > "$pin/src/main.rs" +# shellcheck disable=SC2016 +pin_c="$await"'; echo "start $(date +%s.%N)"; touch /target/sync-c; await /target/sync-w 150; cargo run -q --locked' +# shellcheck disable=SC2016 +pin_w="$await"'; await /target/sync-c 150; cargo run -q --locked; touch /target/sync-w' +spawn "$project" "$pin" --commit "$commit_p2" "${small[@]}" -- bash -c "$pin_c" >"$work/pc.out" 2>"$work/pc.err" & p1=$! +spawn "$project" "$pin" --worktree "${small[@]}" -- bash -c "$pin_w" >"$work/pw.out" 2>"$work/pw.err" & p2=$! +pids=("$p1" "$p2") +wait "$p1" || fail "commit run C failed: $(cat "$work/pc.err")" +wait "$p2" || fail "worktree run W failed: $(cat "$work/pw.err")" +pids=() +grep -qx built-from-P2 "$work/pc.out" || fail "commit run C ran the worktree's build: $(grep built-from "$work/pc.out")" +grep -qx built-from-W "$work/pw.out" || fail "worktree run W did not run its own build: $(cat "$work/pw.out")" + +echo "== a --worktree run after a --commit run sees an edit made before the commit run (issue #9, case 1)" +printf 'fn main() { println!("built-from-E"); }\n' > "$pin/src/main.rs" +[[ "$(prun "$project" "$pin" --commit "$commit_p2" "${small[@]}" -- cargo run -q --locked 2>/dev/null)" == built-from-P2 ]] || + fail "the commit run did not build its commit" +got="$(prun "$project" "$pin" --worktree "${small[@]}" -- cargo run -q --locked 2>/dev/null)" +[[ "$got" == built-from-E ]] || fail "the worktree run reused the commit run's build: $got" +no_run_leftovers + echo "== two runs of the same project at once, from two worktrees of one repository" git -C "$repo" add -A && git -C "$repo" -c user.name=t -c user.email=t@invalid commit -qm "worktree base" git -C "$repo" worktree add -q "$work/wt2" @@ -245,6 +304,70 @@ env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc2.out" || fa [[ -z "$(leftovers_of "$rid")" ]] || fail "gc left resources of dead run $rid: $(leftovers_of "$rid")" [[ ! -e "$private/runs/$rid.lock" ]] || fail "gc did not forget the dead run" +# epoch NAME FIELD: a container's State.StartedAt or State.FinishedAt, in seconds, by the daemon's clock. +epoch() { + python3 -c 'import datetime, re, sys +m = re.fullmatch(r"(\S+?)(?:\.(\d+))?(Z|[+-]\d\d:\d\d)", sys.argv[1]) +zone = "+00:00" if m.group(3) == "Z" else m.group(3) +print("%.3f" % (datetime.datetime.fromisoformat(m.group(1) + zone).timestamp() + float("0." + (m.group(2) or "0"))))' \ + "$(docker inspect -f "{{.State.$2}}" "$1")" +} +running() { [[ "$(docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null)" == true ]]; } +not_running() { ! running "$1"; } +# finished NAME: when the container stopped. Docker reports year 1 for a running container, so wait. +finished() { + wait_for 300 "container $1 to stop" not_running "$1" + local t; t="$(epoch "$1" FinishedAt)" + before 1000000000 "$t" || fail "container $1 has no finish time: $t" + echo "$t" +} +waits_or_starts() { has_line "$1" "$2" || has_line '^start ' "$3"; } + +echo "== SIGKILL: the surviving container keeps its slot, and an exclusive run waits until it ends (issue #12, 2)" +spawn "$project" "$repo" --worktree "${small[@]}" --max-runs 1 -- bash -c "$timed" timed 20 >"$work/s.out" 2>"$work/s.err" & ps_=$! +pids=("$ps_") +wait_for 120 "the run's command to start" has_line '^start ' "$work/s.out" +srid="$(run_id_of "$work/s.err")"; survivor="$project-dev-run-$srid" +kill -KILL "$ps_" +wait "$ps_" 2>/dev/null || true +running "$survivor" || fail "the container did not survive its runner" +spawn "$project" "$work/wt2" --worktree "${small[@]}" --max-runs 1 -- bash -c "$timed" timed 1 >"$work/o.out" 2>"$work/o.err" & po=$! +pids=("$po") +wait_for 120 "the ordinary run to wait or start" waits_or_starts "runner of $survivor is dead" "$work/o.err" "$work/o.out" +if has_line '^start ' "$work/o.out" && running "$survivor"; then fail "an ordinary run started beside the survivor under --max-runs 1"; fi +spawn "$project" "$repo" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 1 >"$work/x.out" 2>"$work/x.err" & px=$! +pids+=("$px") +wait_for 120 "the exclusive run to wait or start" waits_or_starts "exclusive access (the runner of $survivor is dead" "$work/x.err" "$work/x.out" +wait "$po" || fail "the ordinary run failed: $(cat "$work/o.err")" +wait "$px" || fail "the exclusive run failed: $(cat "$work/x.err")" +pids=() +survivor_end="$(finished "$survivor")" +before "$survivor_end" "$(field start "$work/o.out")" || fail "the ordinary run started before the survivor ended" +before "$survivor_end" "$(field start "$work/x.out")" || fail "the exclusive run started before the survivor ended" + +echo "== a survivor past its own timeout is stopped by the run that waits for it, and by nothing earlier" +spawn "$project" "$repo" --worktree "${small[@]}" --timeout 12 -- bash -c "$timed" timed 300 >"$work/v.out" 2>"$work/v.err" & pv=$! +pids=("$pv") +wait_for 120 "the run's command to start" has_line '^start ' "$work/v.out" +vrid="$(run_id_of "$work/v.err")"; overdue="$project-dev-run-$vrid" +kill -KILL "$pv" +wait "$pv" 2>/dev/null || true +pids=() +prun "$project" "$work/wt2" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 1 >"$work/y.out" 2>"$work/y.err" || + fail "the exclusive run failed: $(cat "$work/y.err")" +grep -q "stopping $overdue: its runner is dead and its 12 s timeout has passed" "$work/y.err" || + fail "the exclusive run did not stop the overdue survivor: $(cat "$work/y.err")" +running "$overdue" && fail "the overdue survivor is still running" +overdue_end="$(finished "$overdue")" +lived="$(awk -v a="$(epoch "$overdue" StartedAt)" -v b="$overdue_end" 'BEGIN { printf "%.1f", b - a }')" +before 12 "$lived" || fail "the survivor was stopped after $lived s, before its 12 s timeout" +before 60 "$lived" && fail "the survivor lived $lived s: nothing stopped it at its 12 s timeout" +before "$overdue_end" "$(field start "$work/y.out")" || fail "the exclusive run started before the survivor stopped" +env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc4.out" || fail "gc failed: $(cat "$work/gc4.out")" +for r in "$srid" "$vrid"; do + [[ -z "$(leftovers_of "$r")" ]] || fail "gc left resources of dead run $r: $(leftovers_of "$r")" +done + echo "== --gc removes the target of a deleted worktree and never touches resources it cannot prove are leftovers" gone="$work/gone"; mkdir -p "$gone"; git -C "$gone" init -q -b main; printf 'x\n' > "$gone/x" prun "$project" "$gone" --worktree "${small[@]}" -- true 2>/dev/null || fail "run in the soon-deleted worktree" diff --git a/tests/test_devrun.py b/tests/test_devrun.py index 37f9258..b6f1473 100644 --- a/tests/test_devrun.py +++ b/tests/test_devrun.py @@ -251,6 +251,11 @@ def test_source_key_is_a_stable_hash_of_the_path(self): self.assertEqual(devrun.source_key("/src/geordi"), hashlib.sha256(b"/src/geordi").hexdigest()[:16]) self.assertEqual(devrun.source_key("/src/geordi"), devrun.source_key("/src/geordi")) + def test_a_worktree_has_separate_worktree_and_pinned_targets(self): + key = devrun.source_key("/src/geordi") + self.assertEqual(devrun.target_volume_name("geordi", key), f"geordi-target-{key}") + self.assertEqual(devrun.target_volume_name("geordi", key, "target-pinned"), f"geordi-target-pinned-{key}") + def test_distinct_worktrees_get_distinct_targets_with_safe_names(self): paths = ["/src/geordi", "/src/geordi-2", "/src/geordi/", "/src/a b", "/src/été", "/src/x;rm -rf"] names = [devrun.target_volume_name("geordi", devrun.source_key(p)) for p in paths] @@ -290,7 +295,8 @@ def test_container_carries_owner_role_and_run_id_labels(self): create = self.calls[-1] labels = {create[i + 1] for i, a in enumerate(create) if a == "--label"} self.assertEqual(labels, {"org.flyingrobots.owner=geordi", "org.flyingrobots.role=dev-run", - "org.flyingrobots.run-id=0123456789ab"}) + "org.flyingrobots.run-id=0123456789ab", "org.flyingrobots.phase=command", + "org.flyingrobots.timeout=300"}) def test_run_volumes_are_labelled_and_removed_by_cleanup(self): docker = self.docker() @@ -355,21 +361,23 @@ def sleep(self, seconds): class Contender: """One run asking for admission in its own thread. `attempts` counts its failed polls.""" - def __init__(self, directory, exclusive, max_runs=4): + def __init__(self, directory, exclusive, max_runs=4, survivors=None): self.clock = FakeClock() self.attempts = 0 + self.messages = [] self.changed = threading.Condition() self.admitted = threading.Event() self.let_go = threading.Event() self.done = threading.Event() self.error = None self.gate = devrun.Gate(directory, max_runs, clock=self.clock, sleep=self.clock.sleep, - report=self._report, poll=0.5, report_every=0.5) + report=self._report, poll=0.5, report_every=0.5, survivors=survivors) self.thread = threading.Thread(target=self._run, args=(exclusive,), daemon=True) self.thread.start() - def _report(self, _msg): + def _report(self, msg): with self.changed: + self.messages.append(msg) self.attempts += 1 self.changed.notify_all() @@ -650,10 +658,10 @@ def run_volumes(run_id): return [{"name": names[r], "labels": labels(r, run_id)} for r in devrun.PER_RUN_ROLES] -def target(path, key=None, name=None): +def target(path, key=None, name=None, role="target"): key = key or devrun.source_key(path) - return {"name": name or devrun.target_volume_name("geordi", key), - "labels": labels("target", **{devrun.SOURCE_PATH_LABEL: path, devrun.SOURCE_KEY_LABEL: key})} + return {"name": name or devrun.target_volume_name("geordi", key, role), + "labels": labels(role, **{devrun.SOURCE_PATH_LABEL: path, devrun.SOURCE_KEY_LABEL: key})} class GcSelection(unittest.TestCase): @@ -721,6 +729,16 @@ def test_a_target_whose_labels_do_not_match_its_name_is_kept(self): "labels": labels("target", **{devrun.SOURCE_KEY_LABEL: KEY_GONE})}]) self.assertEqual(plan["volumes"], []) + def test_the_pinned_target_of_a_deleted_worktree_is_selected_and_an_existing_one_kept(self): + plan = self.plan([], [target("/src/deleted", role="target-pinned"), target("/src/present", role="target-pinned")]) + self.assertEqual(plan["volumes"], ["geordi-target-pinned-" + KEY_GONE]) + + def test_a_pinned_target_named_like_a_worktree_target_is_kept(self): + # Role and name must agree: a pinned-role volume under the worktree target's name, and the reverse. + plan = self.plan([], [target("/src/deleted", role="target-pinned", name=devrun.target_volume_name("geordi", KEY_GONE)), + target("/src/deleted", name="geordi-target-pinned-" + KEY_GONE)]) + self.assertEqual(plan["volumes"], []) + def test_a_target_in_use_is_kept(self): name = devrun.target_volume_name("geordi", KEY_GONE) user = {"name": "geordi-dev-run-aaaaaaaaaaaa", "running": False, "labels": labels("dev-run", LIVE), "mounts": [name]} @@ -864,5 +882,602 @@ def test_good_arguments(self): self.assertTrue(ns.fetch) +# --------------------------------------------------------------------------- # +# A fake Docker daemon at the process boundary (devrun.run), for error paths that real Docker +# cannot be made to produce on demand: interrupted creates, failing inspects, and orderings +# around a signal. Other commands (git) run for real; git-locks is never called. +# --------------------------------------------------------------------------- # + + +def _done(cmd, code=0, out=b"", err=b""): + return devrun.subprocess.CompletedProcess(cmd, code, out, err) + + +class FakeDaemon: + """Containers and volumes in dictionaries. `hooks` maps a command prefix (a tuple) to a function + called with the command before the daemon acts on it; it may raise, change `self`, or return a + CompletedProcess to use instead of the daemon's own answer.""" + + def __init__(self): + self.volumes = {} # name -> labels + self.containers = {} # name -> {"id", "labels", "running", "mounts", "started"} + self.hooks = {} + self.after = {} # like hooks, called after the daemon acted, with (cmd, result) + self.calls = [] + self.now = "2026-11-14T03:00:00.000000000Z" + self.serial = 0 + self.real_run = devrun.run + + def install(self, case): + devrun.run = self + case.addCleanup(setattr, devrun, "run", self.real_run) + + def add_container(self, name, labels, running=True, mounts=(), started="2026-11-14T02:00:00.5Z"): + self.serial += 1 + self.containers[name] = {"id": f"{self.serial:064x}", "labels": dict(labels), "running": running, + "mounts": list(mounts), "started": started} + return self.containers[name]["id"] + + def by_ref(self, ref): + for name, c in self.containers.items(): + if ref in (name, c["id"]): + return name + return None + + def __call__(self, cmd, **kw): + if cmd and os.path.basename(cmd[0]) == "git-locks": + return _done(cmd, 0) + if not cmd or cmd[0] != "docker": + return self.real_run(cmd, **kw) + self.calls.append(cmd) + for prefix, hook in self.hooks.items(): + if tuple(cmd[:len(prefix)]) == prefix: + result = hook(cmd) + if result is not None: + return result + result = self.act(cmd) + for prefix, hook in self.after.items(): + if tuple(cmd[:len(prefix)]) == prefix: + hook(cmd, result) + return result + + @staticmethod + def filters(cmd): + out = [] + for i, a in enumerate(cmd): + if a == "--filter": + out.append(cmd[i + 1]) + return out + + @staticmethod + def matches(labels, name, filters): + for f in filters: + kind, _, value = f.partition("=") + if kind == "label": + k, eq, v = value.partition("=") + if k not in labels or (eq and labels[k] != v): + return False + elif kind == "name": + if not re.search(value, name): + return False + return True + + def act(self, cmd): + args = cmd[1:] + if args[:2] == ["volume", "inspect"]: + if "--format" in args: + labels = self.volumes.get(args[2]) + return _done(cmd, 1, b"", b"no such volume") if labels is None else _done(cmd, 0, json.dumps(labels).encode()) + names = args[2:] + found = [{"Name": n, "Labels": self.volumes[n]} for n in names if n in self.volumes] + return _done(cmd, 0 if len(found) == len(names) else 1, json.dumps(found).encode() if found else b"") + if args[:2] == ["volume", "create"]: + labels = dict(a.split("=", 1) for a in args[3:-1:2]) + self.volumes.setdefault(args[-1], labels) + return _done(cmd, 0, args[-1].encode()) + if args[:2] == ["volume", "rm"]: + if args[2] not in self.volumes: + return _done(cmd, 1, b"", b"no such volume") + if any(args[2] in c["mounts"] for c in self.containers.values()): + return _done(cmd, 1, b"", b"volume is in use") + del self.volumes[args[2]] + return _done(cmd, 0) + if args[:2] == ["volume", "ls"]: + names = [n for n, l in sorted(self.volumes.items()) if self.matches(l, n, self.filters(cmd))] + return _done(cmd, 0, "".join(n + "\n" for n in names).encode()) + if args[0] == "ps": + every = "-a" in args or "-aq" in args + rows = [(n, c) for n, c in sorted(self.containers.items()) + if (every or c["running"]) and self.matches(c["labels"], n, self.filters(cmd))] + names = "--format" in args + return _done(cmd, 0, "".join((n if names else c["id"]) + "\n" for n, c in rows).encode()) + if args[:2] == ["container", "inspect"] or args[0] == "inspect": + refs = args[2:] if args[0] == "container" else args[1:] + if "--format" in refs: + name = self.by_ref(refs[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + return _done(cmd, 0, b"true\n" if self.containers[name]["running"] else b"false\n") + found = [] + for ref in refs: + name = self.by_ref(ref) + if name is not None: + c = self.containers[name] + found.append({"Id": c["id"], "Name": "/" + name, "Config": {"Labels": c["labels"]}, + "State": {"Running": c["running"], "StartedAt": c["started"]}, + "Mounts": [{"Type": "volume", "Name": m} for m in c["mounts"]]}) + return _done(cmd, 0 if len(found) == len(refs) else 1, json.dumps(found).encode() if found else b"") + if args[0] == "create": + name = args[args.index("--name") + 1] + if name in self.containers: + return _done(cmd, 1, b"", b"Conflict") + labels = dict(args[i + 1].split("=", 1) for i, a in enumerate(args) if a == "--label") + mounts = [re.search(r"src=([^,]+)", args[i + 1]).group(1) for i, a in enumerate(args) if a == "--mount"] + self.add_container(name, labels, running=False, mounts=mounts) + self.containers[name]["cmd"] = list(args) + return _done(cmd, 0) + if args[0] == "start": + name = self.by_ref(args[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + self.containers[name]["running"] = False + return _done(cmd, 0) + if args[0] in ("kill", "stop"): + name = self.by_ref(args[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + self.containers[name]["running"] = False + return _done(cmd, 0) + if args[0] == "wait": + return _done(cmd, 0 if self.by_ref(args[-1]) else 1, b"137\n") + if args[0] == "rm": + name = self.by_ref(args[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + if self.containers[name]["running"] and "-f" not in args: + return _done(cmd, 1, b"", b"container is running") + del self.containers[name] + return _done(cmd, 0) + if args[0] == "info": + return _done(cmd, 0, json.dumps(self.now).encode()) + if args[0] == "cp": + return _done(cmd, 0) + raise AssertionError(f"the fake daemon does not know {cmd}") + + +def interrupt(*_): + raise devrun.DevrunError("interrupted by signal 15; removing this run's container and volumes") + + +class InterruptedCreates(unittest.TestCase): + """Finding 3: a signal between Docker's create and devrun's bookkeeping.""" + + def setUp(self): + self.daemon = FakeDaemon() + self.daemon.install(self) + self.docker = devrun.Docker("geordi", RUN_ID, "img", "1", "1g") + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.dir = pathlib.Path(self.tmp.name) + (self.dir / "runs").mkdir() + + def test_a_volume_created_just_before_a_signal_is_removed_by_cleanup(self): + self.daemon.after[("docker", "volume", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("work") + self.assertIn("geordi-work-0123456789ab", self.daemon.volumes) + self.assertTrue(self.docker.cleanup()) + self.assertEqual(self.daemon.volumes, {}) + + def test_a_signal_before_docker_created_the_volume_leaves_nothing_to_remove(self): + self.daemon.hooks[("docker", "volume", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("work") + self.assertTrue(self.docker.cleanup()) + self.assertNotIn(["docker", "volume", "rm", "geordi-work-0123456789ab"], self.daemon.calls) + + def test_the_record_is_kept_when_cleanup_cannot_tell_whether_a_volume_exists(self): + record = devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}) + self.daemon.after[("docker", "volume", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("scratch") + self.daemon.hooks[("docker", "volume", "ls")] = lambda cmd: _done(cmd, 1, b"", b"daemon unavailable") + self.daemon.hooks[("docker", "volume", "rm")] = lambda cmd: _done(cmd, 1, b"", b"daemon unavailable") + with contextlib.redirect_stderr(io.StringIO()): + clean = self.docker.cleanup() + record.finish(clean) + self.assertFalse(clean) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") # --gc still finds the run + + def test_a_volume_of_another_owner_under_the_runs_name_is_never_removed(self): + # The create is interrupted and the name turns out to belong to a volume without this run's labels. + def foreign(cmd): + self.daemon.volumes[cmd[-1]] = {devrun.OWNER_LABEL: "someone-else"} + interrupt() + self.daemon.hooks[("docker", "volume", "create")] = foreign + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("work") + self.assertTrue(self.docker.cleanup()) + self.assertIn("geordi-work-0123456789ab", self.daemon.volumes) + + def test_a_container_created_just_before_a_signal_is_removed_by_cleanup(self): + self.daemon.after[("docker", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create(["true"], False, "geordi-target-x") + self.assertIn(self.docker.name, self.daemon.containers) + self.assertTrue(self.docker.cleanup()) + self.assertEqual(self.daemon.containers, {}) + + def test_a_failed_create_under_a_foreign_containers_name_does_not_remove_it(self): + self.daemon.add_container("geordi-dev-run-0123456789ab", {devrun.OWNER_LABEL: "other"}, running=True) + with self.assertRaises(devrun.DevrunError): + self.docker.create(["true"], False, "geordi-target-x") + self.assertTrue(self.docker.cleanup()) + self.assertIn("geordi-dev-run-0123456789ab", self.daemon.containers) + self.assertFalse(any(c[:2] == ["docker", "rm"] for c in self.daemon.calls)) + + +class InspectFailures(unittest.TestCase): + """Finding 4: a failing docker inspect must not make existing resources look absent.""" + + def setUp(self): + self.daemon = FakeDaemon() + self.daemon.install(self) + + def test_an_inspect_that_fails_while_the_resources_still_exist_raises(self): + cid = self.daemon.add_container("geordi-dev-run-dddddddddddd", labels("dev-run", "dddddddddddd"), running=False) + self.daemon.hooks[("docker", "container", "inspect")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + with self.assertRaises(devrun.DevrunError) as ctx: + devrun._inspect(["container"], [cid], devrun.all_container_ids) + self.assertIn("still exist", str(ctx.exception)) + + def test_a_resource_removed_meanwhile_is_accepted_and_the_others_returned(self): + kept = self.daemon.add_container("geordi-dev-run-aaaaaaaaaaaa", labels("dev-run", LIVE), running=False) + gone = "f" * 64 # listed earlier, removed before the inspect: not in a fresh listing either + found = devrun._inspect(["container"], [kept, gone], devrun.all_container_ids) + self.assertEqual([d["Id"] for d in found], [kept]) + + def test_a_failing_listing_raises(self): + self.daemon.hooks[("docker", "volume", "inspect")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + self.daemon.hooks[("docker", "volume", "ls")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + with self.assertRaises(devrun.DevrunError): + devrun._inspect(["volume"], ["geordi-work-dddddddddddd"], lambda: devrun.project_volume_names("geordi")) + + def test_output_that_is_not_json_raises(self): + self.daemon.hooks[("docker", "volume", "inspect")] = lambda cmd: _done(cmd, 1, b"garbage", b"") + with self.assertRaises(devrun.DevrunError): + devrun._inspect(["volume"], ["x"], lambda: set()) + + +class StateDirEnv(unittest.TestCase): + """A temporary devrun state directory in DEVRUN_STATE_DIR, and CI unset, for the test's length.""" + + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.dir = pathlib.Path(self.tmp.name) / "state" + for k, v in ((devrun.STATE_ENV, str(self.dir)), ("CI", None), (devrun.MAX_RUNS_ENV, None)): + saved = os.environ.get(k) + self.addCleanup(lambda k=k, saved=saved: os.environ.pop(k, None) if saved is None else os.environ.__setitem__(k, saved)) + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + devrun.state_dir() + self.daemon = FakeDaemon() + self.daemon.install(self) + + +class GcKeepsRecords(StateDirEnv): + def test_a_dead_runs_record_is_kept_when_the_final_inspect_fails(self): + devrun.RunRecord(self.dir, DEAD, {"project": "geordi"}).finish(clean=False) + self.daemon.add_container("geordi-dev-run-" + DEAD, labels("dev-run", DEAD), running=False) + # The removal fails, and from then on Docker cannot inspect: the container still exists. + def broken(cmd): + self.daemon.hooks[("docker", "container", "inspect")] = lambda c: _done(c, 1, b"", b"daemon error") + return _done(cmd, 1, b"", b"daemon error") + self.daemon.hooks[("docker", "rm")] = broken + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()): + code = devrun.main(["--gc", "geordi"]) + self.assertNotEqual(code, 0) + self.assertTrue((self.dir / "runs" / f"{DEAD}.lock").exists(), "gc forgot a dead run that still has a container") + self.assertEqual(devrun.run_liveness(self.dir, DEAD), "dead") + + def test_a_dead_runs_record_is_forgotten_once_nothing_is_left(self): + devrun.RunRecord(self.dir, DEAD, {"project": "geordi"}).finish(clean=False) + self.daemon.add_container("geordi-dev-run-" + DEAD, labels("dev-run", DEAD), running=False) + with contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(devrun.main(["--gc", "geordi"]), 0) + self.assertFalse((self.dir / "runs" / f"{DEAD}.lock").exists()) + + +# --------------------------------------------------------------------------- # +# Finding 2: containers that outlive their runner, and admission kept until the container is gone. +# --------------------------------------------------------------------------- # + + +def survivor(name="geordi-dev-run-dddddddddddd", phase="command", mounts=(), project="geordi"): + return {"id": "x", "name": name, "project": project, "run_id": "dddddddddddd", "phase": phase, + "timeout": "300", "started": "", "mounts": list(mounts)} + + +class SurvivorAdmission(GateCase): + def contender(self, exclusive=False, max_runs=4, survivors=None): + c = Contender(self.dir, exclusive, max_runs, survivors) + self.contenders.append(c) + return c + + def test_a_survivor_holds_a_slot(self): + left = [survivor()] + a = self.contender(max_runs=2, survivors=lambda: list(left)) + self.admitted(a) # one slot for the survivor, one for a + b = self.contender(max_runs=2, survivors=lambda: list(left)) + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + self.assertTrue(any("geordi-dev-run-dddddddddddd" in m and "dead" in m for m in b.messages), b.messages) + left.clear() + self.admitted(b) + + def test_survivors_of_other_phases_hold_no_slot(self): + left = [survivor(phase="fetch"), survivor(phase="prep")] + self.admitted(self.contender(max_runs=1, survivors=lambda: list(left))) + + def test_a_survivor_of_an_older_runner_without_a_phase_label_holds_a_slot(self): + left = [survivor(phase=None)] + b = self.contender(max_runs=1, survivors=lambda: list(left)) + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + left.clear() + self.admitted(b) + + def test_an_exclusive_run_waits_for_a_survivor_and_says_why(self): + left = [survivor()] + e = self.contender(exclusive=True, survivors=lambda: list(left)) + e.wait_attempts(3) + self.assertFalse(e.admitted.is_set()) + self.assertTrue(any(m.startswith("devrun: waiting for exclusive access (the runner of geordi-dev-run-dddddddddddd is dead") + for m in e.messages), e.messages) + # While it waits, new ordinary runs queue behind it. + b = self.contender() + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + left.clear() + self.admitted(e) + e.release() + self.admitted(b) + + def test_an_exclusive_run_waiting_for_a_survivor_gives_up_at_its_wait_limit(self): + gate = devrun.Gate(self.dir, 1, clock=FakeClock(), sleep=lambda s: None, report=lambda m: None, + survivors=lambda: [survivor()]) + with self.assertRaises(devrun.DevrunError) as ctx: + gate.admit(True, 0) + self.assertIn("gave up", str(ctx.exception)) + self.try_once(exclusive=True).release() # it released the turnstile and the gate + + def test_a_named_lock_waits_for_a_blocking_survivor(self): + left = [survivor(mounts=["geordi-target-pinned-k"])] + clock = FakeClock() + polls = [] + gate = devrun.Gate(self.dir, 1, clock=clock, sleep=lambda s: (polls.append(s), left.clear() if len(polls) == 3 else None), + report=lambda m: None, survivors=lambda: list(left)) + with gate.hold("pinned-geordi-k.lock", "the pinned build target", None, + blocking=lambda s: "geordi-target-pinned-k" in s["mounts"]): + self.assertEqual(len(polls), 3) + # A survivor that does not mount it does not block. + left.append(survivor(mounts=["other"])) + gate.hold("pinned-geordi-k.lock", "the pinned build target", 0, + blocking=lambda s: "geordi-target-pinned-k" in s["mounts"]).release() + + +class SurvivorScan(StateDirEnv): + def setUp(self): + super().setUp() + self.ids = {} + + def run_container(self, rid, state, running=True, owner="geordi", name=None, timeout="300", + started="2026-11-14T02:59:00.5Z", phase="command"): + """A dev-run container of a run whose record is 'alive' (held by a child), 'dead', or 'unknown'.""" + if state == "dead": + devrun.RunRecord(self.dir, rid, {"project": owner}).finish(clean=False) + elif state == "alive": + child = subprocess.Popen([sys.executable, "-c", HOLD_RECORD, str(ROOT / "bin" / "devrun"), str(self.dir), rid], + stdout=subprocess.PIPE, stdin=subprocess.PIPE) + self.assertEqual(child.stdout.readline(), b"recorded\n") + def stop(child=child): + child.kill() + child.wait() + child.stdin.close() + child.stdout.close() + self.addCleanup(stop) + lab = labels("dev-run", rid, owner) + if phase is not None: + lab[devrun.PHASE_LABEL] = phase + if timeout is not None: + lab[devrun.TIMEOUT_LABEL] = timeout + name = name or devrun.run_resource_names(owner, rid)["container"] + self.ids[rid] = self.daemon.add_container(name, lab, running=running, started=started) + return name + + def test_only_running_containers_of_dead_runs_with_valid_labels_are_survivors(self): + dead = self.run_container("d00000000001", "dead") + self.run_container("a00000000001", "alive") + self.run_container("e00000000001", "unknown") + self.run_container("d00000000002", "dead", running=False) + self.run_container("d00000000003", "dead", name="geordi-dev-run-d00000000009") # name and label disagree + self.daemon.add_container("geordi-dev-run", labels("dev-run")) # no run id + self.daemon.add_container("unrelated", {}) + found = devrun.Survivors(self.dir, report=lambda m: None)() + self.assertEqual([s["name"] for s in found], [dead]) + self.assertEqual(found[0]["phase"], "command") + + def test_a_survivor_past_its_own_timeout_is_stopped_and_no_other_container(self): + overdue = self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T02:58:59.9Z") + within = self.run_container("d00000000002", "dead", timeout="60", started="2026-11-14T02:59:30Z") + legacy = self.run_container("d00000000003", "dead", timeout=None, phase=None, started="2026-11-14T01:00:00Z") + self.run_container("a00000000001", "alive", timeout="60", started="2026-11-14T01:00:00Z") + self.run_container("e00000000001", "unknown", timeout="60", started="2026-11-14T01:00:00Z") + messages = [] + found = devrun.Survivors(self.dir, report=messages.append)() + self.assertEqual(sorted(s["name"] for s in found), sorted([within, legacy])) + killed = [c[2] for c in self.daemon.calls if c[:2] == ["docker", "kill"]] + self.assertEqual(killed, [self.ids["d00000000001"]]) + self.assertFalse(self.daemon.containers[overdue]["running"]) + self.assertEqual(messages, [f"devrun: stopping {overdue}: its runner is dead and its 60 s timeout has passed"]) + + def test_gc_dry_run_stops_nothing(self): + self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T01:00:00Z") + with contextlib.redirect_stdout(io.StringIO()) as out: + self.assertEqual(devrun.main(["--gc", "geordi", "--dry-run"]), 0) + self.assertFalse(any(c[:2] in (["docker", "kill"], ["docker", "rm"]) for c in self.daemon.calls)) + self.assertIn("would stop container geordi-dev-run-d00000000001", out.getvalue()) + + def test_gc_stops_an_overdue_survivor_and_then_removes_it(self): + name = self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T01:00:00Z") + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()): + self.assertEqual(devrun.main(["--gc", "geordi"]), 0) + self.assertNotIn(name, self.daemon.containers) + + def test_gc_keeps_a_survivor_within_its_timeout(self): + name = self.run_container("d00000000001", "dead", timeout="7200", started="2026-11-14T02:59:00Z") + with contextlib.redirect_stdout(io.StringIO()) as out: + self.assertEqual(devrun.main(["--gc", "geordi"]), 0) + self.assertTrue(self.daemon.containers[name]["running"]) + self.assertIn("still running, within its timeout", out.getvalue()) + + def test_docker_times_are_read_with_their_zone(self): + self.assertEqual(devrun.parse_docker_time("2026-11-14T03:19:46Z"), 1794626386.0) + self.assertAlmostEqual(devrun.parse_docker_time("2026-11-13T19:21:36.244827538-08:00"), 1794626496.244827538, places=5) + with self.assertRaises(devrun.DevrunError): + devrun.parse_docker_time("yesterday") + + +class HoldsUntilStopped(StateDirEnv): + """The SIGTERM half of finding 2: locks handed to start() outlive a signal until the container is gone.""" + + def setUp(self): + super().setUp() + self.gate = devrun.Gate(self.dir, 1) + self.docker = devrun.Docker("geordi", RUN_ID, "img", "1", "1g") + self.docker.create(["sleep", "100"], False, "geordi-target-x") + + def slot_free(self): + try: + self.gate.admit(False, 0).release() + return True + except devrun.DevrunError: + return False + + def test_a_signal_during_the_command_keeps_the_slot_until_cleanup_removed_the_container(self): + self.daemon.hooks[("docker", "start")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertFalse(self.slot_free()) + self.assertTrue(self.docker.cleanup()) + self.assertTrue(self.slot_free()) + + def test_a_container_still_running_after_start_returned_keeps_the_slot(self): + def detached(cmd): + return _done(cmd, 0) # the client returned, the container did not stop + self.daemon.containers[self.docker.name]["running"] = True + self.daemon.hooks[("docker", "start")] = detached + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertFalse(self.slot_free()) + self.docker.remove() + self.assertTrue(self.slot_free()) + + def test_a_container_that_cannot_be_removed_keeps_the_slot(self): + self.daemon.hooks[("docker", "start")] = interrupt + self.daemon.hooks[("docker", "rm")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + with self.assertRaises(devrun.DevrunError): + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertFalse(self.docker.cleanup()) + self.assertFalse(self.slot_free()) + self.docker.release_holds() # what main_inner does after the record shows the run dead + self.assertTrue(self.slot_free()) + + def test_a_normal_end_releases_the_slot_once_docker_confirms_the_stop(self): + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertTrue(self.slot_free()) + + +class RunUnderSignal(StateDirEnv): + """main() end to end against the fake daemon: what is held when cleanup removes the container.""" + + def setUp(self): + super().setUp() + for sig in (devrun.signal.SIGTERM, devrun.signal.SIGINT): + self.addCleanup(devrun.signal.signal, sig, devrun.signal.getsignal(sig)) + self.repo = pathlib.Path(self.tmp.name) / "repo" + self.repo.mkdir() + git = ["git", "-C", str(self.repo), "-c", "user.name=t", "-c", "user.email=t@invalid"] + subprocess.run(git + ["init", "-q"], check=True) + (self.repo / "x").write_text("x\n") + subprocess.run(git + ["add", "x"], check=True) + subprocess.run(git + ["commit", "-qm", "x"], check=True) + self.key = devrun.source_key(os.path.realpath(str(self.repo))) + self.out = pathlib.Path(self.tmp.name) / "out" + + def main(self, *mode): + with contextlib.redirect_stderr(io.StringIO()) as err: + code = devrun.main(["geordi", str(self.repo), *mode, "--max-runs", "1", "--wait", "0", + "--out", str(self.out), "--", "true"]) + return code, err.getvalue() + + def is_command(self, cmd): + name = self.daemon.by_ref(cmd[-1]) + return name is not None and devrun.INNER_SCRIPT in self.daemon.containers[name].get("cmd", []) + + def lock_free(self, name): + try: + devrun.Gate(self.dir, 1).hold(name, name, 0).release() + return True + except devrun.DevrunError: + return False + + def test_sigterm_during_the_command_keeps_admission_until_the_container_is_removed(self): + seen = {} + + def start(cmd): + if self.is_command(cmd): + self.daemon.containers[self.daemon.by_ref(cmd[-1])]["running"] = True + interrupt() + + def rm(cmd): + if self.is_command(cmd): + try: + devrun.Gate(self.dir, 1).admit(True, 0).release() + seen["exclusive admitted while the container existed"] = True + except devrun.DevrunError: + seen["exclusive admitted while the container existed"] = False + self.daemon.hooks[("docker", "start")] = start + self.daemon.hooks[("docker", "rm")] = rm + code, err = self.main("--worktree") + self.assertEqual(code, 2, err) + self.assertIn("interrupted by signal 15", err) + self.assertEqual(seen, {"exclusive admitted while the container existed": False}) + self.assertEqual(self.daemon.containers, {}) + devrun.Gate(self.dir, 1).admit(True, 0).release() # and released after it + + def test_a_pinned_run_holds_its_targets_lock_while_the_command_runs_and_a_worktree_run_does_not(self): + lock = f"pinned-geordi-{self.key}.lock" + held = [] + mounts = [] + + def start(cmd): + if self.is_command(cmd): + held.append(not self.lock_free(lock)) + mounts.append(self.daemon.containers[self.daemon.by_ref(cmd[-1])]["mounts"]) + self.daemon.hooks[("docker", "start")] = start + for mode in (["--commit", "HEAD"], ["--worktree"]): + code, err = self.main(*mode) + self.assertEqual(code, 0, err) + self.out.rmdir() + self.assertEqual(held, [True, False]) + self.assertIn(f"geordi-target-pinned-{self.key}", mounts[0]) + self.assertIn(f"geordi-target-{self.key}", mounts[1]) + self.assertTrue(self.lock_free(lock)) + self.assertEqual(self.daemon.volumes[f"geordi-target-pinned-{self.key}"][devrun.ROLE_LABEL], "target-pinned") + + if __name__ == "__main__": unittest.main() From 27871e85e5a2484ca3e018312385c7f1092ad572 Mon Sep 17 00:00:00 2001 From: James Ross Date: Fri, 13 Nov 2026 20:29:10 -0800 Subject: [PATCH 2/2] Keep a dead exclusive run's command alone, and probe liveness shared Review of #13. - An --exclusive run's surviving command counted as one slot, so with the default cap an ordinary run started beside it. The command container now carries org.flyingrobots.admission and the run record the mode; an exclusive survivor keeps every ordinary run out until it stops. - run_liveness probed with a non-blocking exclusive lock, so two probes of a dead record at once could make one read "alive" and drop the survivor. It now probes with a shared lock; the run itself holds the record exclusively, so a live run still reads alive. Verified: new integration section fails on the previous head ("an ordinary run started beside a dead exclusive run's command") and passes now; the two liveness tests (a held shared probe, 8 threads probing a dead record) and the exclusive-survivor unit test fail before and pass after; mutations of each fix turn them red. Agent-Tool: Claude Code --- CHANGELOG.md | 5 +++ README.md | 3 +- bin/devrun | 35 +++++++++++++++---- docs/hygiene.md | 18 ++++++---- tests/integration-devrun | 19 ++++++++++- tests/test_devrun.py | 72 +++++++++++++++++++++++++++++++++++++--- 6 files changed, 134 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6df41b2..2adcbd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,11 @@ version numbers yet, so changes are listed under Unreleased. Once its own timeout has passed, a waiting run or `--gc` stops it. Run containers carry two new labels for this, `org.flyingrobots.phase` and `org.flyingrobots.timeout`. +- A surviving command of a killed `--exclusive` run keeps every ordinary run out + until it stops; it no longer counts as one slot. The command container is + labelled `org.flyingrobots.admission` and the run record keeps the mode. +- A liveness probe takes a shared lock, so two probes of a dead run's record at + the same moment no longer make one of them read the run as alive. - A run keeps its slot, the exclusive gate, the fetch lock, and the pinned lock until Docker confirms that its container has stopped or it has been removed; on SIGTERM they were released before cleanup had removed the container. diff --git a/README.md b/README.md index 8faaa17..5ad02ad 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,8 @@ Runs do not wait for each other unless the machine is full. The rules are in and gives up after `--wait SECONDS` (default 3600). A run keeps its slot until its container has stopped. - A command container whose runner was killed (SIGKILL) keeps counting against - the cap, and `--exclusive` runs wait for it, until it ends. Once its own + the cap, and `--exclusive` runs wait for it, until it ends; one from an + `--exclusive` run keeps every other run out. Once its own `--timeout` has passed, the next run that waits for it (or `--gc`) stops it; nothing stops it earlier. - `--exclusive` waits until no other command phase runs and keeps them all out diff --git a/bin/devrun b/bin/devrun index cb2a76e..d87f4c4 100755 --- a/bin/devrun +++ b/bin/devrun @@ -51,6 +51,7 @@ SOURCE_PATH_LABEL = "org.flyingrobots.source-path" SOURCE_KEY_LABEL = "org.flyingrobots.source-key" PHASE_LABEL = "org.flyingrobots.phase" TIMEOUT_LABEL = "org.flyingrobots.timeout" +ADMISSION_LABEL = "org.flyingrobots.admission" RUN_ID_RE = re.compile(r"[0-9a-f]{12}") SOURCE_KEY_RE = re.compile(r"[0-9a-f]{16}") LOCK_STORE = "/Users/Shared/git-locks/workstation.git" @@ -625,9 +626,14 @@ class Gate: # the turnstile one at a time, so no other run takes a slot meanwhile. got = [fd for fd in slots if _try_lock(fd, fcntl.LOCK_EX)] left = [s for s in self.survivors() if counts_as_command(s)] if got else [] - if len(got) <= len(left): + alone = [s for s in left if s.get("exclusive")] + if alone: + w.because(f"a run slot (the runner of {survivor_names(alone)}, an exclusive run, " + f"is dead but its command still runs and keeps the machine to itself)") + elif len(got) <= len(left): w.because(f"a run slot (at most {self.max_runs} at once; the runner of " f"{survivor_names(left)} is dead but its command still runs)" if left else plain) + if alone or len(got) <= len(left): for fd in got: fcntl.flock(fd, fcntl.LOCK_UN) got = [] @@ -710,7 +716,9 @@ def run_liveness(directory: Path, run_id: str) -> str: except FileNotFoundError: return "unknown" try: - if not _try_lock(fd, fcntl.LOCK_EX): + # A shared lock: the run itself holds the record exclusively for as long as it lives, so a + # live run still reads 'alive', and two probes of a dead run's record never see each other. + if not _try_lock(fd, fcntl.LOCK_SH): return "alive" held = os.fstat(fd) try: @@ -791,11 +799,20 @@ class Survivors: if run_liveness(self.directory, rid) != "dead": continue found.append({"id": d["Id"], "name": name, "project": project, "run_id": rid, + "exclusive": labels.get(ADMISSION_LABEL) == "exclusive" or self._recorded_exclusive(rid), "phase": labels.get(PHASE_LABEL), "timeout": labels.get(TIMEOUT_LABEL), "started": d["State"].get("StartedAt") or "", "mounts": [m.get("Name") for m in d.get("Mounts") or [] if m.get("Type") == "volume"]}) return found + def _recorded_exclusive(self, run_id: str) -> bool: + """Whether the dead run's record says it asked for exclusive admission.""" + try: + facts = json_loads((self.directory / "runs" / f"{run_id}.lock").read_text()) + except (OSError, ValueError): + return False + return isinstance(facts, dict) and facts.get("admission") == "exclusive" + def deadline(self, survivor: dict) -> Optional[float]: """When the survivor's own timeout ends, by the daemon's clock; None without a timeout label.""" timeout = survivor.get("timeout") or "" @@ -958,13 +975,15 @@ class Docker: return name def create(self, cmd: List[str], network: bool, target_volume: str, env: Optional[dict] = None, - caches_writable: bool = False, phase: str = "command", timeout: int = 300) -> None: + caches_writable: bool = False, phase: str = "command", timeout: int = 300, + admission: Optional[str] = None) -> None: """Create the run container. The cargo caches are writable only in the fetch phase. The phase and the timeout are labels, so that another run can recognize a container whose runner died and enforce its timeout (Survivors).""" ro = "" if caches_writable else ",readonly" args = ["docker", "create", "--name", self.name, - *self.labels("dev-run", {PHASE_LABEL: phase, TIMEOUT_LABEL: str(timeout)}), + *self.labels("dev-run", {PHASE_LABEL: phase, TIMEOUT_LABEL: str(timeout), + **({ADMISSION_LABEL: admission} if admission else {})}), "--user", f"{UID}:{GID}", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--read-only", "--tmpfs", "/tmp:rw,nosuid,nodev,size=1g,mode=1777", "--pids-limit", "4096", "--memory", self.memory, "--memory-swap", self.memory, "--cpus", self.cpus, @@ -1404,7 +1423,9 @@ def main_inner(ns: argparse.Namespace) -> int: coordinated = os.environ.get("CI") != "true" directory = state_dir() gate = Gate(directory, ns.max_runs, survivors=Survivors(directory)) - record = RunRecord(directory, run_id, {"project": ns.project, "source_key": key, "pid": str(os.getpid())}) + admission_mode = "exclusive" if ns.exclusive else "ordinary" + record = RunRecord(directory, run_id, {"project": ns.project, "source_key": key, "pid": str(os.getpid()), + "admission": admission_mode}) lease = Lease(f"devrun-{ns.project}-{run_id}", f"devrun-{run_id}", f"host/docker/{docker.name}/", 2 * ns.timeout + 2 * ns.wait + 1500, f"devrun {ns.project} run {run_id}") work = Path(tempfile.mkdtemp(prefix="devrun-src-")) @@ -1478,7 +1499,9 @@ def main_inner(ns: argparse.Namespace) -> int: cmd = ["bash", "-c", INNER_SCRIPT, "run", mode, "1" if ns.export_patch else "0", *ns.command] env["CARGO_NET_OFFLINE"] = "false" if ns.network else "true" env["DEVRUN_FRESH_MTIMES"] = "1" if mode == "commit" else "0" - docker.create(cmd, ns.network, target_volume, env, timeout=ns.timeout) + # The admission mode is a label and in the record, so that if this runner dies, other runs + # know that its surviving command had the machine to itself. + docker.create(cmd, ns.network, target_volume, env, timeout=ns.timeout, admission=admission_mode) # The pinned-target lock comes first and admission second: a run never holds a slot or the # exclusive gate while it waits for another run of its worktree. Both stay held until the # container is confirmed stopped or removed (Docker.start, Docker.remove). diff --git a/docs/hygiene.md b/docs/hygiene.md index 7ec9a34..2cbe9e2 100644 --- a/docs/hygiene.md +++ b/docs/hygiene.md @@ -134,7 +134,7 @@ symlinks resolved), so each git worktree has its own key. | --- | --- | --- | --- | | Base image | `flyingrobots/rust-dev:` | `base-image` | Until a new toolchain replaces it | | Upstream base, the build input | `rust@sha256:` from `images/rust-dev/base-digests`, pulled by digest. It carries only upstream labels. Keep it while its digest line exists; after removing the line, remove it with `docker image rm`. | none (upstream image) | As left | -| Run container | `-dev-run-`, also labelled with the run id, `org.flyingrobots.phase=`, and `org.flyingrobots.timeout=` (section 10 uses both when its runner has died) | `dev-run` | One run; removed by the run | +| Run container | `-dev-run-`, also labelled with the run id, `org.flyingrobots.phase=`, `org.flyingrobots.timeout=`, and on the command container `org.flyingrobots.admission=` (section 10 uses them when its runner has died) | `dev-run` | One run; removed by the run | | Cargo registry cache, at `/usr/local/cargo/registry` | `-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` | Kept | | Cargo git cache, at `/usr/local/cargo/git` | `-cargo-git` | `cargo-git` | Kept | | Build output, `--worktree` runs | `-target-`, also labelled `org.flyingrobots.source-path=` and `org.flyingrobots.source-key=` (one per worktree) | `target` | Kept; `devrun --gc` removes it once the worktree path no longer exists | @@ -286,7 +286,9 @@ to themselves. removed it; a timeout, a signal, or an error in between does not release the slot earlier. Preparation, copy-in, the fetch phase, and exports do not take a slot. A survivor (below) in its command phase counts as a held slot: a run is - admitted only when the free slots outnumber the survivors. A run admitted with a + admitted only when the free slots outnumber the survivors. A survivor of an + exclusive run keeps every ordinary run out until it stops, as its runner + would have. A run admitted with a larger N can use slots that a run with a smaller N never looks at, so the cap in force is the one each run was given. A waiting run prints one line when it starts waiting and one every 30 seconds with the time waited so far, and gives @@ -324,7 +326,8 @@ to themselves. says what this guarantees. - **Survivors.** A survivor is a running container that carries the role `dev-run`, a valid owner and run id, and a name that matches them, and whose - run's liveness file in this state directory is free: its runner is dead. + run's liveness file in this state directory is free: its runner is dead. It is + exclusive when its admission label or its run's record says so. Admission, the fetch lock, and the pinned lock find survivors by these labels with `docker ps` and `docker inspect`, but only once the file locks would admit the run. A container of a run in another state directory, or of a live run, is @@ -343,9 +346,12 @@ to themselves. file during the same exit, and a run that passes the turnstile in that instant can see the slot free and the runner still alive, and so start one run over the cap. -- **Run liveness.** Each run locks `runs/.lock` before it creates any - Docker resource and keeps it locked until all of them are gone. The file also - records the project and the source key. +- **Run liveness.** Each run locks `runs/.lock` exclusively before it + creates any Docker resource and keeps it locked until all of them are gone. + The file also records the project, the source key, and the admission mode + (`ordinary` or `exclusive`). A probe takes a shared lock without waiting: it + fails only while the run holds its exclusive lock, so two probes of a dead + run's record at the same moment both read it as dead. - **`devrun --gc `** removes, after printing each item (with `--dry-run`, only prints): - the stopped containers and the per-run volumes of dead runs. A run is dead diff --git a/tests/integration-devrun b/tests/integration-devrun index d3ab057..9447a3f 100755 --- a/tests/integration-devrun +++ b/tests/integration-devrun @@ -345,6 +345,23 @@ survivor_end="$(finished "$survivor")" before "$survivor_end" "$(field start "$work/o.out")" || fail "the ordinary run started before the survivor ended" before "$survivor_end" "$(field start "$work/x.out")" || fail "the exclusive run started before the survivor ended" +echo "== SIGKILL of an exclusive run: its surviving command still keeps ordinary runs out (review of #13)" +# The cap is the default 4, so only the exclusive mode, not the slot count, can keep the ordinary run out. +spawn "$project" "$repo" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 15 >"$work/es.out" 2>"$work/es.err" & pes=$! +pids=("$pes") +wait_for 120 "the exclusive run's command to start" has_line '^start ' "$work/es.out" +erid="$(run_id_of "$work/es.err")"; esurvivor="$project-dev-run-$erid" +kill -KILL "$pes" +wait "$pes" 2>/dev/null || true +running "$esurvivor" || fail "the exclusive run's container did not survive its runner" +spawn "$project" "$work/wt2" --worktree "${small[@]}" -- bash -c "$timed" timed 1 >"$work/eo.out" 2>"$work/eo.err" & peo=$! +pids=("$peo") +wait_for 120 "the ordinary run to wait or start" waits_or_starts "runner of $esurvivor, an exclusive run," "$work/eo.err" "$work/eo.out" +if has_line '^start ' "$work/eo.out" && running "$esurvivor"; then fail "an ordinary run started beside a dead exclusive run's command"; fi +wait "$peo" || fail "the ordinary run failed: $(cat "$work/eo.err")" +pids=() +before "$(finished "$esurvivor")" "$(field start "$work/eo.out")" || fail "the ordinary run started before the exclusive survivor ended" + echo "== a survivor past its own timeout is stopped by the run that waits for it, and by nothing earlier" spawn "$project" "$repo" --worktree "${small[@]}" --timeout 12 -- bash -c "$timed" timed 300 >"$work/v.out" 2>"$work/v.err" & pv=$! pids=("$pv") @@ -364,7 +381,7 @@ before 12 "$lived" || fail "the survivor was stopped after $lived s, before its before 60 "$lived" && fail "the survivor lived $lived s: nothing stopped it at its 12 s timeout" before "$overdue_end" "$(field start "$work/y.out")" || fail "the exclusive run started before the survivor stopped" env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc4.out" || fail "gc failed: $(cat "$work/gc4.out")" -for r in "$srid" "$vrid"; do +for r in "$srid" "$erid" "$vrid"; do [[ -z "$(leftovers_of "$r")" ]] || fail "gc left resources of dead run $r: $(leftovers_of "$r")" done diff --git a/tests/test_devrun.py b/tests/test_devrun.py index b6f1473..77bad3f 100644 --- a/tests/test_devrun.py +++ b/tests/test_devrun.py @@ -605,6 +605,31 @@ def test_a_killed_run_is_dead(self): self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") self.assertEqual(devrun.live_run_facts(self.dir), []) + def test_a_dead_run_stays_dead_while_another_probe_looks_at_it(self): + # Review of #13: two probes at once. The other probe holds its (shared) lock on the record. + devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}).finish(clean=False) + other = os.open(str(self.dir / "runs" / f"{RUN_ID}.lock"), os.O_RDWR) + try: + devrun.fcntl.flock(other, devrun.fcntl.LOCK_SH | devrun.fcntl.LOCK_NB) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") + finally: + os.close(other) + + def test_concurrent_probes_of_a_dead_run_all_see_it_dead(self): + devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}).finish(clean=False) + results, start = [], threading.Barrier(8) + + def probe(): + start.wait(LONG) + for _ in range(200): + results.append(devrun.run_liveness(self.dir, RUN_ID)) + threads = [threading.Thread(target=probe) for _ in range(8)] + for t in threads: + t.start() + for t in threads: + t.join(LONG) + self.assertEqual(set(results), {"dead"}) + def test_an_unknown_run_is_not_dead(self): self.assertEqual(devrun.run_liveness(self.dir, "aaaaaaaaaaaa"), "unknown") @@ -1196,9 +1221,9 @@ def test_a_dead_runs_record_is_forgotten_once_nothing_is_left(self): # --------------------------------------------------------------------------- # -def survivor(name="geordi-dev-run-dddddddddddd", phase="command", mounts=(), project="geordi"): +def survivor(name="geordi-dev-run-dddddddddddd", phase="command", mounts=(), project="geordi", exclusive=False): return {"id": "x", "name": name, "project": project, "run_id": "dddddddddddd", "phase": phase, - "timeout": "300", "started": "", "mounts": list(mounts)} + "timeout": "300", "started": "", "mounts": list(mounts), "exclusive": exclusive} class SurvivorAdmission(GateCase): @@ -1218,6 +1243,16 @@ def test_a_survivor_holds_a_slot(self): left.clear() self.admitted(b) + def test_an_exclusive_survivor_keeps_every_ordinary_run_out(self): + # Review of #13: a dead --exclusive runner's command still has the machine to itself. + left = [survivor(exclusive=True)] + b = self.contender(max_runs=4, survivors=lambda: list(left)) + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + self.assertTrue(any("geordi-dev-run-dddddddddddd" in m and "exclusive" in m for m in b.messages), b.messages) + left.clear() + self.admitted(b) + def test_survivors_of_other_phases_hold_no_slot(self): left = [survivor(phase="fetch"), survivor(phase="prep")] self.admitted(self.contender(max_runs=1, survivors=lambda: list(left))) @@ -1275,10 +1310,13 @@ def setUp(self): self.ids = {} def run_container(self, rid, state, running=True, owner="geordi", name=None, timeout="300", - started="2026-11-14T02:59:00.5Z", phase="command"): + started="2026-11-14T02:59:00.5Z", phase="command", admission=None, record_admission=None): """A dev-run container of a run whose record is 'alive' (held by a child), 'dead', or 'unknown'.""" if state == "dead": - devrun.RunRecord(self.dir, rid, {"project": owner}).finish(clean=False) + facts = {"project": owner} + if record_admission: + facts["admission"] = record_admission + devrun.RunRecord(self.dir, rid, facts).finish(clean=False) elif state == "alive": child = subprocess.Popen([sys.executable, "-c", HOLD_RECORD, str(ROOT / "bin" / "devrun"), str(self.dir), rid], stdout=subprocess.PIPE, stdin=subprocess.PIPE) @@ -1294,6 +1332,8 @@ def stop(child=child): lab[devrun.PHASE_LABEL] = phase if timeout is not None: lab[devrun.TIMEOUT_LABEL] = timeout + if admission is not None: + lab[devrun.ADMISSION_LABEL] = admission name = name or devrun.run_resource_names(owner, rid)["container"] self.ids[rid] = self.daemon.add_container(name, lab, running=running, started=started) return name @@ -1310,6 +1350,14 @@ def test_only_running_containers_of_dead_runs_with_valid_labels_are_survivors(se self.assertEqual([s["name"] for s in found], [dead]) self.assertEqual(found[0]["phase"], "command") + def test_a_survivor_is_exclusive_by_its_label_or_by_its_record(self): + self.run_container("d00000000001", "dead", admission="exclusive") + self.run_container("d00000000002", "dead", admission=None, record_admission="exclusive") + self.run_container("d00000000003", "dead", admission="ordinary") + self.run_container("d00000000004", "dead", admission=None) + found = {s["run_id"]: s["exclusive"] for s in devrun.Survivors(self.dir, report=lambda m: None)()} + self.assertEqual(found, {"d00000000001": True, "d00000000002": True, "d00000000003": False, "d00000000004": False}) + def test_a_survivor_past_its_own_timeout_is_stopped_and_no_other_container(self): overdue = self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T02:58:59.9Z") within = self.run_container("d00000000002", "dead", timeout="60", started="2026-11-14T02:59:30Z") @@ -1458,6 +1506,22 @@ def rm(cmd): self.assertEqual(self.daemon.containers, {}) devrun.Gate(self.dir, 1).admit(True, 0).release() # and released after it + def test_the_admission_mode_is_on_the_command_container_and_in_the_record(self): + seen = [] + + def start(cmd): + if self.is_command(cmd): + c = self.daemon.containers[self.daemon.by_ref(cmd[-1])] + rid = c["labels"][devrun.RUN_ID_LABEL] + record = json.loads((self.dir / "runs" / f"{rid}.lock").read_text()) + seen.append((c["labels"].get(devrun.ADMISSION_LABEL), record.get("admission"))) + self.daemon.hooks[("docker", "start")] = start + for mode in (["--worktree", "--exclusive"], ["--worktree"]): + code, err = self.main(*mode) + self.assertEqual(code, 0, err) + self.out.rmdir() + self.assertEqual(seen, [("exclusive", "exclusive"), ("ordinary", "ordinary")]) + def test_a_pinned_run_holds_its_targets_lock_while_the_command_runs_and_a_worktree_run_does_not(self): lock = f"pinned-geordi-{self.key}.lock" held = []