diff --git a/CHANGELOG.md b/CHANGELOG.md index 32b14f0..2adcbd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,40 @@ version numbers yet, so changes are listed under Unreleased. ## [Unreleased] +### Fixed + +- Two `--commit` runs of different commits of one worktree, started close + together, could share a build: both copies got their time before either build + started, so cargo served the first run's binary to the second (issue #12). A + `--worktree` run between a `--commit` run's copy and its build had the same + effect. Pinned runs (`--commit`, `--history`) now build into their own target + per worktree, `-target-pinned-`, and hold that target's lock from + before their command starts until their container has stopped. Pinned runs of + one worktree now run their commands one at a time. +- A `--worktree` run after a `--commit` run of the same worktree could reuse the + commit's build for an edit older than that build (part of issue #9). The two + modes no longer share a target. +- A container whose runner was killed with SIGKILL no longer escapes the run cap: + it counts as a held slot, and `--exclusive` runs wait for it, until it stops. + Once its own timeout has passed, a waiting run or `--gc` stops it. Run + containers carry two new labels for this, `org.flyingrobots.phase` and + `org.flyingrobots.timeout`. +- A surviving command of a killed `--exclusive` run keeps every ordinary run out + until it stops; it no longer counts as one slot. The command container is + labelled `org.flyingrobots.admission` and the run record keeps the mode. +- A liveness probe takes a shared lock, so two probes of a dead run's record at + the same moment no longer make one of them read the run as alive. +- A run keeps its slot, the exclusive gate, the fetch lock, and the pinned lock + until Docker confirms that its container has stopped or it has been removed; + on SIGTERM they were released before cleanup had removed the container. +- A signal between Docker creating a volume or container and devrun recording it + no longer leaves an untracked resource: devrun records the intent first, and + cleanup lists the run's labelled resources to resolve it, keeping the run's + liveness record when Docker cannot answer. A signal during cleanup is ignored. +- `--gc` no longer forgets a dead run when `docker inspect` fails: a failed + inspect is accepted only when each missing resource is also missing from a + fresh listing. + ### Changed - `bin/devrun` no longer takes the machine-wide `host/heavy-work` lock or a @@ -21,6 +55,10 @@ version numbers yet, so changes are listed under Unreleased. (`-target-evidence-`) and is removed with it. - `--commit` runs give the copied files the current time, so an older commit cannot reuse output built from a newer one (part of issue #9). +- Existing `-target-` volumes stay the targets of `--worktree` + runs. They may still hold output of earlier `--commit` runs, which the first + `--worktree` run after the upgrade can reuse for files older than that output; + remove such a target with `docker volume rm` if that matters. - Each run registers `host/docker/-dev-run-/` in the workstation git-locks store for visibility; a missing or broken git-locks no longer stops a run. diff --git a/README.md b/README.md index 9f03420..5ad02ad 100644 --- a/README.md +++ b/README.md @@ -70,33 +70,45 @@ Runs do not wait for each other unless the machine is full. The rules are in every exit path, including SIGTERM and SIGINT. - At most 4 command phases run at once on the machine. `--max-runs N` or `DEVRUN_MAX_RUNS=N` sets the cap; a waiting run prints how long it has waited, - and gives up after `--wait SECONDS` (default 3600). + and gives up after `--wait SECONDS` (default 3600). A run keeps its slot until + its container has stopped. +- A command container whose runner was killed (SIGKILL) keeps counting against + the cap, and `--exclusive` runs wait for it, until it ends; one from an + `--exclusive` run keeps every other run out. Once its own + `--timeout` has passed, the next run that waits for it (or `--gc`) stops it; + nothing stops it earlier. - `--exclusive` waits until no other command phase runs and keeps them all out until it finishes. Use it for every performance measurement: `bin/devrun echo ~/git/echo --commit HEAD --exclusive -- cargo bench --locked`. - Fetch phases of one project take turns (cargo's own cache lock cannot work on the read-only root). +- `--commit` and `--history` runs of one worktree take turns on that worktree's + pinned build target (below). - With `CI=true` none of this coordination applies; CI machines run one job. - Runs register `host/docker/-dev-run-/` in the workstation git-locks store for visibility; without git-locks they run unregistered. ### Build output and disk -The iteration build output lives in one volume per worktree, -`-target-`, where the key is a hash of the worktree's real path. -Each worktree keeps its own warm cache and never sees another worktree's -artifacts. Two runs from the same worktree share it safely: cargo locks its build -directory. The cost is disk: each target can grow to a full build of the -workspace, so eight worktrees of a large workspace can hold several times the -20 GiB per-project budget. `--evidence` builds into a new, empty volume that -belongs to the one run and is removed with it. - -`--commit` runs give the copied files the current time, so an older commit never -reuses output built from a newer one in the same worktree's target. `--worktree` -runs keep the files' own times, like cargo on the host. A `--worktree` run after a -`--commit` run of the same worktree can still reuse the commit's output for files -that are older than that run (flyingrobots/devcontainers#9); touch the changed -files, or use `--evidence`, when that matters. +Build output lives in kept volumes per worktree, keyed by a hash of the +worktree's real path: `-target-` for `--worktree` runs and +`-target-pinned-` for `--commit` and `--history` runs. Each +worktree keeps its own warm caches and never sees another worktree's artifacts. +Two `--worktree` runs from the same worktree share their target safely: cargo +locks its build directory. The cost is disk: each target can grow to a full +build of the workspace, so eight worktrees of a large workspace can hold several +times the 20 GiB per-project budget. `--evidence` builds into a new, empty volume +that belongs to the one run and is removed with it. + +A pinned run's result does not depend on what else runs: its copy gets the +current time (`--commit`) or is checked out in the container (`--history`), and +no other run can build in the pinned target until its command has ended, so +cargo rebuilds the workspace's crates from this run's source and reuses only +registry and git dependencies. Pinned runs of one worktree therefore run their +commands one at a time. `--worktree` runs keep the files' own times, like cargo +on the host, and never see a pinned run's output; a tool that restores old +modification times can still make them reuse stale output +(flyingrobots/devcontainers#9). ### Leftovers: `bin/devrun --gc` @@ -108,10 +120,13 @@ bin/devrun --gc bunny # remove it `--gc` removes the stopped containers and the volumes of runs that died without cleaning up (killed with SIGKILL, or a crash), and the targets of worktrees that no longer exist. A run counts as dead only when its liveness lock in the devrun -state directory is free, which proves its process exited. `--gc` never removes a -running container, a volume in use, the cargo caches, or anything without a -valid run id or source label, such as the fixed-name volumes of older runner -versions (remove those by hand once nothing uses them). +state directory is free, which proves its process exited. A dead run's container +that is still running is stopped first if its own timeout has passed, and kept +otherwise. `--gc` never removes a running container, a volume in use, the cargo +caches, or anything without a valid run id or source label, such as the +fixed-name volumes of older runner versions (remove those by hand once nothing +uses them). If Docker cannot inspect a resource that still exists, `--gc` stops +with an error and forgets no run. Test it: diff --git a/bin/devrun b/bin/devrun index 81ff613..d87f4c4 100755 --- a/bin/devrun +++ b/bin/devrun @@ -19,6 +19,7 @@ Usage: from __future__ import annotations import argparse +import datetime import fcntl import hashlib import io @@ -48,6 +49,9 @@ SHA_RE = re.compile(r"[0-9a-f]{7,40}") RUN_ID_LABEL = "org.flyingrobots.run-id" SOURCE_PATH_LABEL = "org.flyingrobots.source-path" SOURCE_KEY_LABEL = "org.flyingrobots.source-key" +PHASE_LABEL = "org.flyingrobots.phase" +TIMEOUT_LABEL = "org.flyingrobots.timeout" +ADMISSION_LABEL = "org.flyingrobots.admission" RUN_ID_RE = re.compile(r"[0-9a-f]{12}") SOURCE_KEY_RE = re.compile(r"[0-9a-f]{16}") LOCK_STORE = "/Users/Shared/git-locks/workstation.git" @@ -59,6 +63,8 @@ MAX_RUNS_LIMIT = 64 DEFAULT_WAIT = 3600 # Roles of the volumes that belong to one run and are removed by it (docs/hygiene.md section 5). PER_RUN_ROLES = ("work", "scratch", "target-evidence") +# Roles of the build targets kept per worktree: one for --worktree runs, one for pinned runs. +TARGET_ROLES = ("target", "target-pinned") class DevrunError(Exception): @@ -414,8 +420,11 @@ def source_key(top: str) -> str: return hashlib.sha256(os.fsencode(top)).hexdigest()[:16] -def target_volume_name(project: str, key: str) -> str: - """The iteration target of one source tree (one git worktree).""" +def target_volume_name(project: str, key: str, role: str = "target") -> str: + """A kept build target of one source tree (one git worktree): role "target" for --worktree + runs, "target-pinned" for --commit and --history runs (docs/hygiene.md section 5).""" + if role == "target-pinned": + return f"{project}-target-pinned-{key}" return f"{project}-target-{key}" @@ -489,6 +498,12 @@ class Waiter: self.start = clock() self.next_report = self.start + def because(self, what: str) -> None: + """Change what the wait is for; a new reason is reported at the next poll.""" + if what != self.what: + self.what = what + self.next_report = self.clock() + def wait(self) -> None: now = self.clock() waited = now - self.start @@ -521,6 +536,16 @@ class Admission: self.release() +def survivor_names(survivors: List[dict]) -> str: + return ", ".join(sorted(s["name"] for s in survivors)) + + +def counts_as_command(survivor: dict) -> bool: + """A survivor's container runs a command phase, or is of an older runner that did not label + its phase (then it is treated as one).""" + return survivor.get("phase") in ("command", None) + + class Gate: """At most `max_runs` command phases at once; an exclusive run has the machine to itself. @@ -533,16 +558,23 @@ class Gate: slot- an ordinary run holds one of slot-0 .. slot-. An ordinary run takes the three in one attempt or none, and retries; it never holds the shared lock while it waits for a slot, so it cannot delay an exclusive run that is already waiting. + + A command container can outlive its runner (SIGKILL): its locks are gone but it still runs. + `survivors()` lists such containers (Survivors); each one whose phase is the command + counts as a held slot, and an exclusive run waits until none is left. It is called only once + the locks would admit the run, so a waiting run does not ask Docker on every poll. """ def __init__(self, directory: Path, max_runs: int, clock: Callable[[], float] = time.monotonic, sleep: Callable[[float], None] = time.sleep, report: Callable[[str], None] = _report_stderr, - poll: float = 0.5, report_every: float = 30.0): + poll: float = 0.5, report_every: float = 30.0, + survivors: Optional[Callable[[], List[dict]]] = None): if not 1 <= max_runs <= MAX_RUNS_LIMIT: raise DevrunError(f"the run cap must be between 1 and {MAX_RUNS_LIMIT}") self.directory, self.max_runs = directory, max_runs self.clock, self.sleep, self.report = clock, sleep, report self.poll, self.report_every = poll, report_every + self.survivors = survivors or (lambda: []) def waiter(self, what: str, limit: Optional[float], poll: Optional[float] = None) -> Waiter: return Waiter(what, limit, self.clock, self.sleep, self.report, poll or self.poll, self.report_every) @@ -566,29 +598,54 @@ class Gate: w = self.waiter("exclusive access (another exclusive run is queued or running)", limit) while not _try_lock(turnstile, fcntl.LOCK_EX): w.wait() - w.what = "exclusive access (ordinary runs are finishing; new ones wait behind this run)" + w.because("exclusive access (ordinary runs are finishing; new ones wait behind this run)") while not _try_lock(shared, fcntl.LOCK_EX): w.wait() - return Admission(opened) + # Every admitted run has finished; a container whose runner died may still be running. + while True: + left = [s for s in self.survivors() if counts_as_command(s)] + if not left: + return Admission(opened) + w.because(f"exclusive access (the runner of {survivor_names(left)} is dead but its command " + f"is still running; new runs wait behind this run)") + w.wait() def _admit_ordinary(self, turnstile: int, shared: int, opened: List[int], limit: Optional[float]) -> Admission: slots = [] for i in range(self.max_runs): slots.append(_open_lock(self.directory / f"slot-{i}")) opened.append(slots[-1]) - w = self.waiter(f"a run slot (at most {self.max_runs} at once, or an exclusive run is queued)", limit) + plain = f"a run slot (at most {self.max_runs} at once, or an exclusive run is queued)" + w = self.waiter(plain, limit) while True: - got = None + got: List[int] = [] if _try_lock(turnstile, fcntl.LOCK_EX): try: if _try_lock(shared, fcntl.LOCK_SH): - got = next((fd for fd in slots if _try_lock(fd, fcntl.LOCK_EX)), None) - if got is None: + # Take every free slot for a moment, to count them: ordinary attempts pass + # the turnstile one at a time, so no other run takes a slot meanwhile. + got = [fd for fd in slots if _try_lock(fd, fcntl.LOCK_EX)] + left = [s for s in self.survivors() if counts_as_command(s)] if got else [] + alone = [s for s in left if s.get("exclusive")] + if alone: + w.because(f"a run slot (the runner of {survivor_names(alone)}, an exclusive run, " + f"is dead but its command still runs and keeps the machine to itself)") + elif len(got) <= len(left): + w.because(f"a run slot (at most {self.max_runs} at once; the runner of " + f"{survivor_names(left)} is dead but its command still runs)" if left else plain) + if alone or len(got) <= len(left): + for fd in got: + fcntl.flock(fd, fcntl.LOCK_UN) + got = [] fcntl.flock(shared, fcntl.LOCK_UN) + else: + for fd in got[1:]: + fcntl.flock(fd, fcntl.LOCK_UN) + got = got[:1] finally: fcntl.flock(turnstile, fcntl.LOCK_UN) - if got is not None: - keep = [shared, got] + if got: + keep = [shared, got[0]] for fd in opened: if fd not in keep: os.close(fd) @@ -596,13 +653,22 @@ class Gate: return Admission(list(keep)) w.wait() - def hold(self, name: str, what: str, limit: Optional[float]) -> Admission: - """An exclusive lock on one named file, such as a project's fetch lock.""" + def hold(self, name: str, what: str, limit: Optional[float], + blocking: Optional[Callable[[dict], bool]] = None) -> Admission: + """An exclusive lock on one named file, such as a project's fetch lock. With `blocking`, + the lock is also not taken while a survivor for which blocking(survivor) is true runs.""" fd = _open_lock(self.directory / name) try: w = self.waiter(what, limit) while not _try_lock(fd, fcntl.LOCK_EX): w.wait() + while blocking: + left = [s for s in self.survivors() if blocking(s)] + if not left: + break + w.because(f"{what.split(' (')[0]} (the runner of {survivor_names(left)} is dead " + f"but its container still runs)") + w.wait() except BaseException: os.close(fd) raise @@ -650,7 +716,9 @@ def run_liveness(directory: Path, run_id: str) -> str: except FileNotFoundError: return "unknown" try: - if not _try_lock(fd, fcntl.LOCK_EX): + # A shared lock: the run itself holds the record exclusively for as long as it lives, so a + # live run still reads 'alive', and two probes of a dead run's record never see each other. + if not _try_lock(fd, fcntl.LOCK_SH): return "alive" held = os.fstat(fd) try: @@ -676,6 +744,107 @@ def live_run_facts(directory: Path) -> Optional[List[dict]]: return facts +def parse_docker_time(text: str) -> float: + """Seconds since the epoch for a time as Docker prints it (RFC 3339 with up to nanoseconds, + such as 2026-11-14T03:19:46.51321438Z or 2026-11-13T19:21:36.244827538-08:00).""" + m = re.fullmatch(r"(\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d)(?:\.(\d+))?(Z|[+-]\d\d:\d\d)", text.strip()) + if not m: + raise DevrunError(f"cannot read the Docker time {text!r}") + base, frac, zone = m.groups() + stamp = datetime.datetime.fromisoformat(base + ("+00:00" if zone == "Z" else zone)).timestamp() + return stamp + (float("0." + frac) if frac else 0.0) + + +def docker_now() -> float: + """The Docker daemon's current time, so that a container's start time is compared with the + clock that wrote it.""" + p = run(["docker", "info", "--format", "{{json .SystemTime}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker info failed: {p.stderr.decode(errors='replace').strip()}") + return parse_docker_time(json_loads(p.stdout.decode())) + + +class Survivors: + """Running containers of this tool whose runner is dead (docs/hygiene.md section 10). + + A container counts only when it carries the role dev-run, a valid owner and run id label, a + name that matches them, and a liveness record in this state directory that --gc's test proves + dead. A container of a run in another state directory is of unknown state and never counts. + With `stop_overdue`, a survivor whose own timeout (its timeout label, from its start, by the + daemon's clock) has passed is killed, and is no longer listed once Docker confirms it stopped: + that is the timeout its runner would have enforced. A survivor without a timeout label (from + an older runner) is never stopped, only waited for. + """ + + def __init__(self, directory: Path, project: Optional[str] = None, stop_overdue: bool = True, + report: Callable[[str], None] = _report_stderr, now: Callable[[], float] = docker_now): + self.directory, self.project, self.stop_overdue = directory, project, stop_overdue + self.report, self.now = report, now + + def scan(self) -> List[dict]: + filters = ["--filter", f"label={ROLE_LABEL}=dev-run", "--filter", f"label={RUN_ID_LABEL}"] + if self.project: + filters += ["--filter", f"label={OWNER_LABEL}={self.project}"] + p = run(["docker", "ps", "-q", "--no-trunc", *filters], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") + found = [] + for d in _inspect(["container"], p.stdout.decode().split(), all_container_ids): + labels = d["Config"].get("Labels") or {} + project, rid = labels.get(OWNER_LABEL) or "", labels.get(RUN_ID_LABEL) or "" + name = d["Name"].lstrip("/") + if (labels.get(ROLE_LABEL) != "dev-run" or not PROJECT_RE.fullmatch(project) or not RUN_ID_RE.fullmatch(rid) + or name != run_resource_names(project, rid)["container"] or not d["State"]["Running"]): + continue + if run_liveness(self.directory, rid) != "dead": + continue + found.append({"id": d["Id"], "name": name, "project": project, "run_id": rid, + "exclusive": labels.get(ADMISSION_LABEL) == "exclusive" or self._recorded_exclusive(rid), + "phase": labels.get(PHASE_LABEL), "timeout": labels.get(TIMEOUT_LABEL), + "started": d["State"].get("StartedAt") or "", + "mounts": [m.get("Name") for m in d.get("Mounts") or [] if m.get("Type") == "volume"]}) + return found + + def _recorded_exclusive(self, run_id: str) -> bool: + """Whether the dead run's record says it asked for exclusive admission.""" + try: + facts = json_loads((self.directory / "runs" / f"{run_id}.lock").read_text()) + except (OSError, ValueError): + return False + return isinstance(facts, dict) and facts.get("admission") == "exclusive" + + def deadline(self, survivor: dict) -> Optional[float]: + """When the survivor's own timeout ends, by the daemon's clock; None without a timeout label.""" + timeout = survivor.get("timeout") or "" + if not timeout.isdigit() or not survivor.get("started"): + return None + return parse_docker_time(survivor["started"]) + int(timeout) + + def stop(self, survivor: dict) -> bool: + """Kill one overdue survivor by its id; True once Docker confirms that it stopped.""" + self.report(f"devrun: stopping {survivor['name']}: its runner is dead and its " + f"{survivor['timeout']} s timeout has passed") + run(["docker", "kill", survivor["id"]], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + try: + p = run(["docker", "wait", survivor["id"]], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=60) + except subprocess.TimeoutExpired: + return False + return p.returncode == 0 + + def __call__(self) -> List[dict]: + found = self.scan() + if not self.stop_overdue or not any(self.deadline(s) is not None for s in found): + return found + now = self.now() + left = [] + for s in found: + end = self.deadline(s) + if end is not None and now >= end and self.stop(s): + continue + left.append(s) + return left + + # --------------------------------------------------------------------------- # # git-locks visibility (docs/hygiene.md section 10) # --------------------------------------------------------------------------- # @@ -743,14 +912,21 @@ class Docker: self.memory = memory self.names = run_resource_names(project, run_id) self.name = self.names["container"] - # True only between a successful `docker create` by this run and its removal, so cleanup - # never touches a container that this run did not create. - self.created = False - # Per-run volumes this run created and has not removed yet; cleanup removes only these. + # The run container: "absent", "uncertain" (a create was sent and its outcome is not + # known, for example because a signal interrupted it), or "created". Cleanup removes it + # only when this run created it, resolving "uncertain" by listing this run's labels. + self.state = "absent" + # Per-run volumes that may exist because this run asked Docker to create them, and have + # not been removed yet; cleanup removes only these. A name is added before the create is + # sent, and stays in `unconfirmed` until the create is confirmed. self.volumes: List[str] = [] + self.unconfirmed: Set[str] = set() + # Locks (a run slot, the exclusive gate, the fetch or pinned-build lock) that must stay held + # until the running container is confirmed stopped or removed (docs/hygiene.md section 10). + self.holds: List[Admission] = [] - def labels(self, role: str) -> List[str]: - return label_args(resource_labels(self.project, role, self.run_id)) + def labels(self, role: str, extra: Optional[Dict[str, str]] = None) -> List[str]: + return label_args(resource_labels(self.project, role, self.run_id, extra)) def _volume_labels(self, name: str) -> Optional[dict]: p = run(["docker", "volume", "inspect", name, "--format", "{{json .Labels}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) @@ -758,6 +934,17 @@ class Docker: return None return json_loads(p.stdout.decode() or "null") or {} + def _listed(self, kind: str) -> Optional[Set[str]]: + """The names of this run's own containers ("container") or volumes ("volume") that exist, + selected by this run's owner and run id labels; None when Docker cannot list them.""" + cmd = (["docker", "ps", "-a", "--format", "{{.Names}}"] if kind == "container" + else ["docker", "volume", "ls", "--format", "{{.Name}}"]) + p = run([*cmd, "--filter", f"label={OWNER_LABEL}={self.project}", "--filter", f"label={RUN_ID_LABEL}={self.run_id}"], + stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + return None + return set(p.stdout.decode().split()) + def ensure_volume(self, name: str, role: str, extra: Optional[Dict[str, str]] = None) -> None: """Create a shared volume, or check that the existing one is ours (and, for a target, that it is the target of this source tree). Two runs may create it at the same moment: the @@ -778,17 +965,25 @@ class Docker: name = self.names[role] if self._volume_labels(name) is not None: raise DevrunError(f"refused: volume {name} already exists (run id collision, or litter)") + # From here on the volume may exist, even if the create below is interrupted. + self.volumes.append(name) + self.unconfirmed.add(name) p = run(["docker", "volume", "create", *self.labels(role), name], stdout=subprocess.PIPE, stderr=subprocess.PIPE) if p.returncode != 0: raise DevrunError(f"docker volume create {name} failed: {p.stderr.decode(errors='replace').strip()}") - self.volumes.append(name) + self.unconfirmed.discard(name) return name def create(self, cmd: List[str], network: bool, target_volume: str, env: Optional[dict] = None, - caches_writable: bool = False) -> None: - """Create the run container. The cargo caches are writable only in the fetch phase.""" + caches_writable: bool = False, phase: str = "command", timeout: int = 300, + admission: Optional[str] = None) -> None: + """Create the run container. The cargo caches are writable only in the fetch phase. The + phase and the timeout are labels, so that another run can recognize a container whose + runner died and enforce its timeout (Survivors).""" ro = "" if caches_writable else ",readonly" - args = ["docker", "create", "--name", self.name, *self.labels("dev-run"), + args = ["docker", "create", "--name", self.name, + *self.labels("dev-run", {PHASE_LABEL: phase, TIMEOUT_LABEL: str(timeout), + **({ADMISSION_LABEL: admission} if admission else {})}), "--user", f"{UID}:{GID}", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--read-only", "--tmpfs", "/tmp:rw,nosuid,nodev,size=1g,mode=1777", "--pids-limit", "4096", "--memory", self.memory, "--memory-swap", self.memory, "--cpus", self.cpus, @@ -811,10 +1006,12 @@ class Docker: for k, v in (env or {}).items(): args += ["--env", f"{k}={v}"] args += [self.image, *cmd] + # From here on the container may exist, even if the create below is interrupted. + self.state = "uncertain" p = run(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) if p.returncode != 0: raise DevrunError(f"docker create failed: {p.stderr.decode(errors='replace').strip()}") - self.created = True + self.state = "created" def copy_in(self, tar_path: Path, dest: str) -> None: with open(tar_path, "rb") as fh: @@ -822,14 +1019,30 @@ class Docker: if p.returncode != 0: raise DevrunError(f"docker cp into {dest} failed: {p.stderr.decode(errors='replace').strip()}") - def start(self, timeout: int) -> int: + def start(self, timeout: int, holds: Iterable[Admission] = ()) -> int: + """Run the created container to its end. The `holds` stay held until the container is + confirmed stopped here, or removed by remove() or cleanup(): a timeout, a signal, or an + error in between never releases them while the container may still run.""" + self.holds.extend(holds) try: p = run(["docker", "start", "-a", self.name], timeout=timeout) except subprocess.TimeoutExpired: run(["docker", "kill", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) raise DevrunError(f"timeout after {timeout}s; container killed") + if self.holds and self.stopped(): + self.release_holds() return p.returncode + def stopped(self) -> bool: + """True when Docker confirms that the run container exists and is not running.""" + p = run(["docker", "container", "inspect", "--format", "{{.State.Running}}", self.name], + stdout=subprocess.PIPE, stderr=subprocess.DEVNULL) + return p.returncode == 0 and p.stdout.strip() == b"false" + + def release_holds(self) -> None: + while self.holds: + self.holds.pop().release() + def export(self, path: str, dest: Path) -> List[str]: p = subprocess.Popen(["docker", "cp", f"{self.name}:/scratch/out/{path}", "-"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) try: @@ -842,23 +1055,47 @@ class Docker: return written def remove(self) -> None: - """Remove the run container, only if this run created it.""" - if not self.created: - return - p = run(["docker", "rm", "-f", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - if p.returncode == 0: - self.created = False + """Remove the run container, only if this run created it, and then release the holds.""" + if self.state == "uncertain": + listed = self._listed("container") + if listed is None: + return # cannot tell; keep the holds, and cleanup reports the run as not clean + self.state = "created" if self.name in listed else "absent" + if self.state == "created": + p = run(["docker", "rm", "-f", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + if p.returncode == 0: + self.state = "absent" + else: + listed = self._listed("container") + if listed is not None and self.name not in listed: + self.state = "absent" + if self.state == "absent": + self.release_holds() def cleanup(self) -> bool: - """Remove this run's container and volumes. True when nothing this run created is left.""" + """Remove this run's container and volumes. True when nothing this run created is left. + The holds are released only once the container is gone.""" self.remove() + listed: Optional[Set[str]] = None for name in list(self.volumes): + if name in self.unconfirmed: + # The create may or may not have happened: a fresh listing of this run's labelled + # volumes decides. A volume that is not listed was never created by this run. + listed = self._listed("volume") if listed is None else listed + if listed is None: + print(f"devrun: cannot tell whether volume {name} was created; keeping the run record", file=sys.stderr) + continue + if name not in listed: + self.volumes.remove(name) + self.unconfirmed.discard(name) + continue p = run(["docker", "volume", "rm", name], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE) if p.returncode == 0: self.volumes.remove(name) + self.unconfirmed.discard(name) else: print(f"devrun: could not remove volume {name}: {p.stderr.decode(errors='replace').strip()}", file=sys.stderr) - return not self.created and not self.volumes + return self.state == "absent" and not self.volumes def exists(self) -> bool: p = run(["docker", "ps", "-a", "--filter", f"name=^{self.name}$", "--format", "{{.Names}}"], stdout=subprocess.PIPE) @@ -883,8 +1120,8 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: liveness(run_id) -> 'alive' | 'dead' | 'unknown'; only 'dead' allows removal. protected_keys: source keys of live runs' targets, or None when that is unknown (then no target volume is removed). - Selected: stopped containers and per-run volumes of dead runs, and target volumes whose source - path no longer exists. Never selected: anything without this project's owner label, anything + Selected: stopped containers and per-run volumes of dead runs, and target volumes (worktree or + pinned) whose source path no longer exists. Never selected: anything without this project's owner label, anything whose name does not match its labels, anything attached to a running container or to a container that is not itself selected, the cargo caches, and legacy fixed-name volumes. """ @@ -908,8 +1145,9 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: if state != "dead": kept.append((c["name"], f"run {rid} is {state}")) elif c.get("running"): - kept.append((c["name"], f"run {rid} is dead but its container is still running; " - f"stop it with docker stop {c['name']} and run --gc again")) + kept.append((c["name"], f"run {rid} is dead but its container is still running, within its timeout; " + f"devrun stops it when that has passed, or stop it with docker stop {c['name']}, " + f"then run --gc again")) else: remove_containers.append(c["name"]) @@ -940,12 +1178,12 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: kept.append((name, problem)) else: remove_volumes.append(name) - elif role == "target": + elif role in TARGET_ROLES: path, key = labels.get(SOURCE_PATH_LABEL), labels.get(SOURCE_KEY_LABEL) or "" if not path: kept.append((name, "no source path label (legacy per-project target); not touched")) continue - if not SOURCE_KEY_RE.fullmatch(key) or key != source_key(path) or name != target_volume_name(project, key): + if not SOURCE_KEY_RE.fullmatch(key) or key != source_key(path) or name != target_volume_name(project, key, role): kept.append((name, "its name and source labels do not match; not touched")) continue if path_exists(path): @@ -962,27 +1200,58 @@ def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: return {"containers": remove_containers, "volumes": remove_volumes, "kept": kept} -def _inspect(kind: List[str], names: List[str]) -> List[dict]: +def all_container_ids() -> Set[str]: + p = run(["docker", "ps", "-aq", "--no-trunc"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") + return set(p.stdout.decode().split()) + + +def project_volume_names(project: str) -> Set[str]: + p = run(["docker", "volume", "ls", "-q", "--filter", f"label={OWNER_LABEL}={project}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker volume ls failed: {p.stderr.decode(errors='replace').strip()}") + return set(p.stdout.decode().split()) + + +def _inspect(kind: List[str], names: List[str], listing: Callable[[], Set[str]]) -> List[dict]: + """docker inspect of the named resources (container ids, or volume names). + + A resource removed between the listing and the inspect makes the inspect exit 1 while it still + prints the others. That is accepted only when every requested name missing from the output is + also missing from a fresh `listing()`. Any other failure raises, so that a resource that still + exists is never taken for an absent one (--gc would then forget a dead run that has one). + """ if not names: return [] p = run(["docker", *kind, "inspect", *names], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - # A resource removed between the listing and the inspect makes the inspect exit 1 but still - # prints the others. - return json_loads(p.stdout.decode() or "[]") if p.stdout.strip() else [] + what = f"docker {' '.join(kind)} inspect" + err = p.stderr.decode(errors="replace").strip() + try: + found = json_loads(p.stdout.decode()) if p.stdout.strip() else [] + except ValueError: + raise DevrunError(f"{what} printed something that is not JSON: {err}") + if not isinstance(found, list): + raise DevrunError(f"{what} printed something that is not a list: {err}") + key = "Id" if kind == ["container"] else "Name" + printed = {d.get(key) for d in found if isinstance(d, dict)} + missing = [n for n in names if n not in printed] + if p.returncode != 0 or missing: + if not missing: + raise DevrunError(f"{what} failed: {err}") + still = listing() & set(missing) + if still: + raise DevrunError(f"{what} failed for {', '.join(sorted(still))}, which still exist: {err}") + return found def docker_inventory(project: str) -> tuple: - p = run(["docker", "ps", "-aq", "--no-trunc"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - if p.returncode != 0: - raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") containers = [{"name": d["Name"].lstrip("/"), "running": bool(d["State"]["Running"]), "labels": d["Config"].get("Labels") or {}, "mounts": [m.get("Name") for m in d.get("Mounts") or [] if m.get("Type") == "volume"]} - for d in _inspect(["container"], p.stdout.decode().split())] - p = run(["docker", "volume", "ls", "-q", "--filter", f"label={OWNER_LABEL}={project}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - if p.returncode != 0: - raise DevrunError(f"docker volume ls failed: {p.stderr.decode(errors='replace').strip()}") - volumes = [{"name": d["Name"], "labels": d.get("Labels") or {}} for d in _inspect(["volume"], p.stdout.decode().split())] + for d in _inspect(["container"], sorted(all_container_ids()), all_container_ids)] + volumes = [{"name": d["Name"], "labels": d.get("Labels") or {}} + for d in _inspect(["volume"], sorted(project_volume_names(project)), lambda: project_volume_names(project))] return containers, volumes @@ -996,6 +1265,17 @@ def gc_main(argv: List[str]) -> int: directory = state_dir() live = live_run_facts(directory) protected = None if live is None else {f.get("source_key") for f in live if f.get("project") == ns.project} + # A dead run's container that is past its own timeout is stopped first (dry run: only said). + survivors = Survivors(directory, project=ns.project, stop_overdue=not ns.dry_run) + if ns.dry_run: + found = survivors() + now = docker_now() if any(survivors.deadline(s) is not None for s in found) else 0.0 + for s in found: + end = survivors.deadline(s) + if end is not None and now >= end: + print(f"devrun gc: would stop container {s['name']}: its runner is dead and its {s['timeout']} s timeout has passed") + else: + survivors() containers, volumes = docker_inventory(ns.project) plan = plan_gc(ns.project, containers, volumes, lambda rid: run_liveness(directory, rid), os.path.isdir, protected) verb = "would remove" if ns.dry_run else "remove" @@ -1116,7 +1396,7 @@ def fetch_environment(env: dict, rust_version: str) -> dict: def prepare(docker: "Docker", target_volume: str, empty_target: bool) -> None: """Empty the copy and scratch volumes (and the evidence target when asked), offline.""" - docker.create(["bash", "-c", PREP_SCRIPT, "prep", "1" if empty_target else "0"], False, target_volume) + docker.create(["bash", "-c", PREP_SCRIPT, "prep", "1" if empty_target else "0"], False, target_volume, phase="prep") if docker.start(300) != 0: raise DevrunError("preparing the work and scratch volumes failed") docker.remove() @@ -1124,7 +1404,7 @@ def prepare(docker: "Docker", target_volume: str, empty_target: bool) -> None: def copy_in(docker: "Docker", target_volume: str, tar_path: Path, dest: str, env: dict) -> None: """Extract a sanitized tar into the volumes through a short-lived, offline container.""" - docker.create(["true"], False, target_volume, env) + docker.create(["true"], False, target_volume, env, phase="copy") docker.copy_in(tar_path, dest) if docker.start(300) != 0: raise DevrunError("copy-in container failed") @@ -1142,8 +1422,10 @@ def main_inner(ns: argparse.Namespace) -> int: # CI machines run one job at a time and have no lock store: no slots and no registration there. coordinated = os.environ.get("CI") != "true" directory = state_dir() - gate = Gate(directory, ns.max_runs) - record = RunRecord(directory, run_id, {"project": ns.project, "source_key": key, "pid": str(os.getpid())}) + gate = Gate(directory, ns.max_runs, survivors=Survivors(directory)) + admission_mode = "exclusive" if ns.exclusive else "ordinary" + record = RunRecord(directory, run_id, {"project": ns.project, "source_key": key, "pid": str(os.getpid()), + "admission": admission_mode}) lease = Lease(f"devrun-{ns.project}-{run_id}", f"devrun-{run_id}", f"host/docker/{docker.name}/", 2 * ns.timeout + 2 * ns.wait + 1500, f"devrun {ns.project} run {run_id}") work = Path(tempfile.mkdtemp(prefix="devrun-src-")) @@ -1158,6 +1440,7 @@ def main_inner(ns: argparse.Namespace) -> int: signal.signal(signal.SIGTERM, on_signal) signal.signal(signal.SIGINT, on_signal) clean = False + no_hold = Admission([]) try: if coordinated: lease.claim(gate.waiter("host/docker/ in the lock store (daemon maintenance?)", ns.wait, poll=2.0)) @@ -1173,11 +1456,16 @@ def main_inner(ns: argparse.Namespace) -> int: raise DevrunError(f"refused: container {docker.name} already exists (run id collision, or litter)") docker.ensure_volume(f"{ns.project}-cargo-registry", "cargo-registry") docker.ensure_volume(f"{ns.project}-cargo-git", "cargo-git") + # A pinned run (--commit, --history) builds into the worktree's pinned target, never into + # the one that --worktree runs use, and holds that target's lock from before its command + # starts until its container has stopped (docs/hygiene.md section 5). + pinned = mode in ("commit", "history") and not ns.evidence if ns.evidence: target_volume = docker.create_run_volume("target-evidence") else: - target_volume = target_volume_name(ns.project, key) - docker.ensure_volume(target_volume, "target", {SOURCE_PATH_LABEL: str(top), SOURCE_KEY_LABEL: key}) + target_role = "target-pinned" if pinned else "target" + target_volume = target_volume_name(ns.project, key, target_role) + docker.ensure_volume(target_volume, target_role, {SOURCE_PATH_LABEL: str(top), SOURCE_KEY_LABEL: key}) docker.create_run_volume("work") docker.create_run_volume("scratch") env = {"DEVRUN_HEAD": facts.get("sha") or "", "DEVRUN_BASE": facts.get("base") or ""} @@ -1190,15 +1478,17 @@ def main_inner(ns: argparse.Namespace) -> int: fetch_copy(fin, fout) prepare(docker, target_volume, ns.evidence) copy_in(docker, target_volume, fetch_tar, "/work", env) - docker.create(FETCH_COMMAND, True, target_volume, fetch_environment(env, ns.rust), caches_writable=True) + docker.create(FETCH_COMMAND, True, target_volume, fetch_environment(env, ns.rust), caches_writable=True, + phase="fetch", timeout=ns.timeout) # Cargo's own package-cache lock is $CARGO_HOME/.package-cache, on the read-only root: # cargo warns that it cannot open it and fetches without any lock. Two fetches into the # project's shared caches would then write the same files at once, so fetch phases of # one project take turns on a host file lock (docs/hygiene.md section 10). - fetch_lock = (gate.hold(f"fetch-{ns.project}.lock", f"the {ns.project} fetch lock (another run is fetching)", ns.wait) - if coordinated else Admission([])) - with fetch_lock: - fetched = docker.start(ns.timeout) + # A fetch container whose runner died still writes the caches, so it holds the lock too. + fetch_lock = (gate.hold(f"fetch-{ns.project}.lock", f"the {ns.project} fetch lock (another run is fetching)", ns.wait, + blocking=lambda s: s["project"] == ns.project and s.get("phase") in ("fetch", None)) + if coordinated else no_hold) + fetched = docker.start(ns.timeout, [fetch_lock]) if fetched != 0: raise DevrunError("cargo fetch --locked failed") docker.remove() @@ -1209,10 +1499,19 @@ def main_inner(ns: argparse.Namespace) -> int: cmd = ["bash", "-c", INNER_SCRIPT, "run", mode, "1" if ns.export_patch else "0", *ns.command] env["CARGO_NET_OFFLINE"] = "false" if ns.network else "true" env["DEVRUN_FRESH_MTIMES"] = "1" if mode == "commit" else "0" - docker.create(cmd, ns.network, target_volume, env) - admission = gate.admit(ns.exclusive, ns.wait) if coordinated else Admission([]) - with admission: - status = docker.start(ns.timeout) + # The admission mode is a label and in the record, so that if this runner dies, other runs + # know that its surviving command had the machine to itself. + docker.create(cmd, ns.network, target_volume, env, timeout=ns.timeout, admission=admission_mode) + # The pinned-target lock comes first and admission second: a run never holds a slot or the + # exclusive gate while it waits for another run of its worktree. Both stay held until the + # container is confirmed stopped or removed (Docker.start, Docker.remove). + build_lock = (gate.hold(f"pinned-{ns.project}-{key}.lock", + "the pinned build target of this worktree (another --commit or --history run of it is running)", + ns.wait, blocking=lambda s: target_volume in s["mounts"] and counts_as_command(s)) + if coordinated and pinned else no_hold) + docker.holds.append(build_lock) + admission = gate.admit(ns.exclusive, ns.wait) if coordinated else no_hold + status = docker.start(ns.timeout, [admission]) exported: List[str] = [] for path in ns.export: exported += docker.export(path, out) @@ -1224,9 +1523,16 @@ def main_inner(ns: argparse.Namespace) -> int: print(f"devrun: exit {status}; exports in {out}: {', '.join(exported) or 'none'}", file=sys.stderr) return status finally: + # A signal that arrives now must not cut the cleanup short either. + signal.signal(signal.SIGTERM, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.SIG_IGN) clean = docker.cleanup() lease.release() record.finish(clean) + # Locks still held here belong to a container that cleanup could not confirm gone. They + # are released only after the record is unlocked, which marks the run dead, so that other + # runs count the container as a survivor from the moment the locks are free. + docker.release_holds() if not clean: print(f"devrun: run {run_id} left resources behind; remove them with: devrun --gc {ns.project}", file=sys.stderr) for p in work.glob("*"): diff --git a/docs/hygiene.md b/docs/hygiene.md index df5fdb0..2cbe9e2 100644 --- a/docs/hygiene.md +++ b/docs/hygiene.md @@ -134,10 +134,11 @@ symlinks resolved), so each git worktree has its own key. | --- | --- | --- | --- | | Base image | `flyingrobots/rust-dev:` | `base-image` | Until a new toolchain replaces it | | Upstream base, the build input | `rust@sha256:` from `images/rust-dev/base-digests`, pulled by digest. It carries only upstream labels. Keep it while its digest line exists; after removing the line, remove it with `docker image rm`. | none (upstream image) | As left | -| Run container | `-dev-run-`, also labelled with the run id | `dev-run` | One run; removed by the run | +| Run container | `-dev-run-`, also labelled with the run id, `org.flyingrobots.phase=`, `org.flyingrobots.timeout=`, and on the command container `org.flyingrobots.admission=` (section 10 uses them when its runner has died) | `dev-run` | One run; removed by the run | | Cargo registry cache, at `/usr/local/cargo/registry` | `-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` | Kept | | Cargo git cache, at `/usr/local/cargo/git` | `-cargo-git` | `cargo-git` | Kept | -| Build output, iteration | `-target-`, also labelled `org.flyingrobots.source-path=` and `org.flyingrobots.source-key=` (one per worktree) | `target` | Kept; `devrun --gc` removes it once the worktree path no longer exists | +| Build output, `--worktree` runs | `-target-`, also labelled `org.flyingrobots.source-path=` and `org.flyingrobots.source-key=` (one per worktree) | `target` | Kept; `devrun --gc` removes it once the worktree path no longer exists | +| Build output, pinned runs (`--commit`, `--history`) | `-target-pinned-`, with the same source labels (one per worktree) | `target-pinned` | Kept; `devrun --gc` removes it once the worktree path no longer exists | | Build output, evidence | `-target-evidence-`, new and empty for each `--evidence` run, also labelled with the run id | `target-evidence` | One run; removed by the run | | The copy, at `/work` | `-work-`, also labelled with the run id | `work` | One run; removed by the run | | Inputs, logs, test data, exports, at `/scratch` | `-scratch-`, also labelled with the run id | `scratch` | One run; removed by the run | @@ -149,6 +150,23 @@ symlinks resolved), so each git worktree has its own key. - **Evidence runs use a fresh target.** A merge-gate run never reuses build output, so a stale cache cannot fabricate a passing result. The evidence target belongs to the one run, so concurrent evidence runs never share one. +- **What a pinned run guarantees about its build.** A `--commit` or `--history` + run without `--evidence` builds into its worktree's pinned target, which + `--worktree` runs never use. It takes that target's lock (section 10) before its + command container starts and keeps it until the container has stopped. Inside + the container the copy gets the current time (`--commit`) or is checked out + then (`--history`), so every source file is newer than any build output already + in the target: cargo rebuilds the workspace's own crates from this run's copy + and reuses only registry and git dependencies, which it identifies by version + and checksum rather than by time. No other run can write the target between the + time refresh and the end of the command, so the result does not depend on what + else runs. Limits: the command itself can still change the copy before it + builds, a build script can leave files in the target that a later pinned run + reads (the target is still not evidence), and with `CI=true` there is no lock, + which is safe only because a CI machine runs one job. The cost is a second + target per worktree that runs both modes, and pinned runs of one worktree run + their command phases one at a time; runs from different worktrees, and + `--worktree` runs, are not delayed. - A run refuses a shared volume (a cache or a target) that exists with other labels: another owner, another role, or, for a target, another source path or key. It refuses a per-run volume or a container that already exists under its @@ -173,14 +191,20 @@ project share only these, and each sharing has a reason: write. This is cargo's design and is not tested here. The writer still runs no repository-chosen code, so the poisoning rule above holds as before; what is new is only that a read and a write can happen at the same moment. -- **The iteration target of one worktree**, read-write, when two runs come from - the same worktree at once. Cargo's build-directory lock (`.cargo-lock` in the - target, a file lock that works across containers on one Docker volume) - serializes cargo's own builds in it; cargo prints "Blocking waiting for file - lock on build directory". A command that writes `/target` without cargo is not - serialized. One run's build scripts can change what the other run builds from, - which was already true of consecutive runs: the iteration target is never - evidence. Runs from different worktrees never share a target. +- **The `--worktree` target of one worktree**, read-write, when two + `--worktree` runs come from the same worktree at once. Cargo's build-directory + lock (`.cargo-lock` in the target, a file lock that works across containers on + one Docker volume) serializes cargo's own builds in it; cargo prints "Blocking + waiting for file lock on build directory". A command that writes `/target` + without cargo is not serialized. One run's build scripts can change what the + other run builds from, which was already true of consecutive runs: the + iteration target is never evidence. `--worktree` runs keep the files' own + times, so if a file is edited between the copies of two such runs, the run with + the older copy can reuse the other's build of the newer content; the same is + true of cargo on the host. +- **The pinned target of one worktree**, never at the same time: pinned runs of + one worktree take turns on its lock for their whole command phase (above). + Runs from different worktrees never share a target. ## 6. Outputs @@ -199,19 +223,26 @@ project share only these, and each sharing has a reason: timeout, and SIGTERM or SIGINT. After the first signal, further signals are ignored, so they cannot cut cleanup short. Cleanup removes only what this run created; the run's own records say what that is. +- A run records that a container or volume may exist before it asks Docker to + create it. When a signal interrupts a create, cleanup lists this run's labelled + resources to decide: one that exists is removed, one that does not was never + created. When Docker cannot answer, the run keeps its liveness record, so + `--gc` can still find the run. A signal that arrives during cleanup is ignored. - A run that is killed outright (SIGKILL, a host crash) cannot clean up. Its resources stay, labelled with its run id, and `devrun --gc ` removes - them (section 10). Its container may keep running until its command ends, - because the timeout lives in the killed process; `--gc` leaves a running - container alone and says so. + them (section 10). Its container may keep running, because the timeout lived + in the killed process. Until it stops it still counts against the run cap and + keeps exclusive runs out, and once its own timeout has passed a waiting run or + `--gc` stops it (section 10, "Survivors"). - The command runs under a timeout that is shorter than the `git-locks` lease. ## 8. Budgets and free space - Build output: at most 20 GiB per project. Logs and test data: at most 4 GiB, including at most 128 MiB of logs. -- The build output of a project is the sum over its iteration targets, one per - worktree, plus the evidence targets of the runs in progress. Each target can +- The build output of a project is the sum over its kept targets, up to two per + worktree (one for `--worktree` runs, one for pinned runs), plus the evidence + targets of the runs in progress. Each target can reach the size of a full debug build of the workspace, so a handful of worktrees, or a few concurrent evidence runs, can exceed 20 GiB, which a single per-project target could not. Until the measurement exists, remove the @@ -250,8 +281,14 @@ to themselves. boundary. No container can reach the state directory. - **A cap on command phases.** At most N command phases run at once (default 4; `--max-runs N` or `DEVRUN_MAX_RUNS`, 1 to 64). A run holds one of the slot - files `slot-0` to `slot-` only while its command runs; preparation, copy- - in, the fetch phase, and exports do not take a slot. A run admitted with a + files `slot-0` to `slot-` from just before its command container starts + until Docker confirms that the container has stopped, or until the run has + removed it; a timeout, a signal, or an error in between does not release the + slot earlier. Preparation, copy-in, the fetch phase, and exports do not take a + slot. A survivor (below) in its command phase counts as a held slot: a run is + admitted only when the free slots outnumber the survivors. A survivor of an + exclusive run keeps every ordinary run out until it stops, as its runner + would have. A run admitted with a larger N can use slots that a run with a smaller N never looks at, so the cap in force is the one each run was given. A waiting run prints one line when it starts waiting and one every 30 seconds with the time waited so far, and gives @@ -271,15 +308,50 @@ to themselves. ordinary run is admitted only by passing the `turnstile` file for an instant, and a waiting exclusive run holds the turnstile until it finishes. Ordinary runs hold `runs.lock` shared and an exclusive run holds it exclusively, so the - exclusive run starts only after every admitted ordinary run has finished. + exclusive run starts only after every admitted ordinary run has finished, and + then waits until no survivor is in its command phase, saying which container + it waits for; `--wait` bounds the whole wait. - **Fetch phases of a project take turns.** Cargo's package-cache lock is the file `$CARGO_HOME/.package-cache`, and `CARGO_HOME` is on the read-only root: cargo cannot open it, warns, and fetches without a lock. Two concurrent fetches would write the same cache files, so the fetch phase holds the per-project - file lock `fetch-.lock` while it runs. -- **Run liveness.** Each run locks `runs/.lock` before it creates any - Docker resource and keeps it locked until all of them are gone. The file also - records the project and the source key. + file lock `fetch-.lock` until its container has stopped or been + removed, and does not take it while a survivor of the same project is fetching. +- **Pinned runs of a worktree take turns.** A `--commit` or `--history` run + without `--evidence` holds `pinned--.lock` from before its + command container starts until that container has stopped or been removed, and + does not take it while a survivor's command container uses the same pinned + target. It takes this lock before it asks for a slot, so it never holds a slot + or the exclusive gate while it waits for another run of its worktree. Section 5 + says what this guarantees. +- **Survivors.** A survivor is a running container that carries the role + `dev-run`, a valid owner and run id, and a name that matches them, and whose + run's liveness file in this state directory is free: its runner is dead. It is + exclusive when its admission label or its run's record says so. + Admission, the fetch lock, and the pinned lock find survivors by these labels + with `docker ps` and `docker inspect`, but only once the file locks would admit + the run. A container of a run in another state directory, or of a live run, is + never a survivor. A survivor's timeout died with its runner, so nothing bounds it any + more. The rule: a waiting run, or `--gc`, stops (`docker kill`, then `docker + wait`) a survivor once its own timeout has passed, which is what its runner would + have done. The deadline is the container's `State.StartedAt` plus its + `org.flyingrobots.timeout` label, compared with the daemon's current time + (`docker info`), so host clock skew plays no part. Nothing stops a survivor + before its timeout, or a survivor without the timeout label (from an older + runner): those are waited for, and can be stopped by hand with `docker stop`. + Nothing else is ever stopped. A run whose cleanup cannot remove its container + unlocks its liveness file first and its slot and other locks after, so the + container counts as a survivor from the moment the locks are free. One gap is + left: when a runner is killed, the kernel releases its slot and its liveness + file during the same exit, and a run that passes the turnstile in that instant + can see the slot free and the runner still alive, and so start one run over + the cap. +- **Run liveness.** Each run locks `runs/.lock` exclusively before it + creates any Docker resource and keeps it locked until all of them are gone. + The file also records the project, the source key, and the admission mode + (`ordinary` or `exclusive`). A probe takes a shared lock without waiting: it + fails only while the run holds its exclusive lock, so two probes of a dead + run's record at the same moment both read it as dead. - **`devrun --gc `** removes, after printing each item (with `--dry-run`, only prints): - the stopped containers and the per-run volumes of dead runs. A run is dead @@ -288,9 +360,16 @@ to themselves. about use, so neither is used; `--gc` runs only when someone asks for it. A run whose liveness file is missing (another user or another state directory) is of unknown state and is left alone; - - iteration targets whose recorded worktree path no longer exists, when the - name, the source key, and the path labels agree and no live run uses that - source key. + - worktree and pinned targets whose recorded worktree path no longer exists, + when the role, the name, the source key, and the path labels agree and no + live run uses that source key. + Before it decides, it stops the project's survivors whose own timeout has + passed (with `--dry-run` it only names them); a survivor within its timeout is + kept and named. After removing, it forgets the liveness file of a dead run only + when nothing labelled with that run id is left. When `docker inspect` fails, it + accepts the failure only if every resource missing from the output is also + missing from a fresh listing; otherwise it stops with an error and forgets + nothing, so a resource that still exists is never taken for an absent one. It never removes a resource without this project's owner label or without a valid run id or source label, a resource whose name does not match its labels, a running container, a volume attached to any container that it is not diff --git a/tests/integration-devrun b/tests/integration-devrun index c956500..9447a3f 100755 --- a/tests/integration-devrun +++ b/tests/integration-devrun @@ -162,6 +162,65 @@ commit_b="$(git -C "$bin" rev-parse HEAD)" got="$("$devrun" "$project" "$bin" --commit "$commit_b" "${small[@]}" -- cargo run -q --locked 2>/dev/null)" [[ "$got" == built-from-B ]] || fail "commit B ran the stale build of commit A: $got" +echo "== two --commit runs of one worktree at once: the second never reuses the first one's build (issue #12, 1)" +# A refreshes its copy, then waits until B has refreshed its own copy too, then builds. B waits for A's +# build, then builds. Without the pinned-target lock B's copy predates A's build, and cargo reuses A's +# binary. With the lock B cannot start until A has finished, so A's wait for B ends at its bound. +pin="$work/pin"; mkdir -p "$pin/src" +printf '[package]\nname = "pin"\nversion = "0.1.0"\nedition = "2021"\n' > "$pin/Cargo.toml" +printf 'version = 4\n\n[[package]]\nname = "pin"\nversion = "0.1.0"\n' > "$pin/Cargo.lock" +printf 'fn main() { println!("built-from-P1"); }\n' > "$pin/src/main.rs" +git -C "$pin" init -q -b main && git -C "$pin" add -A && git -C "$pin" -c user.name=t -c user.email=t@invalid commit -qm P1 +commit_p1="$(git -C "$pin" rev-parse HEAD)" +printf 'fn main() { println!("built-from-P2"); }\n' > "$pin/src/main.rs" +git -C "$pin" -c user.name=t -c user.email=t@invalid commit -qam P2 +commit_p2="$(git -C "$pin" rev-parse HEAD)" +# await FILE TENTHS: wait inside the container until FILE exists in the shared target, at most TENTHS/10 s. +# shellcheck disable=SC2016 # expands inside the container. +await='await() { i=0; while [ ! -e "$1" ] && [ "$i" -lt "$2" ]; do sleep 0.1; i=$((i + 1)); done; }' +# shellcheck disable=SC2016 +pin_a="$await"'; echo "start $(date +%s.%N)"; await /target/sync-b 150; echo "build $(date +%s.%N)"; cargo run -q --locked; touch /target/sync-a; echo "end $(date +%s.%N)"' +# shellcheck disable=SC2016 +pin_b="$await"'; echo "start $(date +%s.%N)"; touch /target/sync-b; await /target/sync-a 600; cargo run -q --locked' +spawn "$project" "$pin" --commit "$commit_p1" "${small[@]}" -- bash -c "$pin_a" >"$work/pa.out" 2>"$work/pa.err" & p1=$! +pids=("$p1") +wait_for 120 "commit run A's command to start" has_line '^start ' "$work/pa.out" +spawn "$project" "$pin" --commit "$commit_p2" "${small[@]}" -- bash -c "$pin_b" >"$work/pb.out" 2>"$work/pb.err" & p2=$! +pids+=("$p2") +wait "$p1" || fail "commit run A failed: $(cat "$work/pa.err")" +wait "$p2" || fail "commit run B failed: $(cat "$work/pb.err")" +pids=() +grep -qx built-from-P1 "$work/pa.out" || fail "commit run A did not run its own build: $(cat "$work/pa.out")" +grep -qx built-from-P2 "$work/pb.out" || fail "commit run B ran another commit's build: $(grep built-from "$work/pb.out")" +grep -q 'waiting for the pinned build target of this worktree' "$work/pb.err" || fail "commit run B did not wait for run A" +before "$(field end "$work/pa.out")" "$(field start "$work/pb.out")" || fail "two pinned runs of one worktree overlapped" + +echo "== a --worktree run during a --commit run of the same worktree: the commit run builds its own source" +# C refreshes its copy, waits for W's build, then builds; W builds once C has refreshed. In one shared +# target C would reuse W's build. Pinned runs have their own target, so neither sees the other's marker +# files and both waits end at their bound. +printf 'fn main() { println!("built-from-W"); }\n' > "$pin/src/main.rs" +# shellcheck disable=SC2016 +pin_c="$await"'; echo "start $(date +%s.%N)"; touch /target/sync-c; await /target/sync-w 150; cargo run -q --locked' +# shellcheck disable=SC2016 +pin_w="$await"'; await /target/sync-c 150; cargo run -q --locked; touch /target/sync-w' +spawn "$project" "$pin" --commit "$commit_p2" "${small[@]}" -- bash -c "$pin_c" >"$work/pc.out" 2>"$work/pc.err" & p1=$! +spawn "$project" "$pin" --worktree "${small[@]}" -- bash -c "$pin_w" >"$work/pw.out" 2>"$work/pw.err" & p2=$! +pids=("$p1" "$p2") +wait "$p1" || fail "commit run C failed: $(cat "$work/pc.err")" +wait "$p2" || fail "worktree run W failed: $(cat "$work/pw.err")" +pids=() +grep -qx built-from-P2 "$work/pc.out" || fail "commit run C ran the worktree's build: $(grep built-from "$work/pc.out")" +grep -qx built-from-W "$work/pw.out" || fail "worktree run W did not run its own build: $(cat "$work/pw.out")" + +echo "== a --worktree run after a --commit run sees an edit made before the commit run (issue #9, case 1)" +printf 'fn main() { println!("built-from-E"); }\n' > "$pin/src/main.rs" +[[ "$(prun "$project" "$pin" --commit "$commit_p2" "${small[@]}" -- cargo run -q --locked 2>/dev/null)" == built-from-P2 ]] || + fail "the commit run did not build its commit" +got="$(prun "$project" "$pin" --worktree "${small[@]}" -- cargo run -q --locked 2>/dev/null)" +[[ "$got" == built-from-E ]] || fail "the worktree run reused the commit run's build: $got" +no_run_leftovers + echo "== two runs of the same project at once, from two worktrees of one repository" git -C "$repo" add -A && git -C "$repo" -c user.name=t -c user.email=t@invalid commit -qm "worktree base" git -C "$repo" worktree add -q "$work/wt2" @@ -245,6 +304,87 @@ env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc2.out" || fa [[ -z "$(leftovers_of "$rid")" ]] || fail "gc left resources of dead run $rid: $(leftovers_of "$rid")" [[ ! -e "$private/runs/$rid.lock" ]] || fail "gc did not forget the dead run" +# epoch NAME FIELD: a container's State.StartedAt or State.FinishedAt, in seconds, by the daemon's clock. +epoch() { + python3 -c 'import datetime, re, sys +m = re.fullmatch(r"(\S+?)(?:\.(\d+))?(Z|[+-]\d\d:\d\d)", sys.argv[1]) +zone = "+00:00" if m.group(3) == "Z" else m.group(3) +print("%.3f" % (datetime.datetime.fromisoformat(m.group(1) + zone).timestamp() + float("0." + (m.group(2) or "0"))))' \ + "$(docker inspect -f "{{.State.$2}}" "$1")" +} +running() { [[ "$(docker inspect -f '{{.State.Running}}' "$1" 2>/dev/null)" == true ]]; } +not_running() { ! running "$1"; } +# finished NAME: when the container stopped. Docker reports year 1 for a running container, so wait. +finished() { + wait_for 300 "container $1 to stop" not_running "$1" + local t; t="$(epoch "$1" FinishedAt)" + before 1000000000 "$t" || fail "container $1 has no finish time: $t" + echo "$t" +} +waits_or_starts() { has_line "$1" "$2" || has_line '^start ' "$3"; } + +echo "== SIGKILL: the surviving container keeps its slot, and an exclusive run waits until it ends (issue #12, 2)" +spawn "$project" "$repo" --worktree "${small[@]}" --max-runs 1 -- bash -c "$timed" timed 20 >"$work/s.out" 2>"$work/s.err" & ps_=$! +pids=("$ps_") +wait_for 120 "the run's command to start" has_line '^start ' "$work/s.out" +srid="$(run_id_of "$work/s.err")"; survivor="$project-dev-run-$srid" +kill -KILL "$ps_" +wait "$ps_" 2>/dev/null || true +running "$survivor" || fail "the container did not survive its runner" +spawn "$project" "$work/wt2" --worktree "${small[@]}" --max-runs 1 -- bash -c "$timed" timed 1 >"$work/o.out" 2>"$work/o.err" & po=$! +pids=("$po") +wait_for 120 "the ordinary run to wait or start" waits_or_starts "runner of $survivor is dead" "$work/o.err" "$work/o.out" +if has_line '^start ' "$work/o.out" && running "$survivor"; then fail "an ordinary run started beside the survivor under --max-runs 1"; fi +spawn "$project" "$repo" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 1 >"$work/x.out" 2>"$work/x.err" & px=$! +pids+=("$px") +wait_for 120 "the exclusive run to wait or start" waits_or_starts "exclusive access (the runner of $survivor is dead" "$work/x.err" "$work/x.out" +wait "$po" || fail "the ordinary run failed: $(cat "$work/o.err")" +wait "$px" || fail "the exclusive run failed: $(cat "$work/x.err")" +pids=() +survivor_end="$(finished "$survivor")" +before "$survivor_end" "$(field start "$work/o.out")" || fail "the ordinary run started before the survivor ended" +before "$survivor_end" "$(field start "$work/x.out")" || fail "the exclusive run started before the survivor ended" + +echo "== SIGKILL of an exclusive run: its surviving command still keeps ordinary runs out (review of #13)" +# The cap is the default 4, so only the exclusive mode, not the slot count, can keep the ordinary run out. +spawn "$project" "$repo" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 15 >"$work/es.out" 2>"$work/es.err" & pes=$! +pids=("$pes") +wait_for 120 "the exclusive run's command to start" has_line '^start ' "$work/es.out" +erid="$(run_id_of "$work/es.err")"; esurvivor="$project-dev-run-$erid" +kill -KILL "$pes" +wait "$pes" 2>/dev/null || true +running "$esurvivor" || fail "the exclusive run's container did not survive its runner" +spawn "$project" "$work/wt2" --worktree "${small[@]}" -- bash -c "$timed" timed 1 >"$work/eo.out" 2>"$work/eo.err" & peo=$! +pids=("$peo") +wait_for 120 "the ordinary run to wait or start" waits_or_starts "runner of $esurvivor, an exclusive run," "$work/eo.err" "$work/eo.out" +if has_line '^start ' "$work/eo.out" && running "$esurvivor"; then fail "an ordinary run started beside a dead exclusive run's command"; fi +wait "$peo" || fail "the ordinary run failed: $(cat "$work/eo.err")" +pids=() +before "$(finished "$esurvivor")" "$(field start "$work/eo.out")" || fail "the ordinary run started before the exclusive survivor ended" + +echo "== a survivor past its own timeout is stopped by the run that waits for it, and by nothing earlier" +spawn "$project" "$repo" --worktree "${small[@]}" --timeout 12 -- bash -c "$timed" timed 300 >"$work/v.out" 2>"$work/v.err" & pv=$! +pids=("$pv") +wait_for 120 "the run's command to start" has_line '^start ' "$work/v.out" +vrid="$(run_id_of "$work/v.err")"; overdue="$project-dev-run-$vrid" +kill -KILL "$pv" +wait "$pv" 2>/dev/null || true +pids=() +prun "$project" "$work/wt2" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 1 >"$work/y.out" 2>"$work/y.err" || + fail "the exclusive run failed: $(cat "$work/y.err")" +grep -q "stopping $overdue: its runner is dead and its 12 s timeout has passed" "$work/y.err" || + fail "the exclusive run did not stop the overdue survivor: $(cat "$work/y.err")" +running "$overdue" && fail "the overdue survivor is still running" +overdue_end="$(finished "$overdue")" +lived="$(awk -v a="$(epoch "$overdue" StartedAt)" -v b="$overdue_end" 'BEGIN { printf "%.1f", b - a }')" +before 12 "$lived" || fail "the survivor was stopped after $lived s, before its 12 s timeout" +before 60 "$lived" && fail "the survivor lived $lived s: nothing stopped it at its 12 s timeout" +before "$overdue_end" "$(field start "$work/y.out")" || fail "the exclusive run started before the survivor stopped" +env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc4.out" || fail "gc failed: $(cat "$work/gc4.out")" +for r in "$srid" "$erid" "$vrid"; do + [[ -z "$(leftovers_of "$r")" ]] || fail "gc left resources of dead run $r: $(leftovers_of "$r")" +done + echo "== --gc removes the target of a deleted worktree and never touches resources it cannot prove are leftovers" gone="$work/gone"; mkdir -p "$gone"; git -C "$gone" init -q -b main; printf 'x\n' > "$gone/x" prun "$project" "$gone" --worktree "${small[@]}" -- true 2>/dev/null || fail "run in the soon-deleted worktree" diff --git a/tests/test_devrun.py b/tests/test_devrun.py index 37f9258..77bad3f 100644 --- a/tests/test_devrun.py +++ b/tests/test_devrun.py @@ -251,6 +251,11 @@ def test_source_key_is_a_stable_hash_of_the_path(self): self.assertEqual(devrun.source_key("/src/geordi"), hashlib.sha256(b"/src/geordi").hexdigest()[:16]) self.assertEqual(devrun.source_key("/src/geordi"), devrun.source_key("/src/geordi")) + def test_a_worktree_has_separate_worktree_and_pinned_targets(self): + key = devrun.source_key("/src/geordi") + self.assertEqual(devrun.target_volume_name("geordi", key), f"geordi-target-{key}") + self.assertEqual(devrun.target_volume_name("geordi", key, "target-pinned"), f"geordi-target-pinned-{key}") + def test_distinct_worktrees_get_distinct_targets_with_safe_names(self): paths = ["/src/geordi", "/src/geordi-2", "/src/geordi/", "/src/a b", "/src/été", "/src/x;rm -rf"] names = [devrun.target_volume_name("geordi", devrun.source_key(p)) for p in paths] @@ -290,7 +295,8 @@ def test_container_carries_owner_role_and_run_id_labels(self): create = self.calls[-1] labels = {create[i + 1] for i, a in enumerate(create) if a == "--label"} self.assertEqual(labels, {"org.flyingrobots.owner=geordi", "org.flyingrobots.role=dev-run", - "org.flyingrobots.run-id=0123456789ab"}) + "org.flyingrobots.run-id=0123456789ab", "org.flyingrobots.phase=command", + "org.flyingrobots.timeout=300"}) def test_run_volumes_are_labelled_and_removed_by_cleanup(self): docker = self.docker() @@ -355,21 +361,23 @@ def sleep(self, seconds): class Contender: """One run asking for admission in its own thread. `attempts` counts its failed polls.""" - def __init__(self, directory, exclusive, max_runs=4): + def __init__(self, directory, exclusive, max_runs=4, survivors=None): self.clock = FakeClock() self.attempts = 0 + self.messages = [] self.changed = threading.Condition() self.admitted = threading.Event() self.let_go = threading.Event() self.done = threading.Event() self.error = None self.gate = devrun.Gate(directory, max_runs, clock=self.clock, sleep=self.clock.sleep, - report=self._report, poll=0.5, report_every=0.5) + report=self._report, poll=0.5, report_every=0.5, survivors=survivors) self.thread = threading.Thread(target=self._run, args=(exclusive,), daemon=True) self.thread.start() - def _report(self, _msg): + def _report(self, msg): with self.changed: + self.messages.append(msg) self.attempts += 1 self.changed.notify_all() @@ -597,6 +605,31 @@ def test_a_killed_run_is_dead(self): self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") self.assertEqual(devrun.live_run_facts(self.dir), []) + def test_a_dead_run_stays_dead_while_another_probe_looks_at_it(self): + # Review of #13: two probes at once. The other probe holds its (shared) lock on the record. + devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}).finish(clean=False) + other = os.open(str(self.dir / "runs" / f"{RUN_ID}.lock"), os.O_RDWR) + try: + devrun.fcntl.flock(other, devrun.fcntl.LOCK_SH | devrun.fcntl.LOCK_NB) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") + finally: + os.close(other) + + def test_concurrent_probes_of_a_dead_run_all_see_it_dead(self): + devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}).finish(clean=False) + results, start = [], threading.Barrier(8) + + def probe(): + start.wait(LONG) + for _ in range(200): + results.append(devrun.run_liveness(self.dir, RUN_ID)) + threads = [threading.Thread(target=probe) for _ in range(8)] + for t in threads: + t.start() + for t in threads: + t.join(LONG) + self.assertEqual(set(results), {"dead"}) + def test_an_unknown_run_is_not_dead(self): self.assertEqual(devrun.run_liveness(self.dir, "aaaaaaaaaaaa"), "unknown") @@ -650,10 +683,10 @@ def run_volumes(run_id): return [{"name": names[r], "labels": labels(r, run_id)} for r in devrun.PER_RUN_ROLES] -def target(path, key=None, name=None): +def target(path, key=None, name=None, role="target"): key = key or devrun.source_key(path) - return {"name": name or devrun.target_volume_name("geordi", key), - "labels": labels("target", **{devrun.SOURCE_PATH_LABEL: path, devrun.SOURCE_KEY_LABEL: key})} + return {"name": name or devrun.target_volume_name("geordi", key, role), + "labels": labels(role, **{devrun.SOURCE_PATH_LABEL: path, devrun.SOURCE_KEY_LABEL: key})} class GcSelection(unittest.TestCase): @@ -721,6 +754,16 @@ def test_a_target_whose_labels_do_not_match_its_name_is_kept(self): "labels": labels("target", **{devrun.SOURCE_KEY_LABEL: KEY_GONE})}]) self.assertEqual(plan["volumes"], []) + def test_the_pinned_target_of_a_deleted_worktree_is_selected_and_an_existing_one_kept(self): + plan = self.plan([], [target("/src/deleted", role="target-pinned"), target("/src/present", role="target-pinned")]) + self.assertEqual(plan["volumes"], ["geordi-target-pinned-" + KEY_GONE]) + + def test_a_pinned_target_named_like_a_worktree_target_is_kept(self): + # Role and name must agree: a pinned-role volume under the worktree target's name, and the reverse. + plan = self.plan([], [target("/src/deleted", role="target-pinned", name=devrun.target_volume_name("geordi", KEY_GONE)), + target("/src/deleted", name="geordi-target-pinned-" + KEY_GONE)]) + self.assertEqual(plan["volumes"], []) + def test_a_target_in_use_is_kept(self): name = devrun.target_volume_name("geordi", KEY_GONE) user = {"name": "geordi-dev-run-aaaaaaaaaaaa", "running": False, "labels": labels("dev-run", LIVE), "mounts": [name]} @@ -864,5 +907,641 @@ def test_good_arguments(self): self.assertTrue(ns.fetch) +# --------------------------------------------------------------------------- # +# A fake Docker daemon at the process boundary (devrun.run), for error paths that real Docker +# cannot be made to produce on demand: interrupted creates, failing inspects, and orderings +# around a signal. Other commands (git) run for real; git-locks is never called. +# --------------------------------------------------------------------------- # + + +def _done(cmd, code=0, out=b"", err=b""): + return devrun.subprocess.CompletedProcess(cmd, code, out, err) + + +class FakeDaemon: + """Containers and volumes in dictionaries. `hooks` maps a command prefix (a tuple) to a function + called with the command before the daemon acts on it; it may raise, change `self`, or return a + CompletedProcess to use instead of the daemon's own answer.""" + + def __init__(self): + self.volumes = {} # name -> labels + self.containers = {} # name -> {"id", "labels", "running", "mounts", "started"} + self.hooks = {} + self.after = {} # like hooks, called after the daemon acted, with (cmd, result) + self.calls = [] + self.now = "2026-11-14T03:00:00.000000000Z" + self.serial = 0 + self.real_run = devrun.run + + def install(self, case): + devrun.run = self + case.addCleanup(setattr, devrun, "run", self.real_run) + + def add_container(self, name, labels, running=True, mounts=(), started="2026-11-14T02:00:00.5Z"): + self.serial += 1 + self.containers[name] = {"id": f"{self.serial:064x}", "labels": dict(labels), "running": running, + "mounts": list(mounts), "started": started} + return self.containers[name]["id"] + + def by_ref(self, ref): + for name, c in self.containers.items(): + if ref in (name, c["id"]): + return name + return None + + def __call__(self, cmd, **kw): + if cmd and os.path.basename(cmd[0]) == "git-locks": + return _done(cmd, 0) + if not cmd or cmd[0] != "docker": + return self.real_run(cmd, **kw) + self.calls.append(cmd) + for prefix, hook in self.hooks.items(): + if tuple(cmd[:len(prefix)]) == prefix: + result = hook(cmd) + if result is not None: + return result + result = self.act(cmd) + for prefix, hook in self.after.items(): + if tuple(cmd[:len(prefix)]) == prefix: + hook(cmd, result) + return result + + @staticmethod + def filters(cmd): + out = [] + for i, a in enumerate(cmd): + if a == "--filter": + out.append(cmd[i + 1]) + return out + + @staticmethod + def matches(labels, name, filters): + for f in filters: + kind, _, value = f.partition("=") + if kind == "label": + k, eq, v = value.partition("=") + if k not in labels or (eq and labels[k] != v): + return False + elif kind == "name": + if not re.search(value, name): + return False + return True + + def act(self, cmd): + args = cmd[1:] + if args[:2] == ["volume", "inspect"]: + if "--format" in args: + labels = self.volumes.get(args[2]) + return _done(cmd, 1, b"", b"no such volume") if labels is None else _done(cmd, 0, json.dumps(labels).encode()) + names = args[2:] + found = [{"Name": n, "Labels": self.volumes[n]} for n in names if n in self.volumes] + return _done(cmd, 0 if len(found) == len(names) else 1, json.dumps(found).encode() if found else b"") + if args[:2] == ["volume", "create"]: + labels = dict(a.split("=", 1) for a in args[3:-1:2]) + self.volumes.setdefault(args[-1], labels) + return _done(cmd, 0, args[-1].encode()) + if args[:2] == ["volume", "rm"]: + if args[2] not in self.volumes: + return _done(cmd, 1, b"", b"no such volume") + if any(args[2] in c["mounts"] for c in self.containers.values()): + return _done(cmd, 1, b"", b"volume is in use") + del self.volumes[args[2]] + return _done(cmd, 0) + if args[:2] == ["volume", "ls"]: + names = [n for n, l in sorted(self.volumes.items()) if self.matches(l, n, self.filters(cmd))] + return _done(cmd, 0, "".join(n + "\n" for n in names).encode()) + if args[0] == "ps": + every = "-a" in args or "-aq" in args + rows = [(n, c) for n, c in sorted(self.containers.items()) + if (every or c["running"]) and self.matches(c["labels"], n, self.filters(cmd))] + names = "--format" in args + return _done(cmd, 0, "".join((n if names else c["id"]) + "\n" for n, c in rows).encode()) + if args[:2] == ["container", "inspect"] or args[0] == "inspect": + refs = args[2:] if args[0] == "container" else args[1:] + if "--format" in refs: + name = self.by_ref(refs[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + return _done(cmd, 0, b"true\n" if self.containers[name]["running"] else b"false\n") + found = [] + for ref in refs: + name = self.by_ref(ref) + if name is not None: + c = self.containers[name] + found.append({"Id": c["id"], "Name": "/" + name, "Config": {"Labels": c["labels"]}, + "State": {"Running": c["running"], "StartedAt": c["started"]}, + "Mounts": [{"Type": "volume", "Name": m} for m in c["mounts"]]}) + return _done(cmd, 0 if len(found) == len(refs) else 1, json.dumps(found).encode() if found else b"") + if args[0] == "create": + name = args[args.index("--name") + 1] + if name in self.containers: + return _done(cmd, 1, b"", b"Conflict") + labels = dict(args[i + 1].split("=", 1) for i, a in enumerate(args) if a == "--label") + mounts = [re.search(r"src=([^,]+)", args[i + 1]).group(1) for i, a in enumerate(args) if a == "--mount"] + self.add_container(name, labels, running=False, mounts=mounts) + self.containers[name]["cmd"] = list(args) + return _done(cmd, 0) + if args[0] == "start": + name = self.by_ref(args[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + self.containers[name]["running"] = False + return _done(cmd, 0) + if args[0] in ("kill", "stop"): + name = self.by_ref(args[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + self.containers[name]["running"] = False + return _done(cmd, 0) + if args[0] == "wait": + return _done(cmd, 0 if self.by_ref(args[-1]) else 1, b"137\n") + if args[0] == "rm": + name = self.by_ref(args[-1]) + if name is None: + return _done(cmd, 1, b"", b"No such container") + if self.containers[name]["running"] and "-f" not in args: + return _done(cmd, 1, b"", b"container is running") + del self.containers[name] + return _done(cmd, 0) + if args[0] == "info": + return _done(cmd, 0, json.dumps(self.now).encode()) + if args[0] == "cp": + return _done(cmd, 0) + raise AssertionError(f"the fake daemon does not know {cmd}") + + +def interrupt(*_): + raise devrun.DevrunError("interrupted by signal 15; removing this run's container and volumes") + + +class InterruptedCreates(unittest.TestCase): + """Finding 3: a signal between Docker's create and devrun's bookkeeping.""" + + def setUp(self): + self.daemon = FakeDaemon() + self.daemon.install(self) + self.docker = devrun.Docker("geordi", RUN_ID, "img", "1", "1g") + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.dir = pathlib.Path(self.tmp.name) + (self.dir / "runs").mkdir() + + def test_a_volume_created_just_before_a_signal_is_removed_by_cleanup(self): + self.daemon.after[("docker", "volume", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("work") + self.assertIn("geordi-work-0123456789ab", self.daemon.volumes) + self.assertTrue(self.docker.cleanup()) + self.assertEqual(self.daemon.volumes, {}) + + def test_a_signal_before_docker_created_the_volume_leaves_nothing_to_remove(self): + self.daemon.hooks[("docker", "volume", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("work") + self.assertTrue(self.docker.cleanup()) + self.assertNotIn(["docker", "volume", "rm", "geordi-work-0123456789ab"], self.daemon.calls) + + def test_the_record_is_kept_when_cleanup_cannot_tell_whether_a_volume_exists(self): + record = devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}) + self.daemon.after[("docker", "volume", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("scratch") + self.daemon.hooks[("docker", "volume", "ls")] = lambda cmd: _done(cmd, 1, b"", b"daemon unavailable") + self.daemon.hooks[("docker", "volume", "rm")] = lambda cmd: _done(cmd, 1, b"", b"daemon unavailable") + with contextlib.redirect_stderr(io.StringIO()): + clean = self.docker.cleanup() + record.finish(clean) + self.assertFalse(clean) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") # --gc still finds the run + + def test_a_volume_of_another_owner_under_the_runs_name_is_never_removed(self): + # The create is interrupted and the name turns out to belong to a volume without this run's labels. + def foreign(cmd): + self.daemon.volumes[cmd[-1]] = {devrun.OWNER_LABEL: "someone-else"} + interrupt() + self.daemon.hooks[("docker", "volume", "create")] = foreign + with self.assertRaises(devrun.DevrunError): + self.docker.create_run_volume("work") + self.assertTrue(self.docker.cleanup()) + self.assertIn("geordi-work-0123456789ab", self.daemon.volumes) + + def test_a_container_created_just_before_a_signal_is_removed_by_cleanup(self): + self.daemon.after[("docker", "create")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.create(["true"], False, "geordi-target-x") + self.assertIn(self.docker.name, self.daemon.containers) + self.assertTrue(self.docker.cleanup()) + self.assertEqual(self.daemon.containers, {}) + + def test_a_failed_create_under_a_foreign_containers_name_does_not_remove_it(self): + self.daemon.add_container("geordi-dev-run-0123456789ab", {devrun.OWNER_LABEL: "other"}, running=True) + with self.assertRaises(devrun.DevrunError): + self.docker.create(["true"], False, "geordi-target-x") + self.assertTrue(self.docker.cleanup()) + self.assertIn("geordi-dev-run-0123456789ab", self.daemon.containers) + self.assertFalse(any(c[:2] == ["docker", "rm"] for c in self.daemon.calls)) + + +class InspectFailures(unittest.TestCase): + """Finding 4: a failing docker inspect must not make existing resources look absent.""" + + def setUp(self): + self.daemon = FakeDaemon() + self.daemon.install(self) + + def test_an_inspect_that_fails_while_the_resources_still_exist_raises(self): + cid = self.daemon.add_container("geordi-dev-run-dddddddddddd", labels("dev-run", "dddddddddddd"), running=False) + self.daemon.hooks[("docker", "container", "inspect")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + with self.assertRaises(devrun.DevrunError) as ctx: + devrun._inspect(["container"], [cid], devrun.all_container_ids) + self.assertIn("still exist", str(ctx.exception)) + + def test_a_resource_removed_meanwhile_is_accepted_and_the_others_returned(self): + kept = self.daemon.add_container("geordi-dev-run-aaaaaaaaaaaa", labels("dev-run", LIVE), running=False) + gone = "f" * 64 # listed earlier, removed before the inspect: not in a fresh listing either + found = devrun._inspect(["container"], [kept, gone], devrun.all_container_ids) + self.assertEqual([d["Id"] for d in found], [kept]) + + def test_a_failing_listing_raises(self): + self.daemon.hooks[("docker", "volume", "inspect")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + self.daemon.hooks[("docker", "volume", "ls")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + with self.assertRaises(devrun.DevrunError): + devrun._inspect(["volume"], ["geordi-work-dddddddddddd"], lambda: devrun.project_volume_names("geordi")) + + def test_output_that_is_not_json_raises(self): + self.daemon.hooks[("docker", "volume", "inspect")] = lambda cmd: _done(cmd, 1, b"garbage", b"") + with self.assertRaises(devrun.DevrunError): + devrun._inspect(["volume"], ["x"], lambda: set()) + + +class StateDirEnv(unittest.TestCase): + """A temporary devrun state directory in DEVRUN_STATE_DIR, and CI unset, for the test's length.""" + + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.dir = pathlib.Path(self.tmp.name) / "state" + for k, v in ((devrun.STATE_ENV, str(self.dir)), ("CI", None), (devrun.MAX_RUNS_ENV, None)): + saved = os.environ.get(k) + self.addCleanup(lambda k=k, saved=saved: os.environ.pop(k, None) if saved is None else os.environ.__setitem__(k, saved)) + if v is None: + os.environ.pop(k, None) + else: + os.environ[k] = v + devrun.state_dir() + self.daemon = FakeDaemon() + self.daemon.install(self) + + +class GcKeepsRecords(StateDirEnv): + def test_a_dead_runs_record_is_kept_when_the_final_inspect_fails(self): + devrun.RunRecord(self.dir, DEAD, {"project": "geordi"}).finish(clean=False) + self.daemon.add_container("geordi-dev-run-" + DEAD, labels("dev-run", DEAD), running=False) + # The removal fails, and from then on Docker cannot inspect: the container still exists. + def broken(cmd): + self.daemon.hooks[("docker", "container", "inspect")] = lambda c: _done(c, 1, b"", b"daemon error") + return _done(cmd, 1, b"", b"daemon error") + self.daemon.hooks[("docker", "rm")] = broken + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()): + code = devrun.main(["--gc", "geordi"]) + self.assertNotEqual(code, 0) + self.assertTrue((self.dir / "runs" / f"{DEAD}.lock").exists(), "gc forgot a dead run that still has a container") + self.assertEqual(devrun.run_liveness(self.dir, DEAD), "dead") + + def test_a_dead_runs_record_is_forgotten_once_nothing_is_left(self): + devrun.RunRecord(self.dir, DEAD, {"project": "geordi"}).finish(clean=False) + self.daemon.add_container("geordi-dev-run-" + DEAD, labels("dev-run", DEAD), running=False) + with contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(devrun.main(["--gc", "geordi"]), 0) + self.assertFalse((self.dir / "runs" / f"{DEAD}.lock").exists()) + + +# --------------------------------------------------------------------------- # +# Finding 2: containers that outlive their runner, and admission kept until the container is gone. +# --------------------------------------------------------------------------- # + + +def survivor(name="geordi-dev-run-dddddddddddd", phase="command", mounts=(), project="geordi", exclusive=False): + return {"id": "x", "name": name, "project": project, "run_id": "dddddddddddd", "phase": phase, + "timeout": "300", "started": "", "mounts": list(mounts), "exclusive": exclusive} + + +class SurvivorAdmission(GateCase): + def contender(self, exclusive=False, max_runs=4, survivors=None): + c = Contender(self.dir, exclusive, max_runs, survivors) + self.contenders.append(c) + return c + + def test_a_survivor_holds_a_slot(self): + left = [survivor()] + a = self.contender(max_runs=2, survivors=lambda: list(left)) + self.admitted(a) # one slot for the survivor, one for a + b = self.contender(max_runs=2, survivors=lambda: list(left)) + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + self.assertTrue(any("geordi-dev-run-dddddddddddd" in m and "dead" in m for m in b.messages), b.messages) + left.clear() + self.admitted(b) + + def test_an_exclusive_survivor_keeps_every_ordinary_run_out(self): + # Review of #13: a dead --exclusive runner's command still has the machine to itself. + left = [survivor(exclusive=True)] + b = self.contender(max_runs=4, survivors=lambda: list(left)) + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + self.assertTrue(any("geordi-dev-run-dddddddddddd" in m and "exclusive" in m for m in b.messages), b.messages) + left.clear() + self.admitted(b) + + def test_survivors_of_other_phases_hold_no_slot(self): + left = [survivor(phase="fetch"), survivor(phase="prep")] + self.admitted(self.contender(max_runs=1, survivors=lambda: list(left))) + + def test_a_survivor_of_an_older_runner_without_a_phase_label_holds_a_slot(self): + left = [survivor(phase=None)] + b = self.contender(max_runs=1, survivors=lambda: list(left)) + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + left.clear() + self.admitted(b) + + def test_an_exclusive_run_waits_for_a_survivor_and_says_why(self): + left = [survivor()] + e = self.contender(exclusive=True, survivors=lambda: list(left)) + e.wait_attempts(3) + self.assertFalse(e.admitted.is_set()) + self.assertTrue(any(m.startswith("devrun: waiting for exclusive access (the runner of geordi-dev-run-dddddddddddd is dead") + for m in e.messages), e.messages) + # While it waits, new ordinary runs queue behind it. + b = self.contender() + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + left.clear() + self.admitted(e) + e.release() + self.admitted(b) + + def test_an_exclusive_run_waiting_for_a_survivor_gives_up_at_its_wait_limit(self): + gate = devrun.Gate(self.dir, 1, clock=FakeClock(), sleep=lambda s: None, report=lambda m: None, + survivors=lambda: [survivor()]) + with self.assertRaises(devrun.DevrunError) as ctx: + gate.admit(True, 0) + self.assertIn("gave up", str(ctx.exception)) + self.try_once(exclusive=True).release() # it released the turnstile and the gate + + def test_a_named_lock_waits_for_a_blocking_survivor(self): + left = [survivor(mounts=["geordi-target-pinned-k"])] + clock = FakeClock() + polls = [] + gate = devrun.Gate(self.dir, 1, clock=clock, sleep=lambda s: (polls.append(s), left.clear() if len(polls) == 3 else None), + report=lambda m: None, survivors=lambda: list(left)) + with gate.hold("pinned-geordi-k.lock", "the pinned build target", None, + blocking=lambda s: "geordi-target-pinned-k" in s["mounts"]): + self.assertEqual(len(polls), 3) + # A survivor that does not mount it does not block. + left.append(survivor(mounts=["other"])) + gate.hold("pinned-geordi-k.lock", "the pinned build target", 0, + blocking=lambda s: "geordi-target-pinned-k" in s["mounts"]).release() + + +class SurvivorScan(StateDirEnv): + def setUp(self): + super().setUp() + self.ids = {} + + def run_container(self, rid, state, running=True, owner="geordi", name=None, timeout="300", + started="2026-11-14T02:59:00.5Z", phase="command", admission=None, record_admission=None): + """A dev-run container of a run whose record is 'alive' (held by a child), 'dead', or 'unknown'.""" + if state == "dead": + facts = {"project": owner} + if record_admission: + facts["admission"] = record_admission + devrun.RunRecord(self.dir, rid, facts).finish(clean=False) + elif state == "alive": + child = subprocess.Popen([sys.executable, "-c", HOLD_RECORD, str(ROOT / "bin" / "devrun"), str(self.dir), rid], + stdout=subprocess.PIPE, stdin=subprocess.PIPE) + self.assertEqual(child.stdout.readline(), b"recorded\n") + def stop(child=child): + child.kill() + child.wait() + child.stdin.close() + child.stdout.close() + self.addCleanup(stop) + lab = labels("dev-run", rid, owner) + if phase is not None: + lab[devrun.PHASE_LABEL] = phase + if timeout is not None: + lab[devrun.TIMEOUT_LABEL] = timeout + if admission is not None: + lab[devrun.ADMISSION_LABEL] = admission + name = name or devrun.run_resource_names(owner, rid)["container"] + self.ids[rid] = self.daemon.add_container(name, lab, running=running, started=started) + return name + + def test_only_running_containers_of_dead_runs_with_valid_labels_are_survivors(self): + dead = self.run_container("d00000000001", "dead") + self.run_container("a00000000001", "alive") + self.run_container("e00000000001", "unknown") + self.run_container("d00000000002", "dead", running=False) + self.run_container("d00000000003", "dead", name="geordi-dev-run-d00000000009") # name and label disagree + self.daemon.add_container("geordi-dev-run", labels("dev-run")) # no run id + self.daemon.add_container("unrelated", {}) + found = devrun.Survivors(self.dir, report=lambda m: None)() + self.assertEqual([s["name"] for s in found], [dead]) + self.assertEqual(found[0]["phase"], "command") + + def test_a_survivor_is_exclusive_by_its_label_or_by_its_record(self): + self.run_container("d00000000001", "dead", admission="exclusive") + self.run_container("d00000000002", "dead", admission=None, record_admission="exclusive") + self.run_container("d00000000003", "dead", admission="ordinary") + self.run_container("d00000000004", "dead", admission=None) + found = {s["run_id"]: s["exclusive"] for s in devrun.Survivors(self.dir, report=lambda m: None)()} + self.assertEqual(found, {"d00000000001": True, "d00000000002": True, "d00000000003": False, "d00000000004": False}) + + def test_a_survivor_past_its_own_timeout_is_stopped_and_no_other_container(self): + overdue = self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T02:58:59.9Z") + within = self.run_container("d00000000002", "dead", timeout="60", started="2026-11-14T02:59:30Z") + legacy = self.run_container("d00000000003", "dead", timeout=None, phase=None, started="2026-11-14T01:00:00Z") + self.run_container("a00000000001", "alive", timeout="60", started="2026-11-14T01:00:00Z") + self.run_container("e00000000001", "unknown", timeout="60", started="2026-11-14T01:00:00Z") + messages = [] + found = devrun.Survivors(self.dir, report=messages.append)() + self.assertEqual(sorted(s["name"] for s in found), sorted([within, legacy])) + killed = [c[2] for c in self.daemon.calls if c[:2] == ["docker", "kill"]] + self.assertEqual(killed, [self.ids["d00000000001"]]) + self.assertFalse(self.daemon.containers[overdue]["running"]) + self.assertEqual(messages, [f"devrun: stopping {overdue}: its runner is dead and its 60 s timeout has passed"]) + + def test_gc_dry_run_stops_nothing(self): + self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T01:00:00Z") + with contextlib.redirect_stdout(io.StringIO()) as out: + self.assertEqual(devrun.main(["--gc", "geordi", "--dry-run"]), 0) + self.assertFalse(any(c[:2] in (["docker", "kill"], ["docker", "rm"]) for c in self.daemon.calls)) + self.assertIn("would stop container geordi-dev-run-d00000000001", out.getvalue()) + + def test_gc_stops_an_overdue_survivor_and_then_removes_it(self): + name = self.run_container("d00000000001", "dead", timeout="60", started="2026-11-14T01:00:00Z") + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()): + self.assertEqual(devrun.main(["--gc", "geordi"]), 0) + self.assertNotIn(name, self.daemon.containers) + + def test_gc_keeps_a_survivor_within_its_timeout(self): + name = self.run_container("d00000000001", "dead", timeout="7200", started="2026-11-14T02:59:00Z") + with contextlib.redirect_stdout(io.StringIO()) as out: + self.assertEqual(devrun.main(["--gc", "geordi"]), 0) + self.assertTrue(self.daemon.containers[name]["running"]) + self.assertIn("still running, within its timeout", out.getvalue()) + + def test_docker_times_are_read_with_their_zone(self): + self.assertEqual(devrun.parse_docker_time("2026-11-14T03:19:46Z"), 1794626386.0) + self.assertAlmostEqual(devrun.parse_docker_time("2026-11-13T19:21:36.244827538-08:00"), 1794626496.244827538, places=5) + with self.assertRaises(devrun.DevrunError): + devrun.parse_docker_time("yesterday") + + +class HoldsUntilStopped(StateDirEnv): + """The SIGTERM half of finding 2: locks handed to start() outlive a signal until the container is gone.""" + + def setUp(self): + super().setUp() + self.gate = devrun.Gate(self.dir, 1) + self.docker = devrun.Docker("geordi", RUN_ID, "img", "1", "1g") + self.docker.create(["sleep", "100"], False, "geordi-target-x") + + def slot_free(self): + try: + self.gate.admit(False, 0).release() + return True + except devrun.DevrunError: + return False + + def test_a_signal_during_the_command_keeps_the_slot_until_cleanup_removed_the_container(self): + self.daemon.hooks[("docker", "start")] = interrupt + with self.assertRaises(devrun.DevrunError): + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertFalse(self.slot_free()) + self.assertTrue(self.docker.cleanup()) + self.assertTrue(self.slot_free()) + + def test_a_container_still_running_after_start_returned_keeps_the_slot(self): + def detached(cmd): + return _done(cmd, 0) # the client returned, the container did not stop + self.daemon.containers[self.docker.name]["running"] = True + self.daemon.hooks[("docker", "start")] = detached + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertFalse(self.slot_free()) + self.docker.remove() + self.assertTrue(self.slot_free()) + + def test_a_container_that_cannot_be_removed_keeps_the_slot(self): + self.daemon.hooks[("docker", "start")] = interrupt + self.daemon.hooks[("docker", "rm")] = lambda cmd: _done(cmd, 1, b"", b"daemon error") + with self.assertRaises(devrun.DevrunError): + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertFalse(self.docker.cleanup()) + self.assertFalse(self.slot_free()) + self.docker.release_holds() # what main_inner does after the record shows the run dead + self.assertTrue(self.slot_free()) + + def test_a_normal_end_releases_the_slot_once_docker_confirms_the_stop(self): + self.docker.start(60, [self.gate.admit(False, 0)]) + self.assertTrue(self.slot_free()) + + +class RunUnderSignal(StateDirEnv): + """main() end to end against the fake daemon: what is held when cleanup removes the container.""" + + def setUp(self): + super().setUp() + for sig in (devrun.signal.SIGTERM, devrun.signal.SIGINT): + self.addCleanup(devrun.signal.signal, sig, devrun.signal.getsignal(sig)) + self.repo = pathlib.Path(self.tmp.name) / "repo" + self.repo.mkdir() + git = ["git", "-C", str(self.repo), "-c", "user.name=t", "-c", "user.email=t@invalid"] + subprocess.run(git + ["init", "-q"], check=True) + (self.repo / "x").write_text("x\n") + subprocess.run(git + ["add", "x"], check=True) + subprocess.run(git + ["commit", "-qm", "x"], check=True) + self.key = devrun.source_key(os.path.realpath(str(self.repo))) + self.out = pathlib.Path(self.tmp.name) / "out" + + def main(self, *mode): + with contextlib.redirect_stderr(io.StringIO()) as err: + code = devrun.main(["geordi", str(self.repo), *mode, "--max-runs", "1", "--wait", "0", + "--out", str(self.out), "--", "true"]) + return code, err.getvalue() + + def is_command(self, cmd): + name = self.daemon.by_ref(cmd[-1]) + return name is not None and devrun.INNER_SCRIPT in self.daemon.containers[name].get("cmd", []) + + def lock_free(self, name): + try: + devrun.Gate(self.dir, 1).hold(name, name, 0).release() + return True + except devrun.DevrunError: + return False + + def test_sigterm_during_the_command_keeps_admission_until_the_container_is_removed(self): + seen = {} + + def start(cmd): + if self.is_command(cmd): + self.daemon.containers[self.daemon.by_ref(cmd[-1])]["running"] = True + interrupt() + + def rm(cmd): + if self.is_command(cmd): + try: + devrun.Gate(self.dir, 1).admit(True, 0).release() + seen["exclusive admitted while the container existed"] = True + except devrun.DevrunError: + seen["exclusive admitted while the container existed"] = False + self.daemon.hooks[("docker", "start")] = start + self.daemon.hooks[("docker", "rm")] = rm + code, err = self.main("--worktree") + self.assertEqual(code, 2, err) + self.assertIn("interrupted by signal 15", err) + self.assertEqual(seen, {"exclusive admitted while the container existed": False}) + self.assertEqual(self.daemon.containers, {}) + devrun.Gate(self.dir, 1).admit(True, 0).release() # and released after it + + def test_the_admission_mode_is_on_the_command_container_and_in_the_record(self): + seen = [] + + def start(cmd): + if self.is_command(cmd): + c = self.daemon.containers[self.daemon.by_ref(cmd[-1])] + rid = c["labels"][devrun.RUN_ID_LABEL] + record = json.loads((self.dir / "runs" / f"{rid}.lock").read_text()) + seen.append((c["labels"].get(devrun.ADMISSION_LABEL), record.get("admission"))) + self.daemon.hooks[("docker", "start")] = start + for mode in (["--worktree", "--exclusive"], ["--worktree"]): + code, err = self.main(*mode) + self.assertEqual(code, 0, err) + self.out.rmdir() + self.assertEqual(seen, [("exclusive", "exclusive"), ("ordinary", "ordinary")]) + + def test_a_pinned_run_holds_its_targets_lock_while_the_command_runs_and_a_worktree_run_does_not(self): + lock = f"pinned-geordi-{self.key}.lock" + held = [] + mounts = [] + + def start(cmd): + if self.is_command(cmd): + held.append(not self.lock_free(lock)) + mounts.append(self.daemon.containers[self.daemon.by_ref(cmd[-1])]["mounts"]) + self.daemon.hooks[("docker", "start")] = start + for mode in (["--commit", "HEAD"], ["--worktree"]): + code, err = self.main(*mode) + self.assertEqual(code, 0, err) + self.out.rmdir() + self.assertEqual(held, [True, False]) + self.assertIn(f"geordi-target-pinned-{self.key}", mounts[0]) + self.assertIn(f"geordi-target-{self.key}", mounts[1]) + self.assertTrue(self.lock_free(lock)) + self.assertEqual(self.daemon.volumes[f"geordi-target-pinned-{self.key}"][devrun.ROLE_LABEL], "target-pinned") + + if __name__ == "__main__": unittest.main()