diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..32b14f0 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,44 @@ +# Changelog + +All notable changes to this repository are recorded here. The format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The repository has no +version numbers yet, so changes are listed under Unreleased. + +## [Unreleased] + +### Changed + +- `bin/devrun` no longer takes the machine-wide `host/heavy-work` lock or a + fixed per-project container name. Several runs, including several of the same + project and the same worktree, run at the same time. +- Each run has a run id and its own container (`-dev-run-`), copy + volume (`-work-`), and scratch volume (`-scratch-`), + labelled with `org.flyingrobots.run-id`, and removes them on every exit path. +- The iteration build output is one volume per worktree, + `-target-`, keyed by a hash of the worktree's real path and + labelled with that path. Worktrees no longer share build artifacts. +- `--evidence` builds into a new volume that belongs to the one run + (`-target-evidence-`) and is removed with it. +- `--commit` runs give the copied files the current time, so an older commit + cannot reuse output built from a newer one (part of issue #9). +- Each run registers `host/docker/-dev-run-/` in the workstation + git-locks store for visibility; a missing or broken git-locks no longer stops a + run. + +### Added + +- A cap on concurrent command phases: `--max-runs N` or `DEVRUN_MAX_RUNS` + (default 4), with a waiting message and a `--wait` limit. +- `--exclusive`, which runs a command phase alone on the machine, for + benchmarks; waiting exclusive runs are not starved by new ordinary runs. +- A per-project file lock around the fetch phase, because cargo's package-cache + lock cannot be taken on the read-only root. +- `bin/devrun --gc PROJECT [--dry-run]`, which removes the leftovers of dead + runs and the targets of deleted worktrees, and nothing it cannot prove is a + leftover. + +### Removed + +- The fixed-name volumes `-work`, `-scratch`, + `-target`, and `-target-evidence` are no longer used. They + are not removed automatically. diff --git a/README.md b/README.md index 428cdd2..9f03420 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,10 @@ containers, never on the host. - One base image per Rust toolchain, pinned to the version in each repository's `rust-toolchain.toml`. -- One guarded runner that every repository calls. The runner claims `git-locks` - keys, applies a timeout and resource limits, mounts only named volumes (never - a host path), and audits Docker for litter before and after each run. +- One guarded runner that every repository calls. The runner applies a timeout + and resource limits, mounts only named volumes (never a host path), gives each + run its own container and volumes, and lets several runs work at once under a + small cap. The rules are in [`docs/hygiene.md`](docs/hygiene.md). Work is tracked in GitHub Issues with `issue-dag/2` frontmatter. @@ -55,15 +56,66 @@ bin/devrun ninelives ~/git/ninelives --worktree --export report.txt --export-pat `/scratch/out/` in the container) and `--export-patch` (a patch from the throwaway repository), into a fresh folder outside every repository (`--out DIR`, or a new temporary folder). -- Each run holds the git-locks keys `host/heavy-work` and - `host/docker/-dev-run/`, and removes only the container it created. - `--timeout` (default 1800 s) is shorter than the lease. +- `--timeout` (default 1800 s) bounds each phase's command. - Not yet: the Docker litter audit, run receipts, free-space and volume budget checks (flyingrobots/devcontainers#5). +### Concurrent runs + +Runs do not wait for each other unless the machine is full. The rules are in +[`docs/hygiene.md`](docs/hygiene.md) sections 5 and 10. + +- Each run gets a run id and its own container (`-dev-run-`), copy + volume, and scratch volume, labelled with the run id, and removes them on + every exit path, including SIGTERM and SIGINT. +- At most 4 command phases run at once on the machine. `--max-runs N` or + `DEVRUN_MAX_RUNS=N` sets the cap; a waiting run prints how long it has waited, + and gives up after `--wait SECONDS` (default 3600). +- `--exclusive` waits until no other command phase runs and keeps them all out + until it finishes. Use it for every performance measurement: + `bin/devrun echo ~/git/echo --commit HEAD --exclusive -- cargo bench --locked`. +- Fetch phases of one project take turns (cargo's own cache lock cannot work on + the read-only root). +- With `CI=true` none of this coordination applies; CI machines run one job. +- Runs register `host/docker/-dev-run-/` in the workstation + git-locks store for visibility; without git-locks they run unregistered. + +### Build output and disk + +The iteration build output lives in one volume per worktree, +`-target-`, where the key is a hash of the worktree's real path. +Each worktree keeps its own warm cache and never sees another worktree's +artifacts. Two runs from the same worktree share it safely: cargo locks its build +directory. The cost is disk: each target can grow to a full build of the +workspace, so eight worktrees of a large workspace can hold several times the +20 GiB per-project budget. `--evidence` builds into a new, empty volume that +belongs to the one run and is removed with it. + +`--commit` runs give the copied files the current time, so an older commit never +reuses output built from a newer one in the same worktree's target. `--worktree` +runs keep the files' own times, like cargo on the host. A `--worktree` run after a +`--commit` run of the same worktree can still reuse the commit's output for files +that are older than that run (flyingrobots/devcontainers#9); touch the changed +files, or use `--evidence`, when that matters. + +### Leftovers: `bin/devrun --gc` + +```sh +bin/devrun --gc bunny --dry-run # list what would be removed +bin/devrun --gc bunny # remove it +``` + +`--gc` removes the stopped containers and the volumes of runs that died without +cleaning up (killed with SIGKILL, or a crash), and the targets of worktrees that +no longer exist. A run counts as dead only when its liveness lock in the devrun +state directory is free, which proves its process exited. `--gc` never removes a +running container, a volume in use, the cargo caches, or anything without a +valid run id or source label, such as the fixed-name volumes of older runner +versions (remove those by hand once nothing uses them). + Test it: ```sh bin/devrun devcontainers . --worktree -- python3 -m unittest discover -s tests # unit tests, in a container -tests/integration-devrun # real Docker +tests/integration-devrun # real Docker, on the host ``` diff --git a/bin/devrun b/bin/devrun index e67cd39..81ff613 100755 --- a/bin/devrun +++ b/bin/devrun @@ -3,33 +3,41 @@ Rules: docs/hygiene.md (ULTRAMAX PARANOIA). The repository is copied in, never mounted. The tarball is checked and rewritten on the host before Docker sees it. -Each run claims git-locks keys, uses fixed names and owner labels, and removes -its container on every exit path. +Each run has its own id, container, and copy and scratch volumes, labelled with +the owner, the role, and the run id, and removes them on every exit path. Runs +wait for one of a small number of command slots, not for a machine-wide lock +(docs/hygiene.md section 10). Usage: devrun PROJECT REPO (--commit SHA | --history BASE..HEAD | --worktree) [--rust VERSION | --image IMAGE] [--fetch] [--network] [--evidence] [--export PATH]... [--export-patch] [--out DIR] [--timeout SECONDS] - [--cpus N] [--memory SIZE] -- COMMAND [ARG...] + [--cpus N] [--memory SIZE] [--max-runs N] [--exclusive] [--wait SECONDS] + -- COMMAND [ARG...] + devrun --gc PROJECT [--dry-run] """ from __future__ import annotations import argparse +import fcntl import hashlib import io +import json import os import posixpath import re import shlex +import shutil import signal import stat import subprocess import sys import tarfile import tempfile +import time import uuid from pathlib import Path -from typing import Iterable, List, Optional +from typing import Callable, Dict, Iterable, List, Optional, Set UID = 10001 GID = 10001 @@ -37,9 +45,20 @@ OWNER_LABEL = "org.flyingrobots.owner" ROLE_LABEL = "org.flyingrobots.role" PROJECT_RE = re.compile(r"[a-z0-9][a-z0-9-]{0,39}") SHA_RE = re.compile(r"[0-9a-f]{7,40}") +RUN_ID_LABEL = "org.flyingrobots.run-id" +SOURCE_PATH_LABEL = "org.flyingrobots.source-path" +SOURCE_KEY_LABEL = "org.flyingrobots.source-key" +RUN_ID_RE = re.compile(r"[0-9a-f]{12}") +SOURCE_KEY_RE = re.compile(r"[0-9a-f]{16}") LOCK_STORE = "/Users/Shared/git-locks/workstation.git" -LOCK_ENV = "DEVRUN_LOCK_JOB" DEFAULT_RUST = "1.96.0" +STATE_ENV = "DEVRUN_STATE_DIR" +MAX_RUNS_ENV = "DEVRUN_MAX_RUNS" +DEFAULT_MAX_RUNS = 4 +MAX_RUNS_LIMIT = 64 +DEFAULT_WAIT = 3600 +# Roles of the volumes that belong to one run and are removed by it (docs/hygiene.md section 5). +PER_RUN_ROLES = ("work", "scratch", "target-evidence") class DevrunError(Exception): @@ -337,6 +356,10 @@ if [ "$mode" = history ]; then test "$(git rev-parse HEAD)" = "$DEVRUN_HEAD" || { echo "devrun: clone HEAD does not match $DEVRUN_HEAD" >&2; exit 97; } else if find . -name .git -print -quit | grep -q .; then echo "devrun: .git found in the copy" >&2; exit 98; fi + # git archive gives every file the commit time, which can be older than build output that the + # shared iteration target already holds; cargo would then reuse that output (issue #9). Commit + # mode therefore gives the copy the current time. Worktree mode keeps the files' own times. + if [ "${DEVRUN_FRESH_MTIMES:-0}" = 1 ]; then find . -mindepth 1 -exec touch -h -- {} +; fi git_q init -q . git_q add -A git_q commit -q --allow-empty -m "devrun baseline" @@ -353,8 +376,9 @@ fi exit "$status" ''' -# Empties the copy and scratch volumes completely, hidden entries included, so nothing one run -# plants (such as a .cargo/config.toml) reaches the next (docs/hygiene.md section 2). +# Empties the copy and scratch volumes completely, hidden entries included, so nothing the fetch +# phase leaves (such as a .cargo/config.toml) reaches the command phase (docs/hygiene.md section 2). +# The volumes belong to one run and start empty; emptying them again costs little. PREP_SCRIPT = r''' set -euo pipefail find /work /scratch -mindepth 1 -delete @@ -367,30 +391,398 @@ if [ "$1" = 1 ]; then find /target -mindepth 1 -delete; fi FETCH_COMMAND = ["bash", "-c", 'mkdir -p "$HOME" "$TMPDIR" && exec cargo fetch --locked'] +# --------------------------------------------------------------------------- # +# Names and labels (docs/hygiene.md section 5) +# --------------------------------------------------------------------------- # + + +def new_run_id() -> str: + """Twelve hex digits: unique enough for concurrent runs on one machine, short in names.""" + return uuid.uuid4().hex[:12] + + +def run_resource_names(project: str, run_id: str) -> Dict[str, str]: + """The names of the resources that belong to one run and that the run removes.""" + return {"container": f"{project}-dev-run-{run_id}", + "work": f"{project}-work-{run_id}", + "scratch": f"{project}-scratch-{run_id}", + "target-evidence": f"{project}-target-evidence-{run_id}"} + + +def source_key(top: str) -> str: + """A stable key for a source tree: the first 16 hex digits of SHA-256 over its real path.""" + return hashlib.sha256(os.fsencode(top)).hexdigest()[:16] + + +def target_volume_name(project: str, key: str) -> str: + """The iteration target of one source tree (one git worktree).""" + return f"{project}-target-{key}" + + +def resource_labels(project: str, role: str, run_id: Optional[str] = None, + extra: Optional[Dict[str, str]] = None) -> Dict[str, str]: + labels = {OWNER_LABEL: project, ROLE_LABEL: role} + if run_id is not None: + labels[RUN_ID_LABEL] = run_id + labels.update(extra or {}) + return labels + + +def label_args(labels: Dict[str, str]) -> List[str]: + args: List[str] = [] + for k, v in labels.items(): + args += ["--label", f"{k}={v}"] + return args + + +# --------------------------------------------------------------------------- # +# Concurrency (docs/hygiene.md section 10) +# --------------------------------------------------------------------------- # +# +# All coordination uses fcntl.flock on files in a per-user state directory. The kernel drops a +# flock when its process exits, however it exits, so no crash leaves a stale lock behind. A lock +# belongs to an open file description, so two opens in one process exclude each other like two +# processes do (the unit tests rely on this, with threads). + + +def state_dir(env: Optional[Dict[str, str]] = None) -> Path: + """The per-user directory for slot, fetch, and run-liveness lock files, created 0700.""" + env = os.environ if env is None else env + raw = env.get(STATE_ENV) + if raw: + path = Path(raw) + else: + base = env.get("XDG_STATE_HOME") or os.path.join(os.path.expanduser("~"), ".local", "state") + path = Path(base) / "devrun" + path.mkdir(mode=0o700, parents=True, exist_ok=True) + (path / "runs").mkdir(mode=0o700, exist_ok=True) + st = os.stat(path) + if st.st_uid != os.getuid() or st.st_mode & 0o022: + raise DevrunError(f"refused: state directory {path} is not owned by this user or is writable by others") + return path + + +def _open_lock(path: Path, create: bool = True) -> int: + return os.open(str(path), os.O_RDWR | (os.O_CREAT if create else 0), 0o600) + + +def _try_lock(fd: int, mode: int) -> bool: + try: + fcntl.flock(fd, mode | fcntl.LOCK_NB) + return True + except BlockingIOError: + return False + + +class Waiter: + """Polls with an injected clock and sleep, and reports how long it has waited. + + Prints one line when the wait starts and one every `report_every` seconds after that. + Raises DevrunError once `limit` seconds have passed (None waits forever). + """ + + def __init__(self, what: str, limit: Optional[float], clock: Callable[[], float], + sleep: Callable[[float], None], report: Callable[[str], None], + poll: float, report_every: float): + self.what, self.limit, self.clock, self.sleep = what, limit, clock, sleep + self.report, self.poll, self.report_every = report, poll, report_every + self.start = clock() + self.next_report = self.start + + def wait(self) -> None: + now = self.clock() + waited = now - self.start + if self.limit is not None and waited >= self.limit: + raise DevrunError(f"gave up after waiting {int(waited)} s for {self.what}") + if now >= self.next_report: + self.report(f"devrun: waiting for {self.what}; waited {int(waited)} s so far") + self.next_report = now + self.report_every + self.sleep(self.poll) + + +def _report_stderr(msg: str) -> None: + print(msg, file=sys.stderr, flush=True) + + +class Admission: + """Held lock files; release() (or leaving the with block) closes them, which drops the locks.""" + + def __init__(self, fds: List[int]): + self.fds = fds + + def release(self) -> None: + while self.fds: + os.close(self.fds.pop()) + + def __enter__(self) -> "Admission": + return self + + def __exit__(self, *exc) -> None: + self.release() + + +class Gate: + """At most `max_runs` command phases at once; an exclusive run has the machine to itself. + + Files in the state directory: + turnstile an ordinary run passes it only for the instant of its own admission; an + exclusive run holds it from the moment it starts waiting until it finishes, + so once an exclusive run waits, no new ordinary run is admitted. + runs.lock ordinary runs hold it shared, an exclusive run holds it exclusively, so the + exclusive run starts only after every admitted ordinary run has finished. + slot- an ordinary run holds one of slot-0 .. slot-. + An ordinary run takes the three in one attempt or none, and retries; it never holds the shared + lock while it waits for a slot, so it cannot delay an exclusive run that is already waiting. + """ + + def __init__(self, directory: Path, max_runs: int, clock: Callable[[], float] = time.monotonic, + sleep: Callable[[float], None] = time.sleep, report: Callable[[str], None] = _report_stderr, + poll: float = 0.5, report_every: float = 30.0): + if not 1 <= max_runs <= MAX_RUNS_LIMIT: + raise DevrunError(f"the run cap must be between 1 and {MAX_RUNS_LIMIT}") + self.directory, self.max_runs = directory, max_runs + self.clock, self.sleep, self.report = clock, sleep, report + self.poll, self.report_every = poll, report_every + + def waiter(self, what: str, limit: Optional[float], poll: Optional[float] = None) -> Waiter: + return Waiter(what, limit, self.clock, self.sleep, self.report, poll or self.poll, self.report_every) + + def admit(self, exclusive: bool, limit: Optional[float]) -> Admission: + opened: List[int] = [] + try: + turnstile = _open_lock(self.directory / "turnstile") + opened.append(turnstile) + shared = _open_lock(self.directory / "runs.lock") + opened.append(shared) + if exclusive: + return self._admit_exclusive(turnstile, shared, opened, limit) + return self._admit_ordinary(turnstile, shared, opened, limit) + except BaseException: + for fd in opened: + os.close(fd) + raise + + def _admit_exclusive(self, turnstile: int, shared: int, opened: List[int], limit: Optional[float]) -> Admission: + w = self.waiter("exclusive access (another exclusive run is queued or running)", limit) + while not _try_lock(turnstile, fcntl.LOCK_EX): + w.wait() + w.what = "exclusive access (ordinary runs are finishing; new ones wait behind this run)" + while not _try_lock(shared, fcntl.LOCK_EX): + w.wait() + return Admission(opened) + + def _admit_ordinary(self, turnstile: int, shared: int, opened: List[int], limit: Optional[float]) -> Admission: + slots = [] + for i in range(self.max_runs): + slots.append(_open_lock(self.directory / f"slot-{i}")) + opened.append(slots[-1]) + w = self.waiter(f"a run slot (at most {self.max_runs} at once, or an exclusive run is queued)", limit) + while True: + got = None + if _try_lock(turnstile, fcntl.LOCK_EX): + try: + if _try_lock(shared, fcntl.LOCK_SH): + got = next((fd for fd in slots if _try_lock(fd, fcntl.LOCK_EX)), None) + if got is None: + fcntl.flock(shared, fcntl.LOCK_UN) + finally: + fcntl.flock(turnstile, fcntl.LOCK_UN) + if got is not None: + keep = [shared, got] + for fd in opened: + if fd not in keep: + os.close(fd) + opened[:] = keep + return Admission(list(keep)) + w.wait() + + def hold(self, name: str, what: str, limit: Optional[float]) -> Admission: + """An exclusive lock on one named file, such as a project's fetch lock.""" + fd = _open_lock(self.directory / name) + try: + w = self.waiter(what, limit) + while not _try_lock(fd, fcntl.LOCK_EX): + w.wait() + except BaseException: + os.close(fd) + raise + return Admission([fd]) + + +class RunRecord: + """The run's liveness lock: runs/.lock, locked for as long as the run lives. + + The run locks it before it creates any Docker resource and removes it only after all of them + are gone. `--gc` treats a run as dead only when it can lock this file itself, which is proof + that the process holding it has exited (the kernel drops a flock on any exit, SIGKILL + included). Process ids can be reused and an age proves nothing, so neither is used. + """ + + def __init__(self, directory: Path, run_id: str, facts: Dict[str, str]): + self.path = directory / "runs" / f"{run_id}.lock" + try: + self.fd: Optional[int] = os.open(str(self.path), os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o600) + except FileExistsError: + raise DevrunError(f"refused: run record {self.path} already exists (run id collision)") + if not _try_lock(self.fd, fcntl.LOCK_EX): + raise DevrunError(f"could not lock the new run record {self.path}") + os.write(self.fd, json.dumps(facts, sort_keys=True).encode()) + + def finish(self, clean: bool) -> None: + """Drop the lock. The file is removed only when every resource of the run is gone, so a + run that could not clean up stays visible to --gc as dead.""" + if self.fd is None: + return + if clean: + try: + self.path.unlink() + except FileNotFoundError: + pass + os.close(self.fd) + self.fd = None + + +def run_liveness(directory: Path, run_id: str) -> str: + """'alive', 'dead', or 'unknown' for a run id, read from its liveness lock without waiting.""" + path = directory / "runs" / f"{run_id}.lock" + try: + fd = _open_lock(path, create=False) + except FileNotFoundError: + return "unknown" + try: + if not _try_lock(fd, fcntl.LOCK_EX): + return "alive" + held = os.fstat(fd) + try: + now = os.stat(str(path)) + except FileNotFoundError: + return "unknown" # the run finished and removed its record meanwhile + return "dead" if (held.st_dev, held.st_ino) == (now.st_dev, now.st_ino) else "unknown" + finally: + os.close(fd) + + +def live_run_facts(directory: Path) -> Optional[List[dict]]: + """The facts of every live run, or None when a live run's record cannot be read.""" + facts = [] + for entry in sorted((directory / "runs").glob("*.lock")): + run_id = entry.name[:-len(".lock")] + if not RUN_ID_RE.fullmatch(run_id) or run_liveness(directory, run_id) != "alive": + continue + try: + facts.append(json_loads(entry.read_text())) + except (OSError, ValueError): + return None + return facts + + +# --------------------------------------------------------------------------- # +# git-locks visibility (docs/hygiene.md section 10) +# --------------------------------------------------------------------------- # + + +class Lease: + """Registers the run's container under host/docker// in the workstation store. + + The key is unique, so it never conflicts with another run. It keeps the workstation registry + accurate and keeps daemon-wide maintenance (which claims host/docker/) out while runs are + active. A missing or broken git-locks only prints one line: it never stops a run. A refusal + (exit 1) means a holder of host/docker/; the run waits for it like for a slot. + """ + + def __init__(self, job: str, holder: str, key: str, ttl: int, note: str, + executable: Optional[str] = None, env: Optional[Dict[str, str]] = None): + self.job, self.holder, self.key, self.ttl, self.note = job, holder, key, ttl, note + self.executable = executable if executable is not None else shutil.which("git-locks") + base = dict(os.environ if env is None else env) + base.setdefault("GIT_LOCKS_STORE", LOCK_STORE) + self.env = base + self.held = False + + def _call(self, args: List[str]) -> Optional[int]: + try: + p = run([self.executable, *args], env=self.env, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=60) + except (OSError, subprocess.TimeoutExpired) as exc: + print(f"devrun: git-locks failed ({exc}); continuing without registration", file=sys.stderr) + return None + return p.returncode + + def claim(self, waiter: Waiter) -> bool: + if not self.executable: + print("devrun: git-locks not found; the run is not registered in the lock store", file=sys.stderr) + return False + while True: + code = self._call(["claim", "--job", self.job, "--holder", self.holder, "--ttl", str(self.ttl), + "--note", self.note, self.key]) + if code == 0: + self.held = True + return True + if code != 1: + if code is not None: + print(f"devrun: git-locks claim exited {code}; continuing without registration", file=sys.stderr) + return False + waiter.wait() + + def release(self) -> None: + if self.held: + self.held = False + self._call(["release", "--job", self.job]) + + +# --------------------------------------------------------------------------- # +# Containers (docs/hygiene.md sections 2, 3, 5, 7) +# --------------------------------------------------------------------------- # + + class Docker: - def __init__(self, project: str, image: str, cpus: str, memory: str, network_default: bool): + def __init__(self, project: str, run_id: str, image: str, cpus: str, memory: str): self.project = project + self.run_id = run_id self.image = image self.cpus = cpus self.memory = memory - self.name = f"{project}-dev-run" + self.names = run_resource_names(project, run_id) + self.name = self.names["container"] # True only between a successful `docker create` by this run and its removal, so cleanup - # never touches a container that another run (or litter) owns under the same name. + # never touches a container that this run did not create. self.created = False + # Per-run volumes this run created and has not removed yet; cleanup removes only these. + self.volumes: List[str] = [] def labels(self, role: str) -> List[str]: - return ["--label", f"{OWNER_LABEL}={self.project}", "--label", f"{ROLE_LABEL}={role}"] + return label_args(resource_labels(self.project, role, self.run_id)) - def ensure_volume(self, name: str, role: str) -> None: + def _volume_labels(self, name: str) -> Optional[dict]: p = run(["docker", "volume", "inspect", name, "--format", "{{json .Labels}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) - if p.returncode == 0: - labels = json_loads(p.stdout.decode() or "null") or {} - if labels.get(OWNER_LABEL) != self.project: - raise DevrunError(f"refused: volume {name} exists but is not owned by {self.project}") - return + if p.returncode != 0: + return None + return json_loads(p.stdout.decode() or "null") or {} + + def ensure_volume(self, name: str, role: str, extra: Optional[Dict[str, str]] = None) -> None: + """Create a shared volume, or check that the existing one is ours (and, for a target, that + it is the target of this source tree). Two runs may create it at the same moment: the + labels are checked again after the create.""" + want = resource_labels(self.project, role, None, extra) + labels = self._volume_labels(name) + if labels is None: + p = run(["docker", "volume", "create", *label_args(want), name], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker volume create {name} failed: {p.stderr.decode(errors='replace').strip()}") + labels = self._volume_labels(name) or {} + for k, v in want.items(): + if labels.get(k) != v: + raise DevrunError(f"refused: volume {name} exists but its label {k} is not {v!r}") + + def create_run_volume(self, role: str) -> str: + """Create one of this run's own volumes. It must not exist yet.""" + name = self.names[role] + if self._volume_labels(name) is not None: + raise DevrunError(f"refused: volume {name} already exists (run id collision, or litter)") p = run(["docker", "volume", "create", *self.labels(role), name], stdout=subprocess.PIPE, stderr=subprocess.PIPE) if p.returncode != 0: raise DevrunError(f"docker volume create {name} failed: {p.stderr.decode(errors='replace').strip()}") + self.volumes.append(name) + return name def create(self, cmd: List[str], network: bool, target_volume: str, env: Optional[dict] = None, caches_writable: bool = False) -> None: @@ -406,8 +798,8 @@ class Docker: "--mount", f"type=volume,src={self.project}-cargo-registry,dst=/usr/local/cargo/registry{ro}", "--mount", f"type=volume,src={self.project}-cargo-git,dst=/usr/local/cargo/git{ro}", "--mount", f"type=volume,src={target_volume},dst=/target", - "--mount", f"type=volume,src={self.project}-work,dst=/work", - "--mount", f"type=volume,src={self.project}-scratch,dst=/scratch", + "--mount", f"type=volume,src={self.names['work']},dst=/work", + "--mount", f"type=volume,src={self.names['scratch']},dst=/scratch", "--env", "HOME=/tmp/home", # Docker mounts --tmpfs noexec by default, so test binaries (rustdoc's doctests) # cannot run from /tmp. TMPDIR is a separate tmpfs that allows execution, capped at @@ -453,8 +845,20 @@ class Docker: """Remove the run container, only if this run created it.""" if not self.created: return - run(["docker", "rm", "-f", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) - self.created = False + p = run(["docker", "rm", "-f", self.name], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + if p.returncode == 0: + self.created = False + + def cleanup(self) -> bool: + """Remove this run's container and volumes. True when nothing this run created is left.""" + self.remove() + for name in list(self.volumes): + p = run(["docker", "volume", "rm", name], stdout=subprocess.DEVNULL, stderr=subprocess.PIPE) + if p.returncode == 0: + self.volumes.remove(name) + else: + print(f"devrun: could not remove volume {name}: {p.stderr.decode(errors='replace').strip()}", file=sys.stderr) + return not self.created and not self.volumes def exists(self) -> bool: p = run(["docker", "ps", "-a", "--filter", f"name=^{self.name}$", "--format", "{{.Names}}"], stdout=subprocess.PIPE) @@ -462,15 +866,185 @@ class Docker: def json_loads(text: str): - import json return json.loads(text) +# --------------------------------------------------------------------------- # +# Leftovers of dead runs and deleted worktrees: devrun --gc PROJECT +# --------------------------------------------------------------------------- # + + +def plan_gc(project: str, containers: List[dict], volumes: List[dict], liveness: Callable[[str], str], + path_exists: Callable[[str], bool], protected_keys: Optional[Set[str]]) -> dict: + """Choose what --gc removes. Pure: the inventory and the checks come in as arguments. + + containers: every container on the daemon, as {"name", "running", "labels", "mounts"}. + volumes: the volumes labelled with this project's owner label, as {"name", "labels"}. + liveness(run_id) -> 'alive' | 'dead' | 'unknown'; only 'dead' allows removal. + protected_keys: source keys of live runs' targets, or None when that is unknown (then no + target volume is removed). + Selected: stopped containers and per-run volumes of dead runs, and target volumes whose source + path no longer exists. Never selected: anything without this project's owner label, anything + whose name does not match its labels, anything attached to a running container or to a + container that is not itself selected, the cargo caches, and legacy fixed-name volumes. + """ + remove_containers: List[str] = [] + remove_volumes: List[str] = [] + kept: List[tuple] = [] + attached: Dict[str, List[dict]] = {} + for c in containers: + for m in c.get("mounts") or []: + attached.setdefault(m, []).append(c) + + for c in containers: + labels = c.get("labels") or {} + if labels.get(OWNER_LABEL) != project or labels.get(ROLE_LABEL) != "dev-run": + continue + rid = labels.get(RUN_ID_LABEL) or "" + if not RUN_ID_RE.fullmatch(rid) or c["name"] != run_resource_names(project, rid)["container"]: + kept.append((c["name"], "no valid run id label (legacy or foreign); not touched")) + continue + state = liveness(rid) + if state != "dead": + kept.append((c["name"], f"run {rid} is {state}")) + elif c.get("running"): + kept.append((c["name"], f"run {rid} is dead but its container is still running; " + f"stop it with docker stop {c['name']} and run --gc again")) + else: + remove_containers.append(c["name"]) + + def attachments_ok(name: str) -> Optional[str]: + for c in attached.get(name, []): + if c.get("running"): + return f"attached to running container {c['name']}" + if c["name"] not in remove_containers: + return f"attached to container {c['name']}, which is not being removed" + return None + + for v in volumes: + name, labels = v["name"], v.get("labels") or {} + if labels.get(OWNER_LABEL) != project: + continue + role = labels.get(ROLE_LABEL) + if role in PER_RUN_ROLES: + rid = labels.get(RUN_ID_LABEL) or "" + if not RUN_ID_RE.fullmatch(rid) or name != run_resource_names(project, rid)[role]: + kept.append((name, "no valid run id label (legacy fixed-name volume); not touched")) + continue + state = liveness(rid) + if state != "dead": + kept.append((name, f"run {rid} is {state}")) + continue + problem = attachments_ok(name) + if problem: + kept.append((name, problem)) + else: + remove_volumes.append(name) + elif role == "target": + path, key = labels.get(SOURCE_PATH_LABEL), labels.get(SOURCE_KEY_LABEL) or "" + if not path: + kept.append((name, "no source path label (legacy per-project target); not touched")) + continue + if not SOURCE_KEY_RE.fullmatch(key) or key != source_key(path) or name != target_volume_name(project, key): + kept.append((name, "its name and source labels do not match; not touched")) + continue + if path_exists(path): + kept.append((name, f"worktree {path} exists")) + continue + if protected_keys is None or key in protected_keys: + kept.append((name, "a live run may be using it")) + continue + problem = attachments_ok(name) + if problem: + kept.append((name, problem)) + else: + remove_volumes.append(name) + return {"containers": remove_containers, "volumes": remove_volumes, "kept": kept} + + +def _inspect(kind: List[str], names: List[str]) -> List[dict]: + if not names: + return [] + p = run(["docker", *kind, "inspect", *names], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + # A resource removed between the listing and the inspect makes the inspect exit 1 but still + # prints the others. + return json_loads(p.stdout.decode() or "[]") if p.stdout.strip() else [] + + +def docker_inventory(project: str) -> tuple: + p = run(["docker", "ps", "-aq", "--no-trunc"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker ps failed: {p.stderr.decode(errors='replace').strip()}") + containers = [{"name": d["Name"].lstrip("/"), "running": bool(d["State"]["Running"]), + "labels": d["Config"].get("Labels") or {}, + "mounts": [m.get("Name") for m in d.get("Mounts") or [] if m.get("Type") == "volume"]} + for d in _inspect(["container"], p.stdout.decode().split())] + p = run(["docker", "volume", "ls", "-q", "--filter", f"label={OWNER_LABEL}={project}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if p.returncode != 0: + raise DevrunError(f"docker volume ls failed: {p.stderr.decode(errors='replace').strip()}") + volumes = [{"name": d["Name"], "labels": d.get("Labels") or {}} for d in _inspect(["volume"], p.stdout.decode().split())] + return containers, volumes + + +def gc_main(argv: List[str]) -> int: + ap = argparse.ArgumentParser(prog="devrun --gc", description="Remove leftovers of dead runs and targets of deleted worktrees.") + ap.add_argument("project") + ap.add_argument("--dry-run", action="store_true", help="list what would be removed, remove nothing") + ns = ap.parse_args(argv) + if not PROJECT_RE.fullmatch(ns.project): + raise DevrunError(f"invalid project name {ns.project!r} (lowercase letters, digits, hyphens)") + directory = state_dir() + live = live_run_facts(directory) + protected = None if live is None else {f.get("source_key") for f in live if f.get("project") == ns.project} + containers, volumes = docker_inventory(ns.project) + plan = plan_gc(ns.project, containers, volumes, lambda rid: run_liveness(directory, rid), os.path.isdir, protected) + verb = "would remove" if ns.dry_run else "remove" + failed = False + for name, reason in plan["kept"]: + print(f"devrun gc: keep {name}: {reason}") + for name in plan["containers"]: + print(f"devrun gc: {verb} container {name}") + # No -f: Docker itself refuses to remove a container that started running meanwhile. + if not ns.dry_run and run(["docker", "rm", name], stdout=subprocess.DEVNULL).returncode != 0: + failed = True + for name in plan["volumes"]: + print(f"devrun gc: {verb} volume {name}") + # Docker refuses to remove a volume that any container uses. + if not ns.dry_run and run(["docker", "volume", "rm", name], stdout=subprocess.DEVNULL).returncode != 0: + failed = True + if not ns.dry_run: + # Forget dead runs that have nothing left. + left_containers, left_volumes = docker_inventory(ns.project) + left_ids = {(r.get("labels") or {}).get(RUN_ID_LABEL) for r in left_containers + left_volumes} + for entry in (directory / "runs").glob("*.lock"): + rid = entry.name[:-len(".lock")] + if RUN_ID_RE.fullmatch(rid) and rid not in left_ids and run_liveness(directory, rid) == "dead": + facts = {} + try: + facts = json_loads(entry.read_text()) + except (OSError, ValueError): + pass + if facts.get("project") == ns.project: + entry.unlink() + if not plan["containers"] and not plan["volumes"]: + print(f"devrun gc: nothing to remove for {ns.project}") + return 1 if failed else 0 + + # --------------------------------------------------------------------------- # # Orchestration # --------------------------------------------------------------------------- # +def _max_runs_default() -> int: + raw = os.environ.get(MAX_RUNS_ENV) + if raw is None or raw == "": + return DEFAULT_MAX_RUNS + if not raw.isdigit(): + raise DevrunError(f"{MAX_RUNS_ENV} must be a whole number, not {raw!r}") + return int(raw) + + def parse_args(argv: List[str]) -> argparse.Namespace: if "--" not in argv: raise DevrunError("missing '--' before COMMAND") @@ -490,18 +1064,29 @@ def parse_args(argv: List[str]) -> argparse.Namespace: img.add_argument("--image") ap.add_argument("--fetch", action="store_true", help="run cargo fetch --locked with network first") ap.add_argument("--network", action="store_true", help="give the command itself network access") - ap.add_argument("--evidence", action="store_true", help="use an emptied evidence target volume") + ap.add_argument("--evidence", action="store_true", help="build into this run's own empty target volume") ap.add_argument("--export", action="append", default=[], metavar="PATH") ap.add_argument("--export-patch", action="store_true") ap.add_argument("--out", metavar="DIR") ap.add_argument("--timeout", type=int, default=1800) ap.add_argument("--cpus", default="4") ap.add_argument("--memory", default="8g") + ap.add_argument("--max-runs", type=int, default=None, metavar="N", + help=f"at most N command phases at once (default ${MAX_RUNS_ENV} or {DEFAULT_MAX_RUNS})") + ap.add_argument("--exclusive", action="store_true", help="run the command phase alone on the machine (benchmarks)") + ap.add_argument("--wait", type=int, default=DEFAULT_WAIT, metavar="SECONDS", + help="give up after waiting this long for a slot or a lock") ns = ap.parse_args(head) if not PROJECT_RE.fullmatch(ns.project): raise DevrunError(f"invalid project name {ns.project!r} (lowercase letters, digits, hyphens)") if ns.timeout <= 0 or ns.timeout > 7200: raise DevrunError("--timeout must be between 1 and 7200 seconds") + if ns.wait < 0 or ns.wait > 86400: + raise DevrunError("--wait must be between 0 and 86400 seconds") + if ns.max_runs is None: + ns.max_runs = _max_runs_default() + if not 1 <= ns.max_runs <= MAX_RUNS_LIMIT: + raise DevrunError(f"the run cap (--max-runs or {MAX_RUNS_ENV}) must be between 1 and {MAX_RUNS_LIMIT}") for e in ns.export: _safe_member_path(e) ns.command = command @@ -548,21 +1133,34 @@ def copy_in(docker: "Docker", target_volume: str, tar_path: Path, dest: str, env def main_inner(ns: argparse.Namespace) -> int: repo = Path(ns.repo).resolve() - top = Path(git(repo, "rev-parse", "--show-toplevel")) + top = Path(os.path.realpath(git(repo, "rev-parse", "--show-toplevel"))) + key = source_key(str(top)) out = fresh_out_dir(ns.out) - work = Path(tempfile.mkdtemp(prefix="devrun-src-")) image = ns.image or f"flyingrobots/rust-dev:{ns.rust}" - docker = Docker(ns.project, image, ns.cpus, ns.memory, ns.network) - target_volume = f"{ns.project}-target-evidence" if ns.evidence else f"{ns.project}-target" - stop = {"flag": False} + run_id = new_run_id() + docker = Docker(ns.project, run_id, image, ns.cpus, ns.memory) + # CI machines run one job at a time and have no lock store: no slots and no registration there. + coordinated = os.environ.get("CI") != "true" + directory = state_dir() + gate = Gate(directory, ns.max_runs) + record = RunRecord(directory, run_id, {"project": ns.project, "source_key": key, "pid": str(os.getpid())}) + lease = Lease(f"devrun-{ns.project}-{run_id}", f"devrun-{run_id}", f"host/docker/{docker.name}/", + 2 * ns.timeout + 2 * ns.wait + 1500, f"devrun {ns.project} run {run_id}") + work = Path(tempfile.mkdtemp(prefix="devrun-src-")) + print(f"devrun: run {run_id}, container {docker.name}", file=sys.stderr, flush=True) def on_signal(signum, _frame): - docker.remove() - raise DevrunError(f"interrupted by signal {signum}; container removed") + # Cleanup below must finish: a second signal is ignored rather than cutting it short. + signal.signal(signal.SIGTERM, signal.SIG_IGN) + signal.signal(signal.SIGINT, signal.SIG_IGN) + raise DevrunError(f"interrupted by signal {signum}; removing this run's container and volumes") signal.signal(signal.SIGTERM, on_signal) signal.signal(signal.SIGINT, on_signal) + clean = False try: + if coordinated: + lease.claim(gate.waiter("host/docker/ in the lock store (daemon maintenance?)", ns.wait, poll=2.0)) if ns.history: facts = build_history(top, ns.history, work) tar_path, mode = work / "history.tar", "history" @@ -572,12 +1170,16 @@ def main_inner(ns: argparse.Namespace) -> int: facts = build_source_tar(top, mode, ns.commit, tar_path) print(f"devrun: source {mode} {facts.get('sha') or '(unpinned worktree)'} {facts['tar_sha256']}", file=sys.stderr) if docker.exists(): - raise DevrunError(f"refused: container {docker.name} already exists (another run, or litter)") + raise DevrunError(f"refused: container {docker.name} already exists (run id collision, or litter)") docker.ensure_volume(f"{ns.project}-cargo-registry", "cargo-registry") docker.ensure_volume(f"{ns.project}-cargo-git", "cargo-git") - docker.ensure_volume(target_volume, "target-evidence" if ns.evidence else "target") - docker.ensure_volume(f"{ns.project}-work", "work") - docker.ensure_volume(f"{ns.project}-scratch", "scratch") + if ns.evidence: + target_volume = docker.create_run_volume("target-evidence") + else: + target_volume = target_volume_name(ns.project, key) + docker.ensure_volume(target_volume, "target", {SOURCE_PATH_LABEL: str(top), SOURCE_KEY_LABEL: key}) + docker.create_run_volume("work") + docker.create_run_volume("scratch") env = {"DEVRUN_HEAD": facts.get("sha") or "", "DEVRUN_BASE": facts.get("base") or ""} dest = "/scratch/in" if mode == "history" else "/work" if ns.fetch: @@ -589,7 +1191,15 @@ def main_inner(ns: argparse.Namespace) -> int: prepare(docker, target_volume, ns.evidence) copy_in(docker, target_volume, fetch_tar, "/work", env) docker.create(FETCH_COMMAND, True, target_volume, fetch_environment(env, ns.rust), caches_writable=True) - if docker.start(ns.timeout) != 0: + # Cargo's own package-cache lock is $CARGO_HOME/.package-cache, on the read-only root: + # cargo warns that it cannot open it and fetches without any lock. Two fetches into the + # project's shared caches would then write the same files at once, so fetch phases of + # one project take turns on a host file lock (docs/hygiene.md section 10). + fetch_lock = (gate.hold(f"fetch-{ns.project}.lock", f"the {ns.project} fetch lock (another run is fetching)", ns.wait) + if coordinated else Admission([])) + with fetch_lock: + fetched = docker.start(ns.timeout) + if fetched != 0: raise DevrunError("cargo fetch --locked failed") docker.remove() # Phase 2 starts from empty volumes and a fresh copy, whatever phase 1 left behind. @@ -598,8 +1208,11 @@ def main_inner(ns: argparse.Namespace) -> int: # Phase 2: the command, offline unless --network. cmd = ["bash", "-c", INNER_SCRIPT, "run", mode, "1" if ns.export_patch else "0", *ns.command] env["CARGO_NET_OFFLINE"] = "false" if ns.network else "true" + env["DEVRUN_FRESH_MTIMES"] = "1" if mode == "commit" else "0" docker.create(cmd, ns.network, target_volume, env) - status = docker.start(ns.timeout) + admission = gate.admit(ns.exclusive, ns.wait) if coordinated else Admission([]) + with admission: + status = docker.start(ns.timeout) exported: List[str] = [] for path in ns.export: exported += docker.export(path, out) @@ -611,7 +1224,11 @@ def main_inner(ns: argparse.Namespace) -> int: print(f"devrun: exit {status}; exports in {out}: {', '.join(exported) or 'none'}", file=sys.stderr) return status finally: - docker.remove() + clean = docker.cleanup() + lease.release() + record.finish(clean) + if not clean: + print(f"devrun: run {run_id} left resources behind; remove them with: devrun --gc {ns.project}", file=sys.stderr) for p in work.glob("*"): p.unlink() work.rmdir() @@ -619,16 +1236,9 @@ def main_inner(ns: argparse.Namespace) -> int: def main(argv: List[str]) -> int: try: - ns = parse_args(argv) - if os.environ.get("CI") == "true" or os.environ.get(LOCK_ENV): - return main_inner(ns) - job = f"devrun-{ns.project}-{uuid.uuid4()}" - lock = ["git", "locks", "with", "--job", job, "--holder", f"devrun-{os.getpid()}", - "--ttl", str(ns.timeout + 900), "--wait", "600", "--note", f"devrun {ns.project}", - "host/heavy-work", f"host/docker/{ns.project}-dev-run/", "--", - sys.executable, os.path.abspath(__file__), *argv] - env = dict(os.environ, GIT_LOCKS_STORE=os.environ.get("GIT_LOCKS_STORE", LOCK_STORE), **{LOCK_ENV: job}) - return subprocess.call(lock, env=env) + if argv[:1] == ["--gc"]: + return gc_main(argv[1:]) + return main_inner(parse_args(argv)) except DevrunError as exc: print(f"devrun: {exc}", file=sys.stderr) return 2 diff --git a/docs/hygiene.md b/docs/hygiene.md index 74c43be..df5fdb0 100644 --- a/docs/hygiene.md +++ b/docs/hygiene.md @@ -5,11 +5,12 @@ is hostile. The host, its git repositories, its credentials, and other projects' Docker resources must stay unreachable and unchanged. The runner (`bin/devrun`) enforces every rule here; a rule that only lives in this file is not enforced. -**Status:** `bin/devrun` is not merged yet (flyingrobots/devcontainers#3). Until -it is, the rules about runs (copy-in, network phases, run-phase mounts, prep, -outputs, audit) are its specification, and nothing enforces them. -`scripts/build-image` and `tests/smoke-image` enforce the rules that apply to an -image build and a smoke test. +**Status:** `bin/devrun` enforces the rules about runs (copy-in, network +phases, run-phase mounts, prep, outputs, names, concurrency). The litter audit, +the receipts, and the budget checks (sections 8 and 9) are not implemented yet +(flyingrobots/devcontainers#5): until they are, those sections are a +specification. `scripts/build-image` and `tests/smoke-image` enforce the rules +that apply to an image build and a smoke test. ## 1. Source isolation @@ -58,10 +59,12 @@ image build and a smoke test. `/dev/mqueue`. - **No writable directory above the copy.** Cargo reads `.cargo/config.toml` in the working directory and in every parent directory. The copy is the root of - its own volume, so its only parent is the read-only root. Before each copy-in, - the runner empties the work and scratch volumes completely, hidden entries - included, so a `.cargo/config.toml` that one run leaves behind never reaches - the next. + its own volume, so its only parent is the read-only root. Each run creates its + own work and scratch volumes, new and empty, and removes them at the end, so a + `.cargo/config.toml` that one run leaves behind never reaches another run. + Before each copy-in (the fetch copy, then the command copy) the runner also + empties both volumes completely, hidden entries included, so nothing the fetch + phase leaves reaches the command phase. - `CARGO_HOME` is the image path `/usr/local/cargo`. It is read-only only because the run uses `--read-only`: the base image leaves it and the toolchain world-writable. Under `--read-only`, a run cannot plant a cargo `config.toml` @@ -116,27 +119,68 @@ image build and a smoke test. ## 5. Names, labels, and volumes -Every resource has a fixed name and the labels `org.flyingrobots.owner=` -and `org.flyingrobots.role=`. Cleanup selects by these labels only. - -| Resource | Name | Role label | -| --- | --- | --- | -| Base image | `flyingrobots/rust-dev:` | `base-image` | -| Upstream base, the build input | `rust@sha256:` from `images/rust-dev/base-digests`, pulled by digest. It carries only upstream labels. Keep it while its digest line exists; after removing the line, remove it with `docker image rm`. | none (upstream image) | -| Run container | `-dev-run` (one at a time, under a lock) | `dev-run` | -| Cargo registry cache, at `/usr/local/cargo/registry` | `-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` | -| Cargo git cache, at `/usr/local/cargo/git` | `-cargo-git` | `cargo-git` | -| Build output, iteration | `-target` | `target` | -| Build output, evidence | `-target-evidence`, emptied before each merge-gate run | `target-evidence` | -| The copy, at `/work` | `-work`, emptied before each copy-in | `work` | -| Inputs, logs, test data, exports, at `/scratch` | `-scratch`, emptied before each copy-in | `scratch` | +Every resource has a name built from the project (and, for one run's own +resources, the run id) and the labels `org.flyingrobots.owner=` and +`org.flyingrobots.role=`. A run's own container and volumes also carry +`org.flyingrobots.run-id=`, so cleanup can prove which run created +them. Cleanup selects by these labels only, and a name must match its labels. + +A run id is 12 hexadecimal digits from a random UUID, new for every run. A +source key is the first 16 hexadecimal digits of the SHA-256 of the real path +of the repository's top-level directory (`git rev-parse --show-toplevel`, +symlinks resolved), so each git worktree has its own key. + +| Resource | Name | Role label | Lifetime | +| --- | --- | --- | --- | +| Base image | `flyingrobots/rust-dev:` | `base-image` | Until a new toolchain replaces it | +| Upstream base, the build input | `rust@sha256:` from `images/rust-dev/base-digests`, pulled by digest. It carries only upstream labels. Keep it while its digest line exists; after removing the line, remove it with `docker image rm`. | none (upstream image) | As left | +| Run container | `-dev-run-`, also labelled with the run id | `dev-run` | One run; removed by the run | +| Cargo registry cache, at `/usr/local/cargo/registry` | `-cargo-registry` (per project: no cross-project cache poisoning) | `cargo-registry` | Kept | +| Cargo git cache, at `/usr/local/cargo/git` | `-cargo-git` | `cargo-git` | Kept | +| Build output, iteration | `-target-`, also labelled `org.flyingrobots.source-path=` and `org.flyingrobots.source-key=` (one per worktree) | `target` | Kept; `devrun --gc` removes it once the worktree path no longer exists | +| Build output, evidence | `-target-evidence-`, new and empty for each `--evidence` run, also labelled with the run id | `target-evidence` | One run; removed by the run | +| The copy, at `/work` | `-work-`, also labelled with the run id | `work` | One run; removed by the run | +| Inputs, logs, test data, exports, at `/scratch` | `-scratch-`, also labelled with the run id | `scratch` | One run; removed by the run | - No anonymous volumes. Images must not declare `VOLUME`. - The cargo caches are writable only in the fetch phase, which runs no code that the repository chooses (section 3). The build phase mounts them read-only, so a hostile build script cannot poison the cache for a later run. - **Evidence runs use a fresh target.** A merge-gate run never reuses build - output, so a stale cache cannot fabricate a passing result. + output, so a stale cache cannot fabricate a passing result. The evidence target + belongs to the one run, so concurrent evidence runs never share one. +- A run refuses a shared volume (a cache or a target) that exists with other + labels: another owner, another role, or, for a target, another source path or + key. It refuses a per-run volume or a container that already exists under its + name. +- Fixed-name volumes from earlier versions of the runner (`-work`, + `-scratch`, `-target`, `-target-evidence`) are no + longer used. They have no run id or source label, so `devrun --gc` never + removes them; remove them by hand with `docker volume rm` once no older runner + uses them. + +### What concurrent runs share + +Runs of different projects share nothing but the image. Concurrent runs of one +project share only these, and each sharing has a reason: + +- **The cargo caches**, read-only for every command phase. Only a fetch phase + writes them, and fetch phases of one project take turns (section 10), so + there is never more than one writer. A command phase can read the caches while + another run's fetch phase adds to them. Cargo writes a crate's sources into a + new directory and marks it complete last; a reader that needs a crate it did + not fetch itself finds it missing or incomplete and fails, because it cannot + write. This is cargo's design and is not tested here. The writer still runs no + repository-chosen code, so the poisoning rule above holds as before; what is + new is only that a read and a write can happen at the same moment. +- **The iteration target of one worktree**, read-write, when two runs come from + the same worktree at once. Cargo's build-directory lock (`.cargo-lock` in the + target, a file lock that works across containers on one Docker volume) + serializes cargo's own builds in it; cargo prints "Blocking waiting for file + lock on build directory". A command that writes `/target` without cargo is not + serialized. One run's build scripts can change what the other run builds from, + which was already true of consecutive runs: the iteration target is never + evidence. Runs from different worktrees never share a target. ## 6. Outputs @@ -149,15 +193,30 @@ and `org.flyingrobots.role=`. Cleanup selects by these labels only. ## 7. Lifecycle -- Each run is `docker create`, copy in, `docker start -a`, export, `docker rm`. -- A cleanup trap removes the container on success, failure, timeout, and - signals. +- Each run creates its volumes, then for each phase does `docker create`, copy + in, `docker start -a`, export, `docker rm`, and finally removes its volumes. +- Cleanup removes the container and the per-run volumes on success, failure, + timeout, and SIGTERM or SIGINT. After the first signal, further signals are + ignored, so they cannot cut cleanup short. Cleanup removes only what this run + created; the run's own records say what that is. +- A run that is killed outright (SIGKILL, a host crash) cannot clean up. Its + resources stay, labelled with its run id, and `devrun --gc ` removes + them (section 10). Its container may keep running until its command ends, + because the timeout lives in the killed process; `--gc` leaves a running + container alone and says so. - The command runs under a timeout that is shorter than the `git-locks` lease. ## 8. Budgets and free space - Build output: at most 20 GiB per project. Logs and test data: at most 4 GiB, including at most 128 MiB of logs. +- The build output of a project is the sum over its iteration targets, one per + worktree, plus the evidence targets of the runs in progress. Each target can + reach the size of a full debug build of the workspace, so a handful of + worktrees, or a few concurrent evidence runs, can exceed 20 GiB, which a + single per-project target could not. Until the measurement exists, remove the + targets of deleted worktrees with `devrun --gc ` and keep the number + of worktrees that build in mind. - The runner measures the volumes after each run and fails above a budget. Named volumes do not enforce size, so the measurement is the guard. - The run does not start when the host or the Docker VM has less than 50 GiB @@ -167,19 +226,95 @@ and `org.flyingrobots.role=`. Cleanup selects by these labels only. - Before and after each run, the runner lists containers, volumes, networks, and images. The run fails when Docker has any new resource that is not named and - labeled per section 5, or when the run container still exists. + labeled per section 5, or when the run container still exists. Concurrent + runs of the same project create resources while the audit runs, so the audit + must attribute each new resource by its run id label, not by its project + alone. - Every run writes a receipt: image digest, copy mode, source SHA, tarball SHA-256, command, network mode, limits, exit code, duration, audit result, and volume sizes. -## 10. Locks +## 10. Concurrency + +Runs do not take a machine-wide lock. Several runs, including several of the +same project and of the same worktree, run at the same time; section 5 says +what they share. These rules bound how many run and give benchmarks the machine +to themselves. -- Each run claims `host/heavy-work` and `host/docker/-dev-run/` in the - workstation `git-locks` store, with a lease longer than the timeout. The - second key follows the workstation registry rule - `host/docker//`. +- **Coordination is by file locks** (`flock`) in a per-user state directory: + `$DEVRUN_STATE_DIR`, else `$XDG_STATE_HOME/devrun`, else + `~/.local/state/devrun`, created with mode 0700 and refused when it belongs + to another user or others can write it. The kernel drops a file lock when its + process exits, however it exits, so a crash never leaves a stale lock. These + locks coordinate runs of the same user on one machine; they are not a security + boundary. No container can reach the state directory. +- **A cap on command phases.** At most N command phases run at once (default + 4; `--max-runs N` or `DEVRUN_MAX_RUNS`, 1 to 64). A run holds one of the slot + files `slot-0` to `slot-` only while its command runs; preparation, copy- + in, the fetch phase, and exports do not take a slot. A run admitted with a + larger N can use slots that a run with a smaller N never looks at, so the cap + in force is the one each run was given. A waiting run prints one line when it + starts waiting and one every 30 seconds with the time waited so far, and gives + up after `--wait` seconds (default 3600), like any other wait in the runner. + The default suits the workstation's Docker VM (18 CPUs, about 16 GiB of + memory): 4 runs at the default `--cpus 4` use at most 16 CPUs. Memory is + overcommitted: 4 runs at the default `--memory 8g` could ask for 32 GiB, so + runs that all peak at once can still exhaust the VM. Lower the cap or the + memory limit for memory-heavy work. +- **Exclusive runs for timing.** `--exclusive` waits until no other command + phase runs and keeps every other command phase out until it finishes. + **Performance measurements must use `--exclusive`**; a timing taken next to + other builds is not a measurement. Fetch phases and the preparation of other + runs can still happen during an exclusive run; they are short and mostly + input and output, not compilation. Once an exclusive run waits, new ordinary + runs queue behind it, so a stream of ordinary runs cannot starve it: an + ordinary run is admitted only by passing the `turnstile` file for an instant, + and a waiting exclusive run holds the turnstile until it finishes. Ordinary + runs hold `runs.lock` shared and an exclusive run holds it exclusively, so the + exclusive run starts only after every admitted ordinary run has finished. +- **Fetch phases of a project take turns.** Cargo's package-cache lock is the + file `$CARGO_HOME/.package-cache`, and `CARGO_HOME` is on the read-only root: + cargo cannot open it, warns, and fetches without a lock. Two concurrent fetches + would write the same cache files, so the fetch phase holds the per-project + file lock `fetch-.lock` while it runs. +- **Run liveness.** Each run locks `runs/.lock` before it creates any + Docker resource and keeps it locked until all of them are gone. The file also + records the project and the source key. +- **`devrun --gc `** removes, after printing each item (with + `--dry-run`, only prints): + - the stopped containers and the per-run volumes of dead runs. A run is dead + only when `--gc` can lock its liveness file itself, which proves that the + process has exited. Process ids can be reused and an age proves nothing + about use, so neither is used; `--gc` runs only when someone asks for it. + A run whose liveness file is missing (another user or another state + directory) is of unknown state and is left alone; + - iteration targets whose recorded worktree path no longer exists, when the + name, the source key, and the path labels agree and no live run uses that + source key. + It never removes a resource without this project's owner label or without a + valid run id or source label, a resource whose name does not match its + labels, a running container, a volume attached to any container that it is not + removing in the same pass, or the cargo caches. Docker itself refuses to + remove a running container or a volume in use, which backs these rules. +- **git-locks, for visibility.** Each run registers its container under + `host/docker/-dev-run-/` in the workstation `git-locks` store, + with a lease longer than its timeouts and waits, and releases it at the end. + The key is unique, so runs never conflict over it; it keeps the workstation + registry rule `host/docker//` true, and it keeps + daemon-wide maintenance, which claims `host/docker/`, from starting while runs + are active. A refusal (the key is covered by such a claim) makes the run wait. + A missing `git-locks`, a broken store, or a failed call prints one line and + the run goes on unregistered. +- **CI.** With `CI=true` (a single-use CI machine) there are no slots, no + exclusive gate, no fetch lock, and no registration. The liveness file is still + written, so `--gc` works there too. +- `scripts/build-image` and `tests/smoke-image` still claim `host/heavy-work` + and `host/docker/devcontainers-dev-run/`. `bin/devrun` no longer claims + either, so an image build and devrun command phases can run at the same time. ## 11. Not ours Resources without the `org.flyingrobots.owner` label belong to other projects. -The tools never stop, remove, or prune them. +The tools never stop, remove, or prune them. Resources with the owner label but +without a run id or source label are not provably leftovers; `devrun --gc` never +removes them either. diff --git a/tests/integration-devrun b/tests/integration-devrun index 5ace57b..c956500 100755 --- a/tests/integration-devrun +++ b/tests/integration-devrun @@ -1,29 +1,66 @@ #!/usr/bin/env bash # Real-Docker integration test for bin/devrun. Needs flyingrobots/rust-dev:1.96.0 (scripts/build-image 1.96.0). -# Uses project name devrun-selftest and removes only that project's labeled volumes at the end. +# Uses a throwaway project name (devrun-selftest-), so it never meets another run's resources, +# and at the end removes only containers and volumes labelled with that project. +# The sections about concurrency use a private devrun state directory, so that they control the +# slots themselves; their containers are small (1 CPU, 1 GiB). The other sections use the shared +# state directory and count against the machine's run cap like any other run. set -euo pipefail root="$(cd "$(dirname "$0")/.." && pwd)" devrun="$root/bin/devrun" -project="devrun-selftest" +project="devrun-selftest-$(od -An -N4 -tx1 /dev/urandom | tr -d ' \n')" +owner="label=org.flyingrobots.owner=$project" fail() { echo "integration-devrun: FAIL: $*" >&2; exit 1; } -# A container with the run's name belongs to someone else: refuse to start rather than touch it. -if docker container inspect "$project-dev-run" >/dev/null 2>&1; then - fail "container $project-dev-run already exists; remove it or wait for its owner" -fi work="$(mktemp -d -t devrun-selftest-XXXXXX)" -planted=0 # 1 while the decoy container created by this script exists +private="$work/state" +small=(--cpus 1 --memory 1g --timeout 300) +pids=() # background devrun processes started by this script +foreign_decoy=0 # 1 while the decoy volume with another owner label, created by this script, exists cleanup() { + local status=$? pid + for pid in ${pids[@]+"${pids[@]}"}; do kill -TERM "$pid" 2>/dev/null || true; done + for pid in ${pids[@]+"${pids[@]}"}; do wait "$pid" 2>/dev/null || true; done + # Every resource with this owner label was created by this script or by the runs it started. + docker ps -aq --filter "$owner" | while read -r id; do docker rm -f "$id" >/dev/null || true; done + docker volume ls -q --filter "$owner" | while read -r v; do docker volume rm "$v" >/dev/null || true; done + if [[ "$foreign_decoy" == 1 ]]; then docker volume rm "$project-work-000000000000" >/dev/null 2>&1 || true; fi rm -rf "$work" - # devrun removes its own containers; this script removes only the decoy it created. - if [[ "$planted" == 1 ]]; then docker rm -f "$project-dev-run" >/dev/null 2>&1 || true; fi - for v in cargo-registry cargo-git target target-evidence work scratch; do - if [[ "$(docker volume inspect "$project-$v" --format '{{index .Labels "org.flyingrobots.owner"}}' 2>/dev/null || true)" == "$project" ]]; then - docker volume rm "$project-$v" >/dev/null - fi - done + exit "$status" } trap cleanup EXIT +# prun: devrun with the private state directory, coordinated even in CI (CI=true turns slots off). +# spawn: the same, for `spawn ... &`: exec makes $! the pid of devrun itself, so signals reach it. +prun() { env -u CI DEVRUN_STATE_DIR="$private" "$devrun" "$@"; } +spawn() { exec env -u CI DEVRUN_STATE_DIR="$private" "$devrun" "$@"; } +now() { python3 -c 'import time; print(f"{time.time():.3f}")'; } +elapsed() { awk -v a="$1" -v b="$2" 'BEGIN { printf "%.1f", b - a }'; } +before() { awk -v a="$1" -v b="$2" 'BEGIN { exit !(a + 0 <= b + 0) }'; } +field() { sed -n "s/^$1 //p" "$2" | head -1; } +run_id_of() { sed -n 's/^devrun: run \([0-9a-f]*\),.*/\1/p' "$1" | head -1; } +# wait_for SECONDS WHAT COMMAND...: poll COMMAND until it succeeds, failing after SECONDS. +wait_for() { + local limit=$1 what=$2; shift 2 + local end=$((SECONDS + limit)) + until "$@"; do + (( SECONDS < end )) || fail "timed out after ${limit}s waiting for $what" + sleep 0.2 + done +} +has_line() { grep -q "$1" "$2" 2>/dev/null; } +leftovers_of() { # resources of one run that still exist + docker ps -aq --filter "$owner" --filter "label=org.flyingrobots.run-id=$1" + docker volume ls -q --filter "$owner" --filter "label=org.flyingrobots.run-id=$1" +} +no_run_leftovers() { # no container and no per-run volume of this project is left + [[ -z "$(docker ps -aq --filter "$owner")" ]] || fail "a container is left: $(docker ps -a --filter "$owner" --format '{{.Names}}')" + [[ -z "$(docker volume ls -q --filter "$owner" --filter label=org.flyingrobots.run-id)" ]] || + fail "a per-run volume is left: $(docker volume ls -q --filter "$owner" --filter label=org.flyingrobots.run-id)" +} +# The timed command: prints its start and end time from inside the container, and its marker file. +# shellcheck disable=SC2016 # $(date ...) and $1 must expand inside the container. +timed='echo "start $(date +%s.%N)"; cat marker.txt; sleep "$1"; echo "end $(date +%s.%N)"' + repo="$work/repo"; mkdir -p "$repo/src" git -C "$repo" init -q -b main printf 'hello\n' > "$repo/README" @@ -65,7 +102,7 @@ out1="$work/out1" "$devrun" "$project" "$repo" --commit "$head" --timeout 300 --export result.txt --export-patch --out "$out1" -- bash -c "$probe" [[ "$(cat "$out1/result.txt")" == "ok" ]] || fail "export result.txt" grep -q '^+changed$' "$out1/changes.patch" || fail "patch does not contain the change" -docker ps -a --format '{{.Names}}' | grep -qx "$project-dev-run" && fail "container still exists" +no_run_leftovers echo "== worktree mode: untracked file included, ignored file excluded" printf 'new\n' > "$repo/untracked.txt" @@ -81,6 +118,13 @@ out3="$work/out3" bash -c "test \"\$(git rev-parse HEAD)\" = $head && test \"\$(git rev-parse origin/main)\" = $base && git log -1 --format=%B | grep -q 'Repo-Respect-Receipt: none' && echo ok > /scratch/out/result.txt" [[ "$(cat "$out3/result.txt")" == "ok" ]] || fail "history mode" +echo "== the iteration target is keyed by the worktree path and labelled with it" +real_repo="$(cd "$repo" && pwd -P)" +target_name="$(docker volume ls -q --filter "$owner" --filter label=org.flyingrobots.role=target)" +[[ "$(wc -l <<<"$target_name" | tr -d ' ')" == 1 ]] || fail "expected one target volume, found: $target_name" +[[ "$(docker volume inspect "$target_name" --format '{{index .Labels "org.flyingrobots.source-path"}}')" == "$real_repo" ]] || + fail "the target volume does not record the worktree path" + echo "== fetch phase: a planted cargo config runs no code while the caches are writable" crate="$work/crate"; mkdir -p "$crate/src" "$crate/.cargo" printf '[package]\nname = "selftest"\nversion = "0.1.0"\nedition = "2021"\n' > "$crate/Cargo.toml" @@ -102,18 +146,133 @@ if ! "$devrun" "$project" "$crate" --commit HEAD --fetch --timeout 300 --export fi [[ "$(cat "$out4/result.txt")" == "ok" ]] || fail "fetch phase ran the planted wrapper, or phase 2 lost the config files" -echo "== an existing container with the run's name is refused and left alone" -docker create --name "$project-dev-run" --label org.flyingrobots.owner=other flyingrobots/rust-dev:1.96.0 true >/dev/null -planted=1 -if "$devrun" "$project" "$repo" --commit "$base" --timeout 60 -- true 2>"$work/err-exists"; then fail "ran despite an existing container"; fi -grep -q 'already exists' "$work/err-exists" || fail "wrong refusal: $(cat "$work/err-exists")" -docker ps -a --format '{{.Names}}' | grep -qx "$project-dev-run" || fail "devrun removed a container it did not create" +echo "== commit mode after a newer build: an older commit date does not reuse stale output (issue #9)" +bin="$work/bin"; mkdir -p "$bin/src" +printf '[package]\nname = "stale"\nversion = "0.1.0"\nedition = "2021"\n' > "$bin/Cargo.toml" +printf 'version = 4\n\n[[package]]\nname = "stale"\nversion = "0.1.0"\n' > "$bin/Cargo.lock" +printf 'fn main() { println!("built-from-A"); }\n' > "$bin/src/main.rs" +git -C "$bin" init -q -b main && git -C "$bin" add -A && git -C "$bin" -c user.name=t -c user.email=t@invalid commit -qm A +commit_a="$(git -C "$bin" rev-parse HEAD)" +printf 'fn main() { println!("built-from-B"); }\n' > "$bin/src/main.rs" +# B's commit date is years older than the build of A, so git archive gives B's files old times. +GIT_COMMITTER_DATE="2001-01-01T00:00:00Z" GIT_AUTHOR_DATE="2001-01-01T00:00:00Z" \ + git -C "$bin" -c user.name=t -c user.email=t@invalid commit -qam B +commit_b="$(git -C "$bin" rev-parse HEAD)" +[[ "$("$devrun" "$project" "$bin" --commit "$commit_a" "${small[@]}" -- cargo run -q --locked 2>/dev/null)" == built-from-A ]] || fail "commit A did not build" +got="$("$devrun" "$project" "$bin" --commit "$commit_b" "${small[@]}" -- cargo run -q --locked 2>/dev/null)" +[[ "$got" == built-from-B ]] || fail "commit B ran the stale build of commit A: $got" + +echo "== two runs of the same project at once, from two worktrees of one repository" +git -C "$repo" add -A && git -C "$repo" -c user.name=t -c user.email=t@invalid commit -qm "worktree base" +git -C "$repo" worktree add -q "$work/wt2" +printf 'marker-one\n' > "$repo/marker.txt" +printf 'marker-two\n' > "$work/wt2/marker.txt" +t0="$(now)" +spawn "$project" "$repo" --worktree "${small[@]}" -- bash -c "$timed" timed 5 >"$work/c1.out" 2>"$work/c1.err" & p1=$! +spawn "$project" "$work/wt2" --worktree "${small[@]}" -- bash -c "$timed" timed 5 >"$work/c2.out" 2>"$work/c2.err" & p2=$! +pids=("$p1" "$p2") +wait "$p1" || fail "concurrent run 1 failed: $(cat "$work/c1.err")" +wait "$p2" || fail "concurrent run 2 failed: $(cat "$work/c2.err")" +pids=() +t1="$(now)" +if ! grep -qx marker-one "$work/c1.out" || grep -q marker-two "$work/c1.out"; then fail "run 1 did not see only its own source"; fi +if ! grep -qx marker-two "$work/c2.out" || grep -q marker-one "$work/c2.out"; then fail "run 2 did not see only its own source"; fi +if ! before "$(field start "$work/c2.out")" "$(field end "$work/c1.out")" || ! before "$(field start "$work/c1.out")" "$(field end "$work/c2.out")"; then + fail "the two runs did not overlap" +fi +[[ "$(run_id_of "$work/c1.err")" != "$(run_id_of "$work/c2.err")" ]] || fail "the runs share a run id" +no_run_leftovers +t2="$(now)" +prun "$project" "$repo" --worktree "${small[@]}" -- bash -c "$timed" timed 5 >/dev/null 2>"$work/s1.err" || fail "sequential run 1: $(cat "$work/s1.err")" +prun "$project" "$work/wt2" --worktree "${small[@]}" -- bash -c "$timed" timed 5 >/dev/null 2>"$work/s2.err" || fail "sequential run 2: $(cat "$work/s2.err")" +t3="$(now)" +echo "integration-devrun: two same-project runs (5 s commands): together $(elapsed "$t0" "$t1") s, back to back $(elapsed "$t2" "$t3") s" + +echo "== the cap: with --max-runs 1, two runs take turns" +spawn "$project" "$repo" --worktree "${small[@]}" --max-runs 1 -- bash -c "$timed" timed 3 >"$work/m1.out" 2>"$work/m1.err" & p1=$! +spawn "$project" "$work/wt2" --worktree "${small[@]}" --max-runs 1 -- bash -c "$timed" timed 3 >"$work/m2.out" 2>"$work/m2.err" & p2=$! +pids=("$p1" "$p2") +wait "$p1" || fail "capped run 1 failed: $(cat "$work/m1.err")" +wait "$p2" || fail "capped run 2 failed: $(cat "$work/m2.err")" +pids=() +if before "$(field start "$work/m1.out")" "$(field start "$work/m2.out")"; then first=m1 second=m2; else first=m2 second=m1; fi +before "$(field end "$work/$first.out")" "$(field start "$work/$second.out")" || fail "two runs overlapped under --max-runs 1" +grep -q 'waiting for a run slot' "$work/$second.err" || fail "the second run did not report that it was waiting" + +echo "== an exclusive run does not overlap ordinary runs, and is not starved by later ones" +spawn "$project" "$repo" --worktree "${small[@]}" -- bash -c "$timed" timed 6 >"$work/a.out" 2>"$work/a.err" & pa=$! +pids=("$pa") +wait_for 120 "the ordinary run's command to start" has_line '^start ' "$work/a.out" +spawn "$project" "$repo" --worktree "${small[@]}" --exclusive -- bash -c "$timed" timed 2 >"$work/e.out" 2>"$work/e.err" & pe=$! +pids+=("$pe") +wait_for 120 "the exclusive run to queue" has_line 'waiting for exclusive access' "$work/e.err" +spawn "$project" "$work/wt2" --worktree "${small[@]}" -- bash -c "$timed" timed 1 >"$work/b.out" 2>"$work/b.err" & pb=$! +pids+=("$pb") +wait "$pa" || fail "ordinary run A failed: $(cat "$work/a.err")" +wait "$pe" || fail "exclusive run failed: $(cat "$work/e.err")" +wait "$pb" || fail "ordinary run B failed: $(cat "$work/b.err")" +pids=() +before "$(field end "$work/a.out")" "$(field start "$work/e.out")" || fail "the exclusive run started before ordinary run A ended" +before "$(field end "$work/e.out")" "$(field start "$work/b.out")" || fail "ordinary run B, started after the exclusive run queued, ran before it ended" + +echo "== SIGTERM: the run removes its container and volumes" +spawn "$project" "$repo" --worktree "${small[@]}" -- bash -c "$timed" timed 120 >"$work/t.out" 2>"$work/t.err" & pt=$! +pids=("$pt") +wait_for 120 "the run's command to start" has_line '^start ' "$work/t.out" +rid="$(run_id_of "$work/t.err")" +[[ -n "$(leftovers_of "$rid")" ]] || fail "the run's resources were not found while it ran" +kill -TERM "$pt" +if wait "$pt"; then fail "a run killed with SIGTERM exited 0"; fi +pids=() +grep -q 'interrupted by signal' "$work/t.err" || fail "the run did not report the signal: $(cat "$work/t.err")" +[[ -z "$(leftovers_of "$rid")" ]] || fail "SIGTERM left resources of run $rid: $(leftovers_of "$rid")" + +echo "== SIGKILL: --gc keeps a dead run's running container, then removes everything once it stopped" +spawn "$project" "$repo" --worktree "${small[@]}" -- bash -c "$timed" timed 10 >"$work/k.out" 2>"$work/k.err" & pk=$! +pids=("$pk") +wait_for 120 "the run's command to start" has_line '^start ' "$work/k.out" +rid="$(run_id_of "$work/k.err")" +kill -KILL "$pk" +wait "$pk" 2>/dev/null || true +pids=() +env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc1.out" || fail "gc failed: $(cat "$work/gc1.out")" +grep -q "keep $project-dev-run-$rid: run $rid is dead but its container is still running" "$work/gc1.out" || + fail "gc did not keep the running container: $(cat "$work/gc1.out")" +[[ -n "$(docker ps -q --filter "label=org.flyingrobots.run-id=$rid")" ]] || fail "gc touched a running container" +stopped() { [[ -z "$(docker ps -q --filter "$owner" --filter "label=org.flyingrobots.run-id=$rid")" ]]; } +wait_for 60 "the orphaned container to exit" stopped +env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc2.out" || fail "gc failed: $(cat "$work/gc2.out")" +[[ -z "$(leftovers_of "$rid")" ]] || fail "gc left resources of dead run $rid: $(leftovers_of "$rid")" +[[ ! -e "$private/runs/$rid.lock" ]] || fail "gc did not forget the dead run" + +echo "== --gc removes the target of a deleted worktree and never touches resources it cannot prove are leftovers" +gone="$work/gone"; mkdir -p "$gone"; git -C "$gone" init -q -b main; printf 'x\n' > "$gone/x" +prun "$project" "$gone" --worktree "${small[@]}" -- true 2>/dev/null || fail "run in the soon-deleted worktree" +gone_key="$(python3 -c 'import hashlib, os, sys; print(hashlib.sha256(os.fsencode(os.path.realpath(sys.argv[1]))).hexdigest()[:16])' "$gone")" +gone_target="$project-target-$gone_key" +docker volume inspect "$gone_target" >/dev/null || fail "no target volume $gone_target" +rm -rf "$gone" +docker volume create --label "org.flyingrobots.owner=$project" --label org.flyingrobots.role=work "$project-work" >/dev/null +foreign_decoy=1 +docker volume create --label org.flyingrobots.owner=someone-else --label org.flyingrobots.role=work \ + --label org.flyingrobots.run-id=000000000000 "$project-work-000000000000" >/dev/null +docker create --name "$project-dev-run" --label "org.flyingrobots.owner=$project" --label org.flyingrobots.role=dev-run \ + flyingrobots/rust-dev:1.96.0 true >/dev/null +env DEVRUN_STATE_DIR="$private" "$devrun" --gc "$project" >"$work/gc3.out" || fail "gc failed: $(cat "$work/gc3.out")" +grep -qx "devrun gc: remove volume $gone_target" "$work/gc3.out" || fail "gc did not remove the deleted worktree's target: $(cat "$work/gc3.out")" +if docker volume inspect "$gone_target" >/dev/null 2>&1; then fail "the deleted worktree's target still exists"; fi +docker volume inspect "$target_name" >/dev/null || fail "gc removed the target of an existing worktree" +docker volume inspect "$project-work" >/dev/null || fail "gc removed a volume without a run id" +docker volume inspect "$project-work-000000000000" >/dev/null || fail "gc removed another owner's volume" +docker container inspect "$project-dev-run" >/dev/null || fail "gc removed a container without a run id" docker rm "$project-dev-run" >/dev/null -planted=0 +docker volume rm "$project-work" "$project-work-000000000000" >/dev/null +foreign_decoy=0 echo "== refusals" ln -s /etc/passwd "$repo/evil"; git -C "$repo" add evil; git -C "$repo" -c user.name=t -c user.email=t@invalid commit -qm "evil" if "$devrun" "$project" "$repo" --commit HEAD --timeout 60 -- true 2>"$work/err"; then fail "absolute symlink accepted"; fi grep -q 'absolute symlink' "$work/err" || fail "wrong refusal: $(cat "$work/err")" +no_run_leftovers echo "integration-devrun: PASS" diff --git a/tests/test_devrun.py b/tests/test_devrun.py index bb4c23f..37f9258 100644 --- a/tests/test_devrun.py +++ b/tests/test_devrun.py @@ -1,14 +1,24 @@ -"""Unit tests for bin/devrun: the host-side tarball checks and argument handling. +"""Unit tests for bin/devrun: the host-side tarball checks, argument handling, names and labels, +the run slots and the exclusive gate, run liveness, and --gc selection. Run: python3 -m unittest discover -s tests -These tests need no Docker. +These tests need no Docker. The lock tests use real flock on files in temporary directories. """ +import contextlib +import hashlib import importlib.machinery import importlib.util import io +import json +import os import pathlib +import re +import stat +import subprocess +import sys import tarfile import tempfile +import threading import unittest ROOT = pathlib.Path(__file__).resolve().parents[1] @@ -184,57 +194,620 @@ def test_fetch_environment_pins_the_compiler_and_clears_wrappers(self): self.assertEqual(env["DEVRUN_HEAD"], "abc") -class ContainerOwnership(unittest.TestCase): +class FakeDockerCalls(unittest.TestCase): + """Replaces devrun.run, the process boundary to the docker CLI, with a recorder.""" + def setUp(self): self.calls = [] + self.fail_volume_rm = set() + self.existing_volumes = {} self.original_run = devrun.run - devrun.run = lambda cmd, **kw: self.calls.append(cmd) or devrun.subprocess.CompletedProcess(cmd, 0, b"", b"") + + def fake(cmd, **kw): + self.calls.append(cmd) + if cmd[:3] == ["docker", "volume", "inspect"]: + labels = self.existing_volumes.get(cmd[3]) + if labels is None: + return devrun.subprocess.CompletedProcess(cmd, 1, b"", b"no such volume") + return devrun.subprocess.CompletedProcess(cmd, 0, json.dumps(labels).encode(), b"") + if cmd[:3] == ["docker", "volume", "create"]: + labels = dict(a.split("=", 1) for a in cmd[4:-1:2]) + self.existing_volumes.setdefault(cmd[-1], labels) + if cmd[:3] == ["docker", "volume", "rm"] and cmd[3] in self.fail_volume_rm: + return devrun.subprocess.CompletedProcess(cmd, 1, b"", b"volume is in use") + return devrun.subprocess.CompletedProcess(cmd, 0, b"", b"") + + devrun.run = fake def tearDown(self): devrun.run = self.original_run + def docker(self): + return devrun.Docker("geordi", RUN_ID, "img", "1", "1g") + + +RUN_ID = "0123456789ab" +DOCKER_NAME = re.compile(r"[a-zA-Z0-9][a-zA-Z0-9_.-]+") + + +class ResourceNames(unittest.TestCase): + def test_every_run_resource_is_named_after_the_project_and_the_run(self): + names = devrun.run_resource_names("geordi", RUN_ID) + self.assertEqual(names, {"container": "geordi-dev-run-0123456789ab", "work": "geordi-work-0123456789ab", + "scratch": "geordi-scratch-0123456789ab", + "target-evidence": "geordi-target-evidence-0123456789ab"}) + self.assertEqual(set(names) - {"container"}, set(devrun.PER_RUN_ROLES)) + for name in names.values(): + self.assertRegex(name, DOCKER_NAME) + + def test_run_ids_are_twelve_hex_digits_and_differ(self): + ids = {devrun.new_run_id() for _ in range(200)} + self.assertEqual(len(ids), 200) + for rid in ids: + self.assertRegex(rid, r"^[0-9a-f]{12}$") + + def test_source_key_is_a_stable_hash_of_the_path(self): + # The value is fixed: a change to the key would orphan every existing target volume. + self.assertEqual(devrun.source_key("/src/geordi"), hashlib.sha256(b"/src/geordi").hexdigest()[:16]) + self.assertEqual(devrun.source_key("/src/geordi"), devrun.source_key("/src/geordi")) + + def test_distinct_worktrees_get_distinct_targets_with_safe_names(self): + paths = ["/src/geordi", "/src/geordi-2", "/src/geordi/", "/src/a b", "/src/été", "/src/x;rm -rf"] + names = [devrun.target_volume_name("geordi", devrun.source_key(p)) for p in paths] + self.assertEqual(len(set(names)), len(paths)) + for name in names: + self.assertRegex(name, r"^geordi-target-[0-9a-f]{16}$") + + +class ContainerOwnership(FakeDockerCalls): def test_remove_skips_a_container_this_run_did_not_create(self): - docker = devrun.Docker("geordi", "img", "1", "1g", False) - docker.remove() + self.docker().remove() self.assertEqual(self.calls, []) def test_remove_runs_once_after_create(self): - docker = devrun.Docker("geordi", "img", "1", "1g", False) + docker = self.docker() docker.create(["true"], False, "geordi-target") docker.remove() docker.remove() - self.assertEqual([c for c in self.calls if c[:2] == ["docker", "rm"]], [["docker", "rm", "-f", "geordi-dev-run"]]) + self.assertEqual([c for c in self.calls if c[:2] == ["docker", "rm"]], [["docker", "rm", "-f", "geordi-dev-run-0123456789ab"]]) - def test_run_container_mounts_the_copy_at_work_with_home_set(self): - docker = devrun.Docker("geordi", "img", "1", "1g", False) - docker.create(["true"], False, "geordi-target") + def test_run_container_mounts_its_own_volumes_with_home_set(self): + docker = self.docker() + docker.create(["true"], False, "geordi-target-x") create = self.calls[-1] - self.assertIn("type=volume,src=geordi-work,dst=/work", create) + self.assertIn("type=volume,src=geordi-work-0123456789ab,dst=/work", create) + self.assertIn("type=volume,src=geordi-scratch-0123456789ab,dst=/scratch", create) + self.assertIn("type=volume,src=geordi-target-x,dst=/target", create) self.assertIn("HOME=/tmp/home", create) self.assertIn("TMPDIR=/exec-tmp", create) self.assertIn("/exec-tmp:rw,nosuid,nodev,exec,size=1g,mode=1777", create) self.assertEqual(create[create.index("--workdir") + 1], "/work") + self.assertEqual(create[create.index("--name") + 1], "geordi-dev-run-0123456789ab") + + def test_container_carries_owner_role_and_run_id_labels(self): + docker = self.docker() + docker.create(["true"], False, "geordi-target-x") + create = self.calls[-1] + labels = {create[i + 1] for i, a in enumerate(create) if a == "--label"} + self.assertEqual(labels, {"org.flyingrobots.owner=geordi", "org.flyingrobots.role=dev-run", + "org.flyingrobots.run-id=0123456789ab"}) + + def test_run_volumes_are_labelled_and_removed_by_cleanup(self): + docker = self.docker() + docker.create_run_volume("work") + docker.create_run_volume("scratch") + self.assertEqual(self.existing_volumes["geordi-work-0123456789ab"], + {"org.flyingrobots.owner": "geordi", "org.flyingrobots.role": "work", + "org.flyingrobots.run-id": "0123456789ab"}) + self.assertTrue(docker.cleanup()) + removed = [c[3] for c in self.calls if c[:3] == ["docker", "volume", "rm"]] + self.assertEqual(sorted(removed), ["geordi-scratch-0123456789ab", "geordi-work-0123456789ab"]) + + def test_cleanup_never_removes_shared_volumes(self): + docker = self.docker() + docker.ensure_volume("geordi-cargo-registry", "cargo-registry") + docker.ensure_volume("geordi-target-k", "target", {devrun.SOURCE_PATH_LABEL: "/src/g", devrun.SOURCE_KEY_LABEL: "k"}) + self.assertTrue(docker.cleanup()) + self.assertEqual([c for c in self.calls if c[:3] == ["docker", "volume", "rm"]], []) + + def test_cleanup_reports_a_volume_it_could_not_remove(self): + docker = self.docker() + docker.create_run_volume("work") + self.fail_volume_rm.add("geordi-work-0123456789ab") + with contextlib.redirect_stderr(io.StringIO()): + self.assertFalse(docker.cleanup()) + + def test_an_existing_run_volume_is_refused(self): + self.existing_volumes["geordi-work-0123456789ab"] = {"org.flyingrobots.owner": "other"} + with self.assertRaises(devrun.DevrunError): + self.docker().create_run_volume("work") + self.assertEqual(self.docker().volumes, []) + + def test_a_target_of_another_source_tree_is_refused(self): + self.existing_volumes["geordi-target-k"] = {"org.flyingrobots.owner": "geordi", "org.flyingrobots.role": "target", + devrun.SOURCE_PATH_LABEL: "/src/other", devrun.SOURCE_KEY_LABEL: "k"} + with self.assertRaises(devrun.DevrunError): + self.docker().ensure_volume("geordi-target-k", "target", {devrun.SOURCE_PATH_LABEL: "/src/g", devrun.SOURCE_KEY_LABEL: "k"}) -class Locks(unittest.TestCase): - def test_lock_keys_follow_the_container_name(self): - captured = {} - original = devrun.subprocess.call - devrun.subprocess.call = lambda cmd, env=None: captured.setdefault("cmd", cmd) and 0 +# --------------------------------------------------------------------------- # +# Slots and the exclusive gate: real flock on files in a temporary directory. +# Each contender is a thread with its own Gate and its own open files, which flock treats like a +# separate process. Time is a fake clock that moves only when a waiter polls; threads synchronize +# on events and on counted poll attempts, never on sleeps. +# --------------------------------------------------------------------------- # + +LONG = 10 # seconds; the bound on every event wait, reached only when a test fails + + +class FakeClock: + def __init__(self): + self.now = 1000.0 + + def __call__(self): + return self.now + + def sleep(self, seconds): + self.now += seconds + threading.Event().wait(0.001) # yield the thread; the fake clock carries the time + + +class Contender: + """One run asking for admission in its own thread. `attempts` counts its failed polls.""" + + def __init__(self, directory, exclusive, max_runs=4): + self.clock = FakeClock() + self.attempts = 0 + self.changed = threading.Condition() + self.admitted = threading.Event() + self.let_go = threading.Event() + self.done = threading.Event() + self.error = None + self.gate = devrun.Gate(directory, max_runs, clock=self.clock, sleep=self.clock.sleep, + report=self._report, poll=0.5, report_every=0.5) + self.thread = threading.Thread(target=self._run, args=(exclusive,), daemon=True) + self.thread.start() + + def _report(self, _msg): + with self.changed: + self.attempts += 1 + self.changed.notify_all() + + def _run(self, exclusive): try: - env = dict(devrun.os.environ) - env.pop("CI", None) - env.pop(devrun.LOCK_ENV, None) - saved = dict(devrun.os.environ) - devrun.os.environ.clear() - devrun.os.environ.update(env) - devrun.main(["geordi", ".", "--commit", "HEAD", "--", "true"]) + with self.gate.admit(exclusive, None): + self.admitted.set() + self.let_go.wait(LONG) + except BaseException as exc: # reported to the test + self.error = exc finally: - devrun.subprocess.call = original - devrun.os.environ.clear() - devrun.os.environ.update(saved) - self.assertIn("host/heavy-work", captured["cmd"]) - self.assertIn("host/docker/geordi-dev-run/", captured["cmd"]) + self.done.set() + + def wait_attempts(self, n): + """Block until this contender has failed at least n more polls than it had so far.""" + with self.changed: + target = self.attempts + n + if not self.changed.wait_for(lambda: self.attempts >= target, LONG): + raise AssertionError("contender stopped polling") + + def release(self): + self.let_go.set() + if not self.done.wait(LONG): + raise AssertionError("contender did not finish") + if self.error: + raise self.error + + +class GateCase(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.dir = pathlib.Path(self.tmp.name) + self.contenders = [] + + def tearDown(self): + for c in self.contenders: + c.let_go.set() + c.done.wait(LONG) + self.tmp.cleanup() + + def contender(self, exclusive=False, max_runs=4): + c = Contender(self.dir, exclusive, max_runs) + self.contenders.append(c) + return c + + def admitted(self, c): + self.assertTrue(c.admitted.wait(LONG), "contender was not admitted") + self.assertIsNone(c.error) + + def try_once(self, exclusive=False, max_runs=1): + """Admission with a zero wait limit: succeeds only if admission is possible right now.""" + gate = devrun.Gate(self.dir, max_runs, clock=FakeClock(), sleep=lambda s: None, report=lambda m: None) + return gate.admit(exclusive, 0) + + +class Slots(GateCase): + def test_the_cap_admits_n_runs_and_the_next_waits_for_a_release(self): + a, b = self.contender(max_runs=2), self.contender(max_runs=2) + self.admitted(a) + self.admitted(b) + c = self.contender(max_runs=2) + c.wait_attempts(3) + self.assertFalse(c.admitted.is_set()) + a.release() + self.admitted(c) + + def test_a_slot_is_released_when_the_run_raises(self): + gate = devrun.Gate(self.dir, 1) + with self.assertRaises(ValueError): + with gate.admit(False, 0): + raise ValueError("command failed") + self.try_once().release() + + def test_the_cap_is_held_while_admitted(self): + with self.try_once(): + with self.assertRaises(devrun.DevrunError) as ctx: + self.try_once() + self.assertIn("gave up after waiting 0 s for a run slot", str(ctx.exception)) + + def test_a_waiting_run_reports_how_long_it_has_waited_and_gives_up_at_the_limit(self): + messages = [] + clock = FakeClock() + gate = devrun.Gate(self.dir, 1, clock=clock, sleep=lambda s: setattr(clock, "now", clock.now + s), + report=messages.append, poll=0.5, report_every=30) + with self.try_once(): + with self.assertRaises(devrun.DevrunError) as ctx: + gate.admit(False, 61) + self.assertEqual(len(messages), 3) + for msg, waited in zip(messages, (0, 30, 60)): + self.assertTrue(msg.startswith("devrun: waiting for a run slot"), msg) + self.assertIn(f"waited {waited} s so far", msg) + self.assertIn("gave up after waiting 61 s", str(ctx.exception)) + + def test_a_slot_held_by_a_killed_process_is_free_again(self): + child = subprocess.Popen([sys.executable, "-c", HOLD_SLOT, str(ROOT / "bin" / "devrun"), str(self.dir)], + stdout=subprocess.PIPE, stdin=subprocess.PIPE) + try: + self.assertEqual(child.stdout.readline(), b"admitted\n") + with self.assertRaises(devrun.DevrunError): + self.try_once() + finally: + child.kill() + child.wait() + child.stdin.close() + child.stdout.close() + self.try_once().release() + + def test_the_cap_is_validated(self): + for bad in (0, devrun.MAX_RUNS_LIMIT + 1): + with self.assertRaises(devrun.DevrunError): + devrun.Gate(self.dir, bad) + + +HOLD_SLOT = """ +import importlib.machinery, importlib.util, pathlib, sys +loader = importlib.machinery.SourceFileLoader("devrun", sys.argv[1]) +spec = importlib.util.spec_from_loader("devrun", loader) +devrun = importlib.util.module_from_spec(spec) +loader.exec_module(devrun) +held = devrun.Gate(pathlib.Path(sys.argv[2]), 1).admit(False, 0) +print("admitted", flush=True) +sys.stdin.read() +""" + + +class ExclusiveGate(GateCase): + def test_an_exclusive_run_waits_for_admitted_ordinary_runs(self): + a = self.contender() + self.admitted(a) + e = self.contender(exclusive=True) + e.wait_attempts(3) + self.assertFalse(e.admitted.is_set()) + a.release() + self.admitted(e) + + def test_ordinary_runs_wait_while_an_exclusive_run_holds_the_machine(self): + e = self.contender(exclusive=True) + self.admitted(e) + b = self.contender() + b.wait_attempts(3) + self.assertFalse(b.admitted.is_set()) + e.release() + self.admitted(b) + + def test_a_waiting_exclusive_run_is_not_starved_by_new_ordinary_runs(self): + a = self.contender() + self.admitted(a) + e = self.contender(exclusive=True) + e.wait_attempts(2) + # Three of four slots are free, but an exclusive run is queued: new runs queue behind it. + b, c = self.contender(), self.contender() + b.wait_attempts(3) + c.wait_attempts(3) + self.assertFalse(b.admitted.is_set() or c.admitted.is_set()) + a.release() + self.admitted(e) + b.wait_attempts(2) + self.assertFalse(b.admitted.is_set() or c.admitted.is_set()) + e.release() + self.admitted(b) + self.admitted(c) + + def test_two_exclusive_runs_take_turns(self): + e1 = self.contender(exclusive=True) + self.admitted(e1) + e2 = self.contender(exclusive=True) + e2.wait_attempts(3) + self.assertFalse(e2.admitted.is_set()) + e1.release() + self.admitted(e2) + + def test_the_exclusive_hold_is_released_when_the_run_raises(self): + gate = devrun.Gate(self.dir, 1) + with self.assertRaises(ValueError): + with gate.admit(True, 0): + raise ValueError("benchmark failed") + self.try_once(exclusive=False).release() + self.try_once(exclusive=True).release() + + def test_an_exclusive_run_that_gives_up_lets_ordinary_runs_in(self): + with self.try_once(): + with self.assertRaises(devrun.DevrunError): + self.try_once(exclusive=True) + # The failed exclusive request must not leave the turnstile held. + self.try_once().release() + + +class NamedLocks(GateCase): + def test_a_project_fetch_lock_admits_one_holder(self): + gate = devrun.Gate(self.dir, 1, clock=FakeClock(), sleep=lambda s: None, report=lambda m: None) + with gate.hold("fetch-geordi.lock", "the fetch lock", 0): + with self.assertRaises(devrun.DevrunError): + gate.hold("fetch-geordi.lock", "the fetch lock", 0) + gate.hold("fetch-bunny.lock", "the fetch lock", 0).release() + gate.hold("fetch-geordi.lock", "the fetch lock", 0).release() + + +class Liveness(GateCase): + def setUp(self): + super().setUp() + (self.dir / "runs").mkdir() + + def test_a_run_is_alive_while_its_record_is_held(self): + record = devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "alive") + record.finish(clean=True) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "unknown") + + def test_a_run_that_could_not_clean_up_is_dead_after_it_finishes(self): + devrun.RunRecord(self.dir, RUN_ID, {"project": "geordi"}).finish(clean=False) + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") + + def test_a_killed_run_is_dead(self): + child = subprocess.Popen([sys.executable, "-c", HOLD_RECORD, str(ROOT / "bin" / "devrun"), str(self.dir), RUN_ID], + stdout=subprocess.PIPE, stdin=subprocess.PIPE) + try: + self.assertEqual(child.stdout.readline(), b"recorded\n") + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "alive") + self.assertEqual(devrun.live_run_facts(self.dir), [{"project": "geordi", "source_key": "k"}]) + finally: + child.kill() + child.wait() + child.stdin.close() + child.stdout.close() + self.assertEqual(devrun.run_liveness(self.dir, RUN_ID), "dead") + self.assertEqual(devrun.live_run_facts(self.dir), []) + + def test_an_unknown_run_is_not_dead(self): + self.assertEqual(devrun.run_liveness(self.dir, "aaaaaaaaaaaa"), "unknown") + + def test_a_run_id_cannot_be_recorded_twice(self): + record = devrun.RunRecord(self.dir, RUN_ID, {}) + try: + with self.assertRaises(devrun.DevrunError): + devrun.RunRecord(self.dir, RUN_ID, {}) + finally: + record.finish(clean=True) + + +HOLD_RECORD = """ +import importlib.machinery, importlib.util, pathlib, sys +loader = importlib.machinery.SourceFileLoader("devrun", sys.argv[1]) +spec = importlib.util.spec_from_loader("devrun", loader) +devrun = importlib.util.module_from_spec(spec) +loader.exec_module(devrun) +record = devrun.RunRecord(pathlib.Path(sys.argv[2]), sys.argv[3], {"project": "geordi", "source_key": "k"}) +print("recorded", flush=True) +sys.stdin.read() +""" + + +# --------------------------------------------------------------------------- # +# --gc selection +# --------------------------------------------------------------------------- # + +LIVE, DEAD = "aaaaaaaaaaaa", "dddddddddddd" +KEY_GONE = devrun.source_key("/src/deleted") +KEY_HERE = devrun.source_key("/src/present") + + +def labels(role, run_id=None, owner="geordi", **extra): + out = {devrun.OWNER_LABEL: owner, devrun.ROLE_LABEL: role} + if run_id: + out[devrun.RUN_ID_LABEL] = run_id + out.update(extra) + return out + + +def container(run_id, running=False, owner="geordi", name=None, mounts=None, label_run_id=None): + names = devrun.run_resource_names("geordi", run_id) + return {"name": name or names["container"], "running": running, + "labels": labels("dev-run", label_run_id or run_id, owner), + "mounts": [names["work"], names["scratch"]] if mounts is None else mounts} + + +def run_volumes(run_id): + names = devrun.run_resource_names("geordi", run_id) + return [{"name": names[r], "labels": labels(r, run_id)} for r in devrun.PER_RUN_ROLES] + + +def target(path, key=None, name=None): + key = key or devrun.source_key(path) + return {"name": name or devrun.target_volume_name("geordi", key), + "labels": labels("target", **{devrun.SOURCE_PATH_LABEL: path, devrun.SOURCE_KEY_LABEL: key})} + + +class GcSelection(unittest.TestCase): + def plan(self, containers, volumes, protected=frozenset()): + return devrun.plan_gc("geordi", containers, volumes, + liveness=lambda rid: {LIVE: "alive", DEAD: "dead"}.get(rid, "unknown"), + path_exists=lambda p: p == "/src/present", + protected_keys=None if protected is None else set(protected)) + + def test_a_dead_runs_stopped_container_and_volumes_are_selected(self): + plan = self.plan([container(DEAD)], run_volumes(DEAD)) + self.assertEqual(plan["containers"], ["geordi-dev-run-dddddddddddd"]) + self.assertEqual(sorted(plan["volumes"]), sorted(v["name"] for v in run_volumes(DEAD))) + + def test_a_live_runs_resources_are_never_selected(self): + # A live run between phases has a stopped container, or none at all. + for containers in ([container(LIVE)], []): + plan = self.plan(containers, run_volumes(LIVE)) + self.assertEqual((plan["containers"], plan["volumes"]), ([], [])) + + def test_a_run_of_unknown_state_is_never_selected(self): + plan = self.plan([container("eeeeeeeeeeee")], run_volumes("eeeeeeeeeeee")) + self.assertEqual((plan["containers"], plan["volumes"]), ([], [])) + + def test_a_dead_runs_running_container_and_its_volumes_are_kept(self): + plan = self.plan([container(DEAD, running=True)], run_volumes(DEAD)) + self.assertEqual(plan["containers"], []) + self.assertEqual(plan["volumes"], [devrun.run_resource_names("geordi", DEAD)["target-evidence"]]) + self.assertTrue(any("still running" in reason for _, reason in plan["kept"])) + + def test_a_dead_runs_volume_used_by_a_foreign_container_is_kept(self): + foreign = {"name": "other-thing", "running": False, "labels": {}, "mounts": [devrun.run_resource_names("geordi", DEAD)["work"]]} + plan = self.plan([foreign], run_volumes(DEAD)) + self.assertNotIn("geordi-work-dddddddddddd", plan["volumes"]) + + def test_unlabelled_and_foreign_resources_are_never_selected(self): + containers = [ + {"name": "geordi-dev-run-dddddddddddd", "running": False, "labels": {}, "mounts": []}, + container(DEAD, owner="bunny"), + {"name": "geordi-dev-run", "running": False, "labels": labels("dev-run"), "mounts": []}, + container(DEAD, name="geordi-dev-run-ffffffffffff"), # name and run-id label disagree + container(DEAD, label_run_id="not-a-run-id", name="geordi-dev-run-not-a-run-id"), + ] + volumes = [ + {"name": "geordi-work-dddddddddddd", "labels": {}}, + {"name": "geordi-scratch-dddddddddddd", "labels": labels("scratch", DEAD, owner="bunny")}, + {"name": "geordi-work", "labels": labels("work")}, + {"name": "geordi-target", "labels": labels("target")}, + {"name": "geordi-target-evidence", "labels": labels("target-evidence")}, + {"name": "geordi-cargo-registry", "labels": labels("cargo-registry", DEAD)}, + {"name": "geordi-cargo-git", "labels": labels("cargo-git")}, + {"name": "geordi-work-ffffffffffff", "labels": labels("work", DEAD)}, + ] + plan = self.plan(containers, volumes) + self.assertEqual((plan["containers"], plan["volumes"]), ([], [])) + + def test_the_target_of_a_deleted_worktree_is_selected_and_an_existing_one_kept(self): + plan = self.plan([], [target("/src/deleted"), target("/src/present")]) + self.assertEqual(plan["volumes"], [devrun.target_volume_name("geordi", KEY_GONE)]) + + def test_a_target_whose_labels_do_not_match_its_name_is_kept(self): + plan = self.plan([], [target("/src/deleted", key=KEY_HERE), + target("/src/deleted", name="geordi-target-0000000000000000"), + {"name": devrun.target_volume_name("geordi", KEY_GONE), + "labels": labels("target", **{devrun.SOURCE_KEY_LABEL: KEY_GONE})}]) + self.assertEqual(plan["volumes"], []) + + def test_a_target_in_use_is_kept(self): + name = devrun.target_volume_name("geordi", KEY_GONE) + user = {"name": "geordi-dev-run-aaaaaaaaaaaa", "running": False, "labels": labels("dev-run", LIVE), "mounts": [name]} + self.assertEqual(self.plan([user], [target("/src/deleted")])["volumes"], []) + self.assertEqual(self.plan([], [target("/src/deleted")], protected={KEY_GONE})["volumes"], []) + self.assertEqual(self.plan([], [target("/src/deleted")], protected=None)["volumes"], []) + + +# --------------------------------------------------------------------------- # +# git-locks registration: a fake git-locks executable at the process boundary +# --------------------------------------------------------------------------- # + + +class LeaseRegistration(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.dir = pathlib.Path(self.tmp.name) + self.waits = 0 + + def tearDown(self): + self.tmp.cleanup() + + def fake_git_locks(self, exit_codes): + """A git-locks that records its arguments and exits with the next code in the list.""" + script = self.dir / "git-locks" + (self.dir / "codes").write_text("".join(f"{c}\n" for c in exit_codes)) + script.write_text("#!/bin/sh\n" + f'echo "$*" >> "{self.dir}/calls"\n' + f'n=$(wc -l < "{self.dir}/calls" | tr -d " ")\n' + f'code=$(sed -n "${{n}}p" "{self.dir}/codes")\n' + 'exit "${code:-0}"\n') + script.chmod(0o755) + return str(script) + + def calls(self): + return (self.dir / "calls").read_text().splitlines() + + def lease(self, executable): + return devrun.Lease("devrun-geordi-x", "devrun-x", "host/docker/geordi-dev-run-x/", 100, "note", executable, env={}) + + def waiter(self): + def count(_s): + self.waits += 1 + return devrun.Waiter("lock", None, FakeClock(), count, lambda m: None, 1, 30) + + def test_a_claimed_key_is_released_at_the_end(self): + lease = self.lease(self.fake_git_locks([0, 0])) + self.assertTrue(lease.claim(self.waiter())) + lease.release() + lease.release() + claim, release = self.calls() + self.assertIn("claim --job devrun-geordi-x --holder devrun-x --ttl 100", claim) + self.assertTrue(claim.endswith("host/docker/geordi-dev-run-x/")) + self.assertEqual(release, "release --job devrun-geordi-x") + self.assertEqual(lease.env["GIT_LOCKS_STORE"], devrun.LOCK_STORE) + + def test_a_refusal_waits_and_retries(self): + lease = self.lease(self.fake_git_locks([1, 1, 0])) + self.assertTrue(lease.claim(self.waiter())) + self.assertEqual(self.waits, 2) + + def test_a_broken_or_missing_git_locks_does_not_stop_the_run(self): + for executable in (self.fake_git_locks([2]), "", str(self.dir / "absent")): + with contextlib.redirect_stderr(io.StringIO()) as err: + lease = self.lease(executable) + self.assertFalse(lease.claim(self.waiter())) + lease.release() + self.assertIn("devrun: git-locks", err.getvalue()) + self.assertEqual(len(self.calls()), 1) # the one failed claim, and no release after it + + +class StateDirectory(unittest.TestCase): + def test_created_private_with_a_runs_directory(self): + with tempfile.TemporaryDirectory() as d: + path = devrun.state_dir({devrun.STATE_ENV: f"{d}/state"}) + self.assertEqual(stat.S_IMODE(os.stat(path).st_mode) & 0o077, 0) + self.assertTrue((path / "runs").is_dir()) + + def test_a_directory_writable_by_others_is_refused(self): + with tempfile.TemporaryDirectory() as d: + os.chmod(d, 0o777) + with self.assertRaises(devrun.DevrunError): + devrun.state_dir({devrun.STATE_ENV: d}) + + def test_default_follows_xdg_state_home(self): + with tempfile.TemporaryDirectory() as d: + self.assertEqual(devrun.state_dir({"XDG_STATE_HOME": d}), pathlib.Path(d) / "devrun") class Arguments(unittest.TestCase): @@ -259,6 +832,32 @@ def test_export_paths_are_checked(self): with self.assertRaises(devrun.DevrunError): devrun.parse_args(["geordi", ".", "--commit", "HEAD", "--export", "../../etc", "--", "true"]) + def test_run_cap_flag_environment_and_bounds(self): + base = ["geordi", ".", "--commit", "HEAD"] + saved = devrun.os.environ.get(devrun.MAX_RUNS_ENV) + try: + devrun.os.environ.pop(devrun.MAX_RUNS_ENV, None) + self.assertEqual(devrun.parse_args(base + ["--", "true"]).max_runs, 4) + devrun.os.environ[devrun.MAX_RUNS_ENV] = "2" + self.assertEqual(devrun.parse_args(base + ["--", "true"]).max_runs, 2) + self.assertEqual(devrun.parse_args(base + ["--max-runs", "1", "--", "true"]).max_runs, 1) + for bad in ("x", "-1", "0", "65"): + devrun.os.environ[devrun.MAX_RUNS_ENV] = bad + with self.assertRaises(devrun.DevrunError): + devrun.parse_args(base + ["--", "true"]) + finally: + if saved is None: + devrun.os.environ.pop(devrun.MAX_RUNS_ENV, None) + else: + devrun.os.environ[devrun.MAX_RUNS_ENV] = saved + + def test_exclusive_and_wait(self): + ns = devrun.parse_args(["geordi", ".", "--commit", "HEAD", "--exclusive", "--wait", "0", "--", "true"]) + self.assertTrue(ns.exclusive) + self.assertEqual(ns.wait, 0) + with self.assertRaises(devrun.DevrunError): + devrun.parse_args(["geordi", ".", "--commit", "HEAD", "--wait", "-1", "--", "true"]) + def test_good_arguments(self): ns = devrun.parse_args(["geordi", ".", "--commit", "HEAD", "--fetch", "--export", "hashes.txt", "--", "cargo", "test"]) self.assertEqual(ns.command, ["cargo", "test"])