Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,40 @@ version numbers yet, so changes are listed under Unreleased.

## [Unreleased]

### Fixed

- Two `--commit` runs of different commits of one worktree, started close
together, could share a build: both copies got their time before either build
started, so cargo served the first run's binary to the second (issue #12). A
`--worktree` run between a `--commit` run's copy and its build had the same
effect. Pinned runs (`--commit`, `--history`) now build into their own target
per worktree, `<project>-target-pinned-<key>`, and hold that target's lock from
before their command starts until their container has stopped. Pinned runs of
one worktree now run their commands one at a time.
- A `--worktree` run after a `--commit` run of the same worktree could reuse the
commit's build for an edit older than that build (part of issue #9). The two
modes no longer share a target.
- A container whose runner was killed with SIGKILL no longer escapes the run cap:
it counts as a held slot, and `--exclusive` runs wait for it, until it stops.
Once its own timeout has passed, a waiting run or `--gc` stops it. Run
containers carry two new labels for this, `org.flyingrobots.phase` and
`org.flyingrobots.timeout`.
- A surviving command of a killed `--exclusive` run keeps every ordinary run out
until it stops; it no longer counts as one slot. The command container is
labelled `org.flyingrobots.admission` and the run record keeps the mode.
- A liveness probe takes a shared lock, so two probes of a dead run's record at
the same moment no longer make one of them read the run as alive.
- A run keeps its slot, the exclusive gate, the fetch lock, and the pinned lock
until Docker confirms that its container has stopped or it has been removed;
on SIGTERM they were released before cleanup had removed the container.
- A signal between Docker creating a volume or container and devrun recording it
no longer leaves an untracked resource: devrun records the intent first, and
cleanup lists the run's labelled resources to resolve it, keeping the run's
liveness record when Docker cannot answer. A signal during cleanup is ignored.
- `--gc` no longer forgets a dead run when `docker inspect` fails: a failed
inspect is accepted only when each missing resource is also missing from a
fresh listing.

### Changed

- `bin/devrun` no longer takes the machine-wide `host/heavy-work` lock or a
Expand All @@ -21,6 +55,10 @@ version numbers yet, so changes are listed under Unreleased.
(`<project>-target-evidence-<id>`) and is removed with it.
- `--commit` runs give the copied files the current time, so an older commit
cannot reuse output built from a newer one (part of issue #9).
- Existing `<project>-target-<key>` volumes stay the targets of `--worktree`
runs. They may still hold output of earlier `--commit` runs, which the first
`--worktree` run after the upgrade can reuse for files older than that output;
remove such a target with `docker volume rm` if that matters.
- Each run registers `host/docker/<project>-dev-run-<id>/` in the workstation
git-locks store for visibility; a missing or broken git-locks no longer stops a
run.
Expand Down
55 changes: 35 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,33 +70,45 @@ Runs do not wait for each other unless the machine is full. The rules are in
every exit path, including SIGTERM and SIGINT.
- At most 4 command phases run at once on the machine. `--max-runs N` or
`DEVRUN_MAX_RUNS=N` sets the cap; a waiting run prints how long it has waited,
and gives up after `--wait SECONDS` (default 3600).
and gives up after `--wait SECONDS` (default 3600). A run keeps its slot until
its container has stopped.
- A command container whose runner was killed (SIGKILL) keeps counting against
the cap, and `--exclusive` runs wait for it, until it ends; one from an
`--exclusive` run keeps every other run out. Once its own
`--timeout` has passed, the next run that waits for it (or `--gc`) stops it;
nothing stops it earlier.
- `--exclusive` waits until no other command phase runs and keeps them all out
until it finishes. Use it for every performance measurement:
`bin/devrun echo ~/git/echo --commit HEAD --exclusive -- cargo bench --locked`.
- Fetch phases of one project take turns (cargo's own cache lock cannot work on
the read-only root).
- `--commit` and `--history` runs of one worktree take turns on that worktree's
pinned build target (below).
- With `CI=true` none of this coordination applies; CI machines run one job.
- Runs register `host/docker/<project>-dev-run-<id>/` in the workstation
git-locks store for visibility; without git-locks they run unregistered.

### Build output and disk

The iteration build output lives in one volume per worktree,
`<project>-target-<key>`, where the key is a hash of the worktree's real path.
Each worktree keeps its own warm cache and never sees another worktree's
artifacts. Two runs from the same worktree share it safely: cargo locks its build
directory. The cost is disk: each target can grow to a full build of the
workspace, so eight worktrees of a large workspace can hold several times the
20 GiB per-project budget. `--evidence` builds into a new, empty volume that
belongs to the one run and is removed with it.

`--commit` runs give the copied files the current time, so an older commit never
reuses output built from a newer one in the same worktree's target. `--worktree`
runs keep the files' own times, like cargo on the host. A `--worktree` run after a
`--commit` run of the same worktree can still reuse the commit's output for files
that are older than that run (flyingrobots/devcontainers#9); touch the changed
files, or use `--evidence`, when that matters.
Build output lives in kept volumes per worktree, keyed by a hash of the
worktree's real path: `<project>-target-<key>` for `--worktree` runs and
`<project>-target-pinned-<key>` for `--commit` and `--history` runs. Each
worktree keeps its own warm caches and never sees another worktree's artifacts.
Two `--worktree` runs from the same worktree share their target safely: cargo
locks its build directory. The cost is disk: each target can grow to a full
build of the workspace, so eight worktrees of a large workspace can hold several
times the 20 GiB per-project budget. `--evidence` builds into a new, empty volume
that belongs to the one run and is removed with it.

A pinned run's result does not depend on what else runs: its copy gets the
current time (`--commit`) or is checked out in the container (`--history`), and
no other run can build in the pinned target until its command has ended, so
cargo rebuilds the workspace's crates from this run's source and reuses only
registry and git dependencies. Pinned runs of one worktree therefore run their
commands one at a time. `--worktree` runs keep the files' own times, like cargo
on the host, and never see a pinned run's output; a tool that restores old
modification times can still make them reuse stale output
(flyingrobots/devcontainers#9).

### Leftovers: `bin/devrun --gc`

Expand All @@ -108,10 +120,13 @@ bin/devrun --gc bunny # remove it
`--gc` removes the stopped containers and the volumes of runs that died without
cleaning up (killed with SIGKILL, or a crash), and the targets of worktrees that
no longer exist. A run counts as dead only when its liveness lock in the devrun
state directory is free, which proves its process exited. `--gc` never removes a
running container, a volume in use, the cargo caches, or anything without a
valid run id or source label, such as the fixed-name volumes of older runner
versions (remove those by hand once nothing uses them).
state directory is free, which proves its process exited. A dead run's container
that is still running is stopped first if its own timeout has passed, and kept
otherwise. `--gc` never removes a running container, a volume in use, the cargo
caches, or anything without a valid run id or source label, such as the
fixed-name volumes of older runner versions (remove those by hand once nothing
uses them). If Docker cannot inspect a resource that still exists, `--gc` stops
with an error and forgets no run.

Test it:

Expand Down
Loading
Loading