Skip to content

devrun: stale builds between concurrent pinned runs, and admission that ignores surviving containers #12

Closed

Description

@flyingrobots
.dag: issue-dag/2
.id: flyingrobots/devcontainers#12
.kind: task
.type: bug
.blocked_by: []
.size_hours: 6
area: `bin/devrun`, `tests/`, `docs/hygiene.md`
source: review of PR #11, which arrived after the merge

Bug

1. Background Context

PR #11 removed devrun's machine-wide lock: runs now proceed in parallel under a cap, each worktree has its own build cache, --exclusive isolates timing runs, and --gc removes leftovers. It merged before its automated review finished. The review left four findings. They have been read against the code as described, not yet reproduced.

2. Problem Description

  1. Two --commit runs of one worktree can share a stale build (major). --commit copies get the current modification time so the cache does not serve another commit's build. Two concurrent --commit runs of different commits of the same worktree can both refresh their copies before either build starts; both copies then predate the first build's start, and the second run can reuse the first run's binary although its source differs. Evidence for a pinned commit must not depend on what else is running.
  2. A surviving container is invisible to admission (major). If a runner process is killed with SIGKILL during its command, its admission locks vanish while its container keeps running. Another run can then exceed the cap, and an --exclusive timing run can start beside the survivor, which invalidates the timing. The same gap exists briefly on SIGTERM, where admission is released before cleanup stops the container.
  3. An interrupted volume creation can leave an untracked volume (minor). If SIGTERM arrives after Docker creates a volume and before devrun records it, cleanup reports success, the liveness record is deleted, and --gc later treats the volume as unknown.
  4. --gc can drop a dead run's record when inspection fails (minor). _inspect returns an empty list when docker inspect exits non-zero with empty output, so a resource that still exists can look absent, the run record is unlinked, and later --gc runs lose the evidence that the run is dead.

2b. Proposed Solution

  1. Hold a per-source lock across the timestamp refresh and the build for --commit runs that share a target volume, or give pinned-commit runs a target keyed by the commit; add a concurrent regression test for the ordering described.
  2. Make admission count live command containers (by label), not only live runner processes, and keep a run's admission until its container is confirmed stopped.
  3. Record the intent to create a volume before creating it and resolve that state during cleanup.
  4. Treat an inspect failure as an error unless every missing name is confirmed absent by a fresh listing.

2d. Acceptance Criteria

  • Each case has a test that fails before the fix (real Docker for 1 and 2; real file locks and injected failures for 3 and 4).
  • docs/hygiene.md states the guarantees and their limits.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions