Skip to content

Queue and cancel remote Builds on one Machine - #838

Merged
nick-potts merged 6 commits into
mainfrom
t3code/implement-request-807
Sep 9, 2026
Merged

nick-potts merged 6 commits into
mainfrom
t3code/implement-request-807

Conversation

@nick-potts

Copy link
Copy Markdown
Contributor

Remote Builds now wait in a Machine-local FIFO instead of rejecting every competing request as busy. One Build runs at a time, with eight waiting slots by default; source and secrets stay on the client until admission. Machine-local policy validates queue capacity and separate queue/active timeouts.

Cancellation, disconnect and expiry remove waiters or request bounded active termination. Upload staging retains exclusive ownership through cleanup. Restart discards waiting work and attempts bounded abandoned-builder cleanup; uncertain termination keeps ownership quarantined. Progress reports the selected Machine, separate waiting/execution timings and completed versus unattempted work.

Validation:

  • Rung 1: controlled-time FIFO, capacity, expiry, cancellation, overtaking regression, policy validation and staging/quarantine ownership tests.
  • Rungs 2–3: owned transport and actual CLI checks for upload after admission, cancellation, outcome rendering and no local Docker access.
  • Rung 4: real remote build/cancellation/quarantine exercise and upload-timeout/restart/subsequent-safe-build exercise passed with current binaries in the testkit.
  • cargo test --workspace --all-features, workspace Clippy with warnings denied, and formatting passed.
  • Standards, Spec, Maintainability and Simplicity reviews all passed.

Closes #807

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-09T04:03:30.399727Z 3a0b136 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6788e4921a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/ployzd/src/build.rs Resolved
Comment thread crates/ployz-build/src/execution.rs Resolved
Preserve FIFO queue admission and timing alongside bounded terminal reports and observed-work evidence.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 267209e3e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/ployz-build/src/builder.rs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28b12e5aae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/ployz-build/src/builder.rs Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1ae705b9e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

{
Ok(Ok(admission)) => admission,
Ok(Err(error)) => return failure(Stage::Admission, error),
Ok(Err(error)) => return failure(Stage::Admission, error).with_work(work),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Queue remote attempts behind local lock holders

When a direct local ployz build already owns the shared builder lock but no remote RPC is active, runner.enter() grants this request the active semaphore permit because that semaphore tracks only RPCs; Admission::try_acquire_with then returns Busy, and this branch immediately rejects the request instead of placing it in the Machine-local FIFO. Coordinate admission with the shared filesystem lock so local-versus-remote contention also waits within the configured queue timeout.

Useful? React with 👍 / 👎.

Comment thread crates/ployz-build/src/upload.rs Resolved
@nick-potts
nick-potts merged commit a56e3e2 into main Sep 9, 2026
7 checks passed
@nick-potts
nick-potts deleted the t3code/implement-request-807 branch September 30, 2026 15:15
nick-potts added a commit that referenced this pull request Sep 30, 2026
* feat(build): queue and cancel remote Machine builds

Closes #807

* fix(build): clean abandoned builders at daemon startup

* fix(build): quarantine ownership before Docker preflight

* fix(build): remove abandoned uploads during clean restart

---------

Co-authored-by: Codex <codex@codex-vm.tailcb9c5.ts.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Queue and cancel Builds on one Machine

1 participant