Repository navigation
Keep automations starting when Daytona's limits are reached - #349
arseniycodes wants to merge 3 commits into
Conversation
Daytona rejects new sandboxes once the organization reaches its disk, CPU or memory limit. Stopped sandboxes count toward the 2000 GiB disk limit. Each automation run keeps its 10 GiB sandbox stopped for a day, so about 200 runs a day filled the limit and every new run failed on 2026-10-16. - Paused sandboxes are archived after 30 minutes. Archived sandboxes do not count toward the disk limit and still resume. - When Daytona rejects a creation at a limit, the worker archives the longest-stopped sandboxes (disk) and retries the same name for about four minutes (CPU and memory) before failing with a clear message. - A worker sweep every 10 minutes archives paused sandboxes, deletes archived and failed ones older than a day, and logs usage. - A sandbox that never appeared during cleanup no longer replaces the creation error, and the remaining aggregate message includes it.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 04e7c5ec29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (sandbox.state === SandboxState.ARCHIVED || sandbox.state === SandboxState.ERROR) && | ||
| Date.parse(sandbox.updatedAt ?? "") < expiredBefore |
There was a problem hiding this comment.
When a Responder sandbox remains in Daytona's distinct BUILD_FAILED terminal state past the retention window, this predicate never adds it to expired because it only recognizes ARCHIVED and ERROR. Such orphaned build failures therefore remain indefinitely and continue contributing their disk allocation, undermining the quota-recovery sweep; include SandboxState.BUILD_FAILED among the expired failure states.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f83ddd1a2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| function isDaytonaQuotaError(error: unknown): boolean { | ||
| return error instanceof Error && /\blimit exceeded\b/i.test(error.message); | ||
| } |
There was a problem hiding this comment.
When Daytona returns a 429 whose message contains “rate limit exceeded,” this predicate also matches it. Because creator.create() includes post-create startup polling, that rate-limit error can occur after the sandbox exists; on the final retry or an abort, the quota branch calls failCreation without adding the current name to abandoned, leaving that sandbox undeleted and reporting a misleading DaytonaCapacityError. Restrict this check to the known total CPU, memory, and disk quota messages, or explicitly exclude provider errors with status 429 so they retain the normal cleanup path.
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Why
On 2026-10-16 from 12:00 UTC, automation runs in the EU cell failed with "Unable to create or clean up Daytona sandbox …". The real error was hidden:
Stopped sandboxes count toward Daytona's disk limit. Every automation run without a native subscription keeps its sandbox stopped for 24 h. With the 10 GiB snapshot, about 200 runs a day filled the limit (196 stopped sandboxes, 1960 GiB). Archiving them by hand fixed it for now.
What changes
setAutoArchiveInterval). Archived sandboxes do not count toward the disk limit and still resume, more slowly.responder-*sandboxes (stopped > 5 min);DaytonaCapacityError, "Responder's sandbox capacity is temporarily full. Try again in a few minutes." This replaces Daytona's "upgrade your Tier" text.responder-*sandboxes older than 24 h, and logsdaytona_sandbox_sweepwith disk, CPU and memory in use. Deleting does not rely on Daytona's auto-delete covering archived sandboxes.Tests
sandbox.test.ts: disk-full retry archives only long-stopped sandboxes and reuses the name without waiting for the rejected sandbox to appear; CPU-full waits the full budget, then throwsDaytonaCapacityError; paused sandboxes get the archive interval and sandboxes deleted on stop do not; the sweep archives, deletes expired, and logs usage.pnpm typecheck, eslint on changed files, andvitest run apps/worker(674 passed).Follow-ups
daytona_sandbox_sweep.diskGiB(hosted repo).DaytonaCapacityErrorinstead of failing it.Summary by cubic
Fixes automation runs failing when the organization's Daytona limits are reached, as happened in the EU cell on 2026-10-16 when stopped sandboxes filled the 2000 GiB disk limit.
responder-*sandboxes (disk only) and retries the same name for about 4 minutes before failing with a clearDaytonaCapacityError. Rate limits keep the normal delete-and-retry cleanup.responder-*sandboxes older than 24 hours, and logsdaytona_sandbox_sweepwith usage.Written for commit 8b02fd3. Summary will update on new commits.