fix(smoketests): align blueprint test timeouts with long-poll budgets - #824
Closed
jason-rl wants to merge 1 commit into
Closed
fix(smoketests): align blueprint test timeouts with long-poll budgets#824jason-rl wants to merge 1 commit into
jason-rl wants to merge 1 commit into
Conversation
The blueprint smoketests wrapped 20-30 minute long-poll budgets in a 120s jest timeout, so any build slower than 120s aborted the test before the SDK could report why, and left the blueprint behind because the timed-out call never returned an id for the cleanup path to use. Cap the long-poll budgets at MEDIUM_TIMEOUT inside a LONG_TIMEOUT jest budget, and clean blueprints up by name so cleanup runs even when the await times out.
Contributor
🤖 CodeAnt AI — Review Status
|
Contributor
Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
| }, | ||
| { | ||
| longPoll: { timeoutMs: 20 * 60 * 1000 }, | ||
| longPoll: { timeoutMs: MEDIUM_TIMEOUT }, |
Contributor
There was a problem hiding this comment.
Suggestion: createAndAwaitRunning creates the devbox before it begins polling and only returns the created view after the devbox reaches the target state. If this new 300-second polling timeout fires, devbox is still undefined, so the finally block cannot shut down the server-side devbox and the smoketest leaks it. [resource leak]
Severity Level: Major ⚠️
- ❌ Blueprint lifecycle tests can leak created devboxes.
- ⚠️ Leaked devboxes consume smoke-test environment resources.
- ⚠️ Both ID and name devbox paths are affected.(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** tests/smoketests/blueprints.test.ts
**Line:** 46:46
**Comment:**
*Resource Leak: `createAndAwaitRunning` creates the devbox before it begins polling and only returns the created view after the devbox reaches the target state. If this new 300-second polling timeout fires, `devbox` is still undefined, so the `finally` block cannot shut down the server-side devbox and the smoketest leaks it.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Description
The blueprint smoketests pass a 120s (
SHORT_TIMEOUT) jest timeout while asking the SDK to long-poll for up to 20–30 minutes. The jest budget always wins, so any build slower than 120s fails as a bareExceeded timeout of 120000 ms for a testwith no SDK error, no build state, and no cleanup — jest aborts the test beforefinallyruns.That is what broke the prod deploy smoketest in runloopai/runloop run 31224813985:
create blueprint with secrets and await buildtimed out at 120002 ms (http1) and 120001 ms (http2), while the siblingcreate blueprint and await buildin the same file passed in 12675 ms and other Dockerfile builds in the same run completed in 25–50s. The environment was mid-deploy and prod blueprint builds were running long that hour; the test converted normal build variance into a deploy-blocking failure with zero diagnostics, and leaked the blueprint on the way out.Motivation
A smoketest should fail with the reason the build didn't finish, not with a stopwatch that expires 15× earlier than the budget it's supervising — and it should never leak resources in prod when it does fail.
Changes
longPoll.timeoutMsatMEDIUM_TIMEOUT(5 min) and raise the jest timeouts toLONG_TIMEOUT(10 min) for the four blueprint tests, so the SDK's own timeout fires first and reports the terminal state it was waiting on.cleanUpBlueprintsByNametotests/smoketests/utils.tsand use it inafterAlland the secrets test'sfinally, so cleanup works even when the create-and-await call timed out before returning an id.let bpt/BlueprintViewimport that the cleanup change orphaned.No production SDK code is touched; this is test-only.
Testing
Test-only change; verification is the smoketest workflow itself (both the http1 and http2 transports).
Breaking Changes
None.
Checklist
CodeAnt-AI Description
Prevent blueprint smoketests from timing out before builds finish
What Changed
Impact
✅ Fewer deploy-blocking smoketest timeouts✅ Clearer build and startup failure results✅ Fewer leaked test blueprints💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.