Skip to content

fix(smoketests): align blueprint test timeouts with long-poll budgets - #824

Closed
jason-rl wants to merge 1 commit into
mainfrom
fix/blueprint-smoketest-timeout-budget
Closed

fix(smoketests): align blueprint test timeouts with long-poll budgets#824
jason-rl wants to merge 1 commit into
mainfrom
fix/blueprint-smoketest-timeout-budget

Conversation

@jason-rl

@jason-rl jason-rl commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

User description

Description

The blueprint smoketests pass a 120s (SHORT_TIMEOUT) jest timeout while asking the SDK to long-poll for up to 20–30 minutes. The jest budget always wins, so any build slower than 120s fails as a bare Exceeded timeout of 120000 ms for a test with no SDK error, no build state, and no cleanup — jest aborts the test before finally runs.

That is what broke the prod deploy smoketest in runloopai/runloop run 31224813985: create blueprint with secrets and await build timed out at 120002 ms (http1) and 120001 ms (http2), while the sibling create blueprint and await build in the same file passed in 12675 ms and other Dockerfile builds in the same run completed in 25–50s. The environment was mid-deploy and prod blueprint builds were running long that hour; the test converted normal build variance into a deploy-blocking failure with zero diagnostics, and leaked the blueprint on the way out.

Motivation

A smoketest should fail with the reason the build didn't finish, not with a stopwatch that expires 15× earlier than the budget it's supervising — and it should never leak resources in prod when it does fail.

Changes

  • Cap longPoll.timeoutMs at MEDIUM_TIMEOUT (5 min) and raise the jest timeouts to LONG_TIMEOUT (10 min) for the four blueprint tests, so the SDK's own timeout fires first and reports the terminal state it was waiting on.
  • Add cleanUpBlueprintsByName to tests/smoketests/utils.ts and use it in afterAll and the secrets test's finally, so cleanup works even when the create-and-await call timed out before returning an id.
  • Drop the now-unused let bpt / BlueprintView import that the cleanup change orphaned.

No production SDK code is touched; this is test-only.

Testing

  • Unit tests added/updated
  • Smoke Tests added/updated
  • Manual testing performed

Test-only change; verification is the smoketest workflow itself (both the http1 and http2 transports).

Breaking Changes

None.

Checklist

  • Code follows the project's style guidelines
  • Self-review of the code has been performed
  • Changes generate no new warnings

CodeAnt-AI Description

Prevent blueprint smoketests from timing out before builds finish

What Changed

  • Blueprint build and devbox startup tests now allow up to 10 minutes for completion while limiting long-polling to 5 minutes.
  • Failed or slow operations can report their build or startup outcome instead of being stopped by the previous 2-minute test limit.
  • Cleanup now finds blueprints by name, including cases where a timed-out operation never returned an ID, and removes them after tests finish.

Impact

✅ Fewer deploy-blocking smoketest timeouts
✅ Clearer build and startup failure results
✅ Fewer leaked test blueprints

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

The blueprint smoketests wrapped 20-30 minute long-poll budgets in a
120s jest timeout, so any build slower than 120s aborted the test before
the SDK could report why, and left the blueprint behind because the
timed-out call never returned an id for the cleanup path to use.

Cap the long-poll budgets at MEDIUM_TIMEOUT inside a LONG_TIMEOUT jest
budget, and clean blueprints up by name so cleanup runs even when the
await times out.
@codeant-ai

codeant-ai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

🤖 CodeAnt AI — Review Status

Status Commit Started (UTC) Finished (UTC)
✅ Reviewed your PR 44d11cc Aug 07, 2026 · 23:12 23:14

@codeant-ai

codeant-ai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@codeant-ai codeant-ai Bot added the size:S This PR changes 10-29 lines, ignoring generated files label Aug 7, 2026
},
{
longPoll: { timeoutMs: 20 * 60 * 1000 },
longPoll: { timeoutMs: MEDIUM_TIMEOUT },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: createAndAwaitRunning creates the devbox before it begins polling and only returns the created view after the devbox reaches the target state. If this new 300-second polling timeout fires, devbox is still undefined, so the finally block cannot shut down the server-side devbox and the smoketest leaks it. [resource leak]

Severity Level: Major ⚠️
- ❌ Blueprint lifecycle tests can leak created devboxes.
- ⚠️ Leaked devboxes consume smoke-test environment resources.
- ⚠️ Both ID and name devbox paths are affected.

Fix in Cursor Fix in VSCode Claude

(Use Cmd/Ctrl + Click for best experience)

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** tests/smoketests/blueprints.test.ts
**Line:** 46:46
**Comment:**
	*Resource Leak: `createAndAwaitRunning` creates the devbox before it begins polling and only returns the created view after the devbox reaches the target state. If this new 300-second polling timeout fires, `devbox` is still undefined, so the `finally` block cannot shut down the server-side devbox and the smoketest leaks it.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

@jason-rl jason-rl closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S This PR changes 10-29 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant