Skip to content

Switch to fail-fast prompt waits after WaitUntilAsync in CLI E2E tests #15934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Copilot wants to merge 2 commits into
base: main
Choose a base branch
from
+9 −9
Open

Switch to fail-fast prompt waits after WaitUntilAsync in CLI E2E tests #15934

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2fabe65
feat: add fail-fast error detection to WaitUntilAsync lambdas in CLI …
Copilot Apr 7, 2026
4c1b555
refactor: use WaitForSuccessPromptFailFastAsync instead of inline ERR…
Copilot Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions tests/Aspire.Cli.EndToEnd.Tests/AgentCommandTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ public async Task AgentCommands_AllHelpOutputs_AreCorrect()
await auto.WaitUntilAsync(
s => s.ContainsText("mcp") && s.ContainsText("init"),
timeout: TimeSpan.FromSeconds(30), description: "agent help showing mcp and init subcommands");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fail-fast prompt detection occurs only after WaitUntilAsync succeeds. If the help command fails before printing the expected text, WaitUntilAsync will still time out. If the intent is to avoid long timeouts on non-zero exits, consider adding an ERR-prompt check inside WaitUntilAsync (or use a helper that fails fast on ERR while waiting for output).

Copilot uses AI. Check for mistakes.

// Test 2: aspire agent mcp --help
await auto.TypeAsync("aspire agent mcp --help");
Expand All @@ -66,7 +66,7 @@ await auto.WaitUntilAsync(
await auto.WaitUntilAsync(
s => s.ContainsText("tools") && s.ContainsText("call"),
timeout: TimeSpan.FromSeconds(30), description: "mcp help showing tools and call subcommands");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same concern: this only fails fast after WaitUntilAsync succeeds. If aspire mcp --help exits with an error before producing the expected tokens, the test will still wait the full WaitUntilAsync timeout. Consider integrating ERR prompt detection into the WaitUntilAsync predicate (or a wrapper helper).

Copilot uses AI. Check for mistakes.

// Test 5: aspire mcp tools --help
await auto.TypeAsync("aspire mcp tools --help");
Expand Down Expand Up @@ -186,7 +186,7 @@ public async Task DoctorCommand_DetectsDeprecatedAgentConfig()
await auto.WaitUntilAsync(
s => s.ContainsText("dev-certs") && s.ContainsText("deprecated") && s.ContainsText("aspire agent init"),
timeout: TimeSpan.FromSeconds(60), description: "doctor output with deprecated warning and fix suggestion");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fail-fast prompt wait comes after the output-text WaitUntilAsync. If aspire doctor fails before emitting the expected warning text, WaitUntilAsync will still run until timeout. To match the stated goal of fail-fast on ERR prompts, consider checking for [N ERR: within the WaitUntilAsync predicate as well.

Copilot uses AI. Check for mistakes.

await auto.TypeAsync("exit");
await auto.EnterAsync();
Expand Down
6 changes: 3 additions & 3 deletions tests/Aspire.Cli.EndToEnd.Tests/BannerTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ public async Task Banner_DisplayedOnFirstRun()
await auto.WaitUntilAsync(
s => s.ContainsText(RootCommandStrings.BannerWelcomeText) && s.ContainsText("Telemetry"),
timeout: TimeSpan.FromSeconds(30), description: "waiting for banner and telemetry notice on first run");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching to WaitForSuccessPromptFailFastAsync only checks for an ERR prompt after this WaitUntilAsync has already succeeded. If the command exits with an error before printing the expected banner text, WaitUntilAsync will still spin until its timeout. To truly fail fast on non-zero exits, consider adding an ERR-prompt check inside the WaitUntilAsync predicate (or wrap WaitUntilAsync in a helper that returns/throws when [N ERR: is observed).

Copilot uses AI. Check for mistakes.
await auto.TypeAsync("exit");
await auto.EnterAsync();

Expand Down Expand Up @@ -81,7 +81,7 @@ public async Task Banner_DisplayedWithExplicitFlag()
await auto.WaitUntilAsync(
s => s.ContainsText(RootCommandStrings.BannerWelcomeText) && s.ContainsText("CLI"),
timeout: TimeSpan.FromSeconds(30), description: "waiting for banner with version info");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same concern as above: this fail-fast prompt check runs only after WaitUntilAsync completes, so it won’t shorten timeouts when the command fails before producing the expected output. Consider integrating ERR prompt detection into the WaitUntilAsync predicate (or using a helper that does both).

Copilot uses AI. Check for mistakes.
await auto.TypeAsync("exit");
await auto.EnterAsync();

Expand Down Expand Up @@ -129,7 +129,7 @@ await auto.WaitUntilAsync(s =>
// Only return true once the help hint is visible at the end of the output
return s.ContainsText(HelpGroupStrings.HelpHint);
}, timeout: TimeSpan.FromSeconds(30), description: "waiting for help output to complete");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fail-fast prompt wait happens after the help-output WaitUntilAsync, so it won’t prevent a full WaitUntilAsync timeout if the command errors before emitting HelpHint. If the goal is to fail immediately on non-zero exits, consider checking for an ERR prompt in the WaitUntilAsync loop as well.

Copilot uses AI. Check for mistakes.
await auto.TypeAsync("exit");
await auto.EnterAsync();

Expand Down
2 changes: 1 addition & 1 deletion tests/Aspire.Cli.EndToEnd.Tests/CentralPackageManagementTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ await auto.WaitUntilAsync(s =>
await auto.EnterAsync();
}

await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this test, WaitUntilAsync above waits for either the version-selection prompt or a success prompt. If the command exits with an error prompt, WaitUntilAsync will still time out before reaching this fail-fast check. Consider updating the preceding WaitUntilAsync predicate to also return on an ERR prompt (and throw), or using a helper that waits for OK/ERR and fails fast on ERR while still allowing the optional version-selection interaction.

Copilot uses AI. Check for mistakes.

// Verify the AppHost project does not end up with a version-pinned Redis PackageReference.
{
Expand Down
4 changes: 2 additions & 2 deletions tests/Aspire.Cli.EndToEnd.Tests/DoctorCommandTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ public async Task DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted()
await auto.WaitUntilAsync(
s => s.ContainsText("dev-certs") && s.ContainsText("partially trusted"),
timeout: TimeSpan.FromSeconds(60), description: "doctor to complete with partial trust warning");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WaitForSuccessPromptFailFastAsync runs after the output-text WaitUntilAsync. If aspire doctor fails before emitting the expected text, the test will still wait the full WaitUntilAsync timeout. If you’re aiming to fail fast on [N ERR: prompts, consider adding ERR-prompt detection to the WaitUntilAsync predicate (or using a helper that combines both checks).

Copilot uses AI. Check for mistakes.
await auto.TypeAsync("exit");
await auto.EnterAsync();

Expand Down Expand Up @@ -99,7 +99,7 @@ await auto.WaitUntilAsync(s =>

return s.ContainsText("certificate is trusted");
}, timeout: TimeSpan.FromSeconds(60), description: "doctor to complete with trusted certificate");
await auto.WaitForSuccessPromptAsync(counter);
await auto.WaitForSuccessPromptFailFastAsync(counter);
Copy link

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same pattern here: switching the prompt wait to fail-fast doesn’t help if WaitUntilAsync never becomes true due to an early command failure; it will still spin until the timeout. Consider detecting [N ERR: inside the WaitUntilAsync predicate (or using a wrapper helper) so failures surface immediately.

Copilot uses AI. Check for mistakes.
await auto.TypeAsync("exit");
await auto.EnterAsync();

Expand Down
Loading