Add comprehensive test coverage for CLI acquisition scripts#15995
Draft
Add comprehensive test coverage for CLI acquisition scripts#15995
Conversation
Add Aspire.Cli.Scripts.Tests project with functional tests for the 4 CLI acquisition scripts in eng/scripts/: - get-aspire-cli.sh (release bash script) - get-aspire-cli.ps1 (release PowerShell script) - get-aspire-cli-pr.sh (PR bash script) - get-aspire-cli-pr.ps1 (PR PowerShell script) Tests validate parameter parsing, help output, flag recognition, and error handling using --dry-run / -WhatIf modes. PR script tests use a mock gh CLI returning canned JSON responses. Integration tests query real GitHub PRs but are disabled in CI via ActiveIssue attribute. Test infrastructure: - ScriptToolCommand: extends ToolCommand for bash/pwsh script execution - TestEnvironment: provides isolated temp directories and mock gh CLI - RequiresGHCliAttribute: skips tests when gh CLI unavailable - RealGitHubPRFixture: discovers real PRs for integration tests 60 unit tests, all passing on macOS/Linux. PowerShell PR tests skipped on Windows due to WhatIf + mock gh interaction issues. Bash tests skipped on Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use Assert.Skip() instead of silent return in integration tests so test reports show proper skip status when GH_TOKEN is unavailable - Capture stdout separately in RealGitHubPRFixture to prevent gh CLI stderr warnings (upgrade notices, rate limits) from breaking JSON parsing - Make mock gh script fail (exit 1) for unhandled commands instead of silently succeeding, and add handlers for --version and api subcommands Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement unit-level testing of individual bash/PowerShell functions in the CLI acquisition scripts, enabling fast, isolated verification of script logic without requiring full end-to-end execution. New infrastructure: - ScriptFunctionCommand: Sources a script and calls individual functions via temp wrapper scripts. For bash, uses BASH_SOURCE guard. For PowerShell, strips main execution block and Help check to load only function definitions. - FakeArchiveHelper: Creates fake tar.gz/zip archives with SHA-512 checksums for testing validate_checksum and install_archive functions. - EnhancedMockGhHelper: URL-pattern-aware mock gh CLI with error injection support via environment variables. New test files: - SourceabilityTests: 6 tests verifying scripts can be sourced without executing their main flow. - ReleaseScriptFunctionTests: 17 tests for bash release script functions (construct_aspire_cli_url, map_quality_to_channel, validate_checksum, etc.) - PRScriptFunctionTests: 14 tests for bash PR script functions (get_runtime_identifier, get_cli_architecture_from_architecture, etc.) Script changes: - Wrap main execution in bash scripts with main() function and BASH_SOURCE guard to enable sourcing without side effects. Other changes: - Add PowerShell function-level parity tests in ReleaseScriptPowerShellTests for ConvertTo-ChannelName and Get-AspireCliUrl. - Replace ActiveIssue with OuterloopTest on PRScriptIntegrationTests since they require real GitHub API access. - Add using to Process in TestEnvironment.cs to prevent resource leak. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Make bash wrapper idempotent: save/restore shell options and guard readonly redeclarations so sourced scripts don't leak state - Fix PowerShell path escaping for paths containing apostrophes - Fix RealGitHubPRFixture: query microsoft/aspire (matching script default) instead of dotnet/aspire - Remove dead code: EnhancedMockGhHelper.cs (238 lines, never referenced) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add output validation assertions to existing tests that previously only checked exit codes. Tests now verify: - Dry-run output contains [DRY RUN] markers (bash) or What if (PowerShell) - Error messages contain relevant keywords (quality, PR number, etc.) - Flag-specific output (--skip-path, --hive-only, --skip-extension, etc.) - Custom paths and parameter values appear in output - Verbose mode produces substantial output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cripts New tests cover: - Version/quality mutual exclusion (bash + PowerShell) - Default install path verification ($HOME/.aspire/bin) - GitHub Actions GITHUB_PATH integration - Dev quality URL correctness (distinguishes from staging/release) - Version-specific URL output - install-extension gating for staging quality (bash + PowerShell parity) - PR number validation: non-numeric, zero, negative, option-as-first-arg - Unknown flags for PR script - ASPIRE_REPO env var override - Artifact name patterns (cli-native-archives) - NuGet hive path construction - --hive-only skipping CLI download (asserts specific message) - Platform detection (detect_os, detect_architecture) for both scripts - Architecture normalization with case sensitivity - Zip archive extraction (complements existing tar.gz test) - Extension URL construction (quality-based and version-based) - Default install path for PR script (PowerShell) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…p-archive, and PR artifact selection Add 12 new test methods across two test files: ReleaseScriptFunctionTests.cs: - add_to_shell_profile: Tests PATH modification for bash/zsh profiles, duplicate detection, and dry-run mode - validate_content_type: Tests HTML error page detection and valid binary content-type acceptance PRScriptFunctionTests.cs: - remove_temp_dir: Tests --keep-archive flag preserves temp directory, cleanup when disabled, and dry-run behavior - archive file discovery: Tests correct archive selection by OS/arch, missing archive error handling, and multiple archive disambiguation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15995Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15995" |
Move ProcessExtensions to the shared Aspire.TestUtilities project, removing duplicate copies from Aspire.Acquisition.Tests and Infrastructure.Tests (PowerShellCommand.cs).
Move SkipOnPlatform(Windows) from class level to individual tests that use mock gh CLI with -WhatIf. Pure help and parameter validation tests now run on Windows too. Also enhance the Windows mock gh.cmd to handle 'run download' with -D argument parsing, matching the Unix mock behavior.
Restructure Windows batch mock gh.cmd to use top-level goto dispatch instead of exit /b inside nested if() blocks. CMD's exit /b is unreliable inside parenthesized blocks and can fall through to the error handler, producing stderr output that breaks PowerShell's output capture when combined with WhatIf/ShouldProcess. Changes: - Rewrite gh.cmd to use goto :ver/:api/:pr/:run labels at top level - Use %~1 to strip surrounding quotes from arguments - Add setlocal for clean variable scope - Remove all 16 [SkipOnPlatform(Windows)] attributes from tests - Remove unused using Microsoft.DotNet.XUnitExtensions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When PowerShell invokes a .cmd file with arguments containing '&' (e.g. API URLs like '?event=pull_request&head_sha=abc'), CMD interprets '&' as a command separator and executes the text after it as a separate command, producing spurious stderr output. Using 'exit' (without /b) terminates the entire CMD process spawned by PowerShell, preventing the phantom second command from running. Also restructure :pr handler to use goto :pr_list instead of exit /b inside if() block. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add RunOnGithubActionsMacOS=true to run tests on macOS CI - Simplify System.Environment.GetEnvironmentVariable to Environment.GetEnvironmentVariable (using directive already present) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
Contributor
|
🎬 CLI E2E Test Recordings — 55 recordings uploaded (commit View recordings
📹 Recordings uploaded automatically from CI run #24176816448 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new test project (
Aspire.Acquisition.Scripts.Tests) with 160 tests covering the two CLI acquisition scripts (get-aspire-cli.shandget-aspire-cli-pr.sh). The goal is confidence to refactor these scripts — any incorrect change should break at least one test.What's tested
Release script (
get-aspire-cli.sh)detect_os,detect_architecture, arch normalizationGITHUB_PATHintegrationadd_to_shell_profilefor bash/zsh, duplicate detection, dry-runPR script (
get-aspire-cli-pr.sh)detect_os,detect_architectureremove_temp_dirwith keep-archive and dry-run modesASPIRE_REPOoverride, artifact name prefixes, hive-only mode, unknown flagsPowerShell variants
Test infrastructure
ScriptFunctionCommand— sources a bash script and calls individual functions in isolation, enabling unit-level testing of internal helpersScriptToolCommand— runs the full script with arguments for end-to-end behavior testsTestEnvironment— creates per-test temp directories with mockHOME, mockcurl/ghscripts, andASPIRE_TEST_MODE=trueFakeArchiveHelper— generates fake.tar.gz/.ziparchives with.sha512sidecar filesghCLI handles--version,pr list,run list,run view,api, andrun downloadcommandsFiles changed
tests/Aspire.Acquisition.Scripts.Tests/(~3,400 lines)Aspire.slnxeng/testing/github-ci-trigger-patterns.txtNo changes to the scripts themselves.