Add tracking, project exclusion, and execution summary to GCP sizing#108
Add tracking, project exclusion, and execution summary to GCP sizing#108visharma28 wants to merge 2 commits into
Conversation
…n GCP sizing script - Add null check for \ to skip instance processing when API calls fail - Add try-catch wrapper around entire instance processing loop to handle null reference errors - Add try-catch blocks around individual disk retrieval operations with graceful error handling - Add null check for \ to skip failed disk operations - Add null check for \ to skip disk processing when API calls fail - Fix bug where \.Status should be \.Status (line 333) This prevents the script from crashing when encountering API errors, permission issues, or null disk sources, allowing it to continue processing other projects and instances.
… summary to GCP sizing script - Add ExcludeProjects parameter to filter out projects matching wildcard patterns (e.g., 'sys-*,test-*,*-sandbox') - Add comprehensive counters system tracking projects, instances, and disks (successful, failed, API disabled, permission denied) - Add Show-Progress function for enhanced progress reporting with real-time statistics every 50 projects - Add Show-ExecutionSummary function to display detailed execution statistics at script completion - Add project exclusion logic with pattern matching and per-pattern statistics - Add enhanced error categorization distinguishing between API disabled, permission denied, and other errors - Add execution timing tracking (start time, end time, total duration) - Add success rate calculations for projects, instances, and disks - Add running progress updates showing processed counts and success rates This builds on PR #1 (null checks) to provide comprehensive visibility into script execution, performance optimization through project filtering, and detailed success/failure analytics.
visharma28
force-pushed
the
gcp-sizing-enhanced-features
branch
from
19ec13c to
45cc8f5
Compare
| GitHub: stevenctong | ||
| Date: 11/9/21 | ||
| Updated: 2/24/22 | ||
| Updated: 11/14/25 |
There was a problem hiding this comment.
Do we need duplicate update, instead can we override the above line itself
| $instanceInfo = Get-GceInstance -Project $($project.ProjectId) | ||
| try { | ||
| # Display progress with counters | ||
| Show-Progress -Activity "Processing project: $($project.ProjectId)" -Current $projectCounter -Total $counters.ProjectsTotal -Counters $counters |
There was a problem hiding this comment.
$counters.ProjectsTotal includes the excluded projets, but we are looping the projects after exclusion. this might cause incorrect progress. Instead we can use $projectList.Count, that gives the total number we are processing
| Write-Progress -ID 3 -Activity "Processing disk: $($disk.Name)" -Completed | ||
|
|
||
| # Mark project as successful if we got this far | ||
| $counters.ProjectsSuccessful++ |
There was a problem hiding this comment.
Can we recheck the placement, coz, we are marking the project as succesful, even if API calls or anyother exceptions that is caught and skipped in the previous try catches, for example line 400 catches API errors, that will also, account as project successful
| # Track project-level processing | ||
| $projectProcessedSuccessfully = $false | ||
| $projectInstanceCount = 0 | ||
| $projectDiskCount = 0 |
| # Calculate rates based on total discovered projects | ||
| $projectsProcessed = $Counters.ProjectsTotal - $Counters.ProjectsExcluded | ||
| $projectSuccessRate = if ($projectsProcessed -gt 0) { [math]::Round(($Counters.ProjectsSuccessful / $projectsProcessed) * 100, 1) } else { 0 } | ||
| $projectFailureRate = if ($projectsProcessed -gt 0) { [math]::Round((($Counters.ProjectsFailed + $Counters.ProjectsApiDisabled + $Counters.ProjectsPermissionDenied + $Counters.ProjectsOtherErrors) / $projectsProcessed) * 100, 1) } else { 0 } |
There was a problem hiding this comment.
NIT: after making changes to get the right successful project count, lets check these lines also, if this causing any regression,because of the new change we making
|
@visharma28 I just finished troubleshooting similar issues from the end user side. One thing you missed that caused my runs to fail is that when projects are deleted, they go into a pending deleition status. I would suggest adding a filter to ensure that you only retrieve active projects ( |
|
This has merge conflicts with master. Could you rebase and resolve? Happy to review once updated. |
4 participants
Summary:
This PR adds comprehensive execution tracking, project filtering capabilities, and detailed execution summaries to the GCP sizing script, building on top of PR #107 (null checks).
What?
ExcludeProjectsparameter to filter out projects matching wildcard patterns (comma-separated, e.g., "sys-,test-,*-sandbox")Show-Progressfunction that displays real-time progress with success rates and running totals every 50 projectsShow-ExecutionSummaryfunction that displays detailed execution statistics at script completionWhy?
How?
$countershashtable with 15 counters initialized at script start: ProjectsTotal, ProjectsExcluded, ProjectsSuccessful, ProjectsFailed, ProjectsApiDisabled, ProjectsPermissionDenied, ProjectsOtherErrors, InstancesTotal, InstancesSuccessful, InstancesFailed, DisksTotal, DisksAttachedTotal, DisksUnattachedTotal, DisksSuccessful, DisksFailedShow-Progressfunction that calculates percent complete and success rates, displays Write-Progress bar, and prints summary every 50 projects or at completionShow-ExecutionSummaryfunction that calculates execution time, success rates, and displays formatted summary with color-coded output (Green for success, Red for failures, Yellow for warnings, Cyan for section headers)ExcludeProjectsparameter validation and parsing logic that splits comma-separated patterns, trims whitespace, and matches against project IDs using PowerShell-likeoperatorGet-GceInstancetry-catch block to categorize errors using regex pattern matching on exception messages (API disabled, permission denied, other)$scriptStartTimevariable captured at script initialization and passed toShow-ExecutionSummaryfor duration calculation[math]::Round()for percentage calculations with 1 decimal place precisionShow-Progressat the start of each project processing iteration with current project counter and total project countShow-ExecutionSummaryafter all processing completes but before final output file creationTest Plan:
[System.Management.Automation.Language.Parser]::ParseFile()ExcludeProjectsparameter accepts comma-separated wildcard patterns and correctly filters projectsShow-Progressdisplays progress bar and prints summary every 50 projects with accurate statisticsShow-ExecutionSummarydisplays comprehensive statistics including execution time, project counts, instance counts, disk counts, and success rateshttps://phabricator.rubrik.com/P19813
[scale run in progress]
JIRA Issues:
SIZENG-49
Revert Plan:
Revert the commit