diff --git a/README.md b/README.md index a72708d..90a6d96 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,9 @@ A single KQL query against the Resource Graph `resources` table returns all VMs, ### Step 2 — Fetch availability metrics -Azure Monitor is queried per-resource in parallel (configurable via `--parallelism`, default auto-scales to CPU cores) with retry on 429/5xx errors. Granularity is PT1M (one data point per minute). +By default, Azure Monitor is queried per-resource in parallel (configurable via `--parallelism`, default auto-scales to CPU cores) with retry on 429/5xx errors. Granularity is PT1M (one data point per minute). + +With `--batch` / `-Batch`, the tool uses the regional [Azure Monitor Metrics Batch API](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/migrate-to-batch-api) instead of individual per-resource calls. Resources are grouped by (subscription, region, kind) and sent in configurable chunks (`--batch-size` / `-BatchSize`, default 10, max 50). This reduces the total number of API calls and can significantly improve throughput for large inventories. The batch endpoint requires a separate token (`https://metrics.monitor.azure.com`) and the tool validates each regional endpoint before fetching. Wave-based processing with garbage collection between waves keeps memory usage bounded. | Resource type | Metrics requested | Native scale | Aggregation | |---|---|---|---| @@ -191,6 +193,8 @@ Storage Account `Availability` is a transaction-success-rate metric — it is on | `--resource` | `-r` | *(all)* | Filter to a single resource name | | `--parallelism` | `-p` | *(auto)* | Max concurrent API calls (scales to CPU cores) | | `--activity-grace-minutes` | `-g` | `10` | Post-operation grace window for supported Activity Log lifecycle events | +| `--batch` | `-b` | off | Use the regional Metrics Batch API instead of per-resource calls | +| `--batch-size` | | `10` | Max resources per batch call (1–50); implies `--batch` | | `--version` | `-v` | | Print version and exit | ### PowerShell (`get-availability.ps1`) @@ -203,6 +207,8 @@ Storage Account `Availability` is a transaction-success-rate metric — it is on | `-Resource` | *(all)* | Filter to a single resource name | | `-Parallelism` | *(auto)* | Max concurrent API calls (scales to CPU cores, 4–16) | | `-ActivityGraceMinutes` | `10` | Post-operation grace window for supported Activity Log lifecycle events | +| `-Batch` | off | Use the regional Metrics Batch API instead of per-resource calls | +| `-BatchSize` | `10` | Max resources per batch call (1–50); implies `-Batch` | | `-Version` | | Print version and exit | Both versions enforce the same constraints: `-Month` / `--month` cannot point to a month whose first day is more than 90 days before the current UTC time. @@ -258,6 +264,12 @@ dotnet publish -c Release -r win-x64 # output in bin/Release/net10.0/win-x64/p # Override the Activity Log grace window ./GetAvailability --subscriptions Contoso-Development --month 202603 --resource myvm02 --activity-grace-minutes 15 +# Use batch API mode +./GetAvailability --subscriptions Contoso-Production --month 202603 --batch + +# Use batch API with custom batch size +./GetAvailability --subscriptions Contoso-Production Contoso-Development --month 202603 --batch-size 20 + # Or run directly without publishing cd csharp/GetAvailability dotnet run -- --subscriptions Contoso-Production --month 202603 @@ -281,6 +293,12 @@ dotnet run -- --subscriptions Contoso-Production --month 202603 # Override the Activity Log grace window ./get-availability.ps1 -Subscriptions 'Contoso-Development' -Month 202603 -Resource myvm02 -ActivityGraceMinutes 15 +# Use batch API mode +./get-availability.ps1 -Subscriptions 'Contoso-Production' -Month 202603 -Batch + +# Use batch API with custom batch size +./get-availability.ps1 -Subscriptions 'Contoso-Production','Contoso-Development' -Month 202603 -Batch -BatchSize 20 + # Pipe results to CSV ./get-availability.ps1 -Subscriptions 'Contoso-Production' -Month 202603 | Export-Csv availability.csv ``` @@ -349,6 +367,7 @@ Console output also reports per-resource classification details: - **Conservative on failure.** If a Resource Health API call fails for a period where Health History should exist, Activity Log matches still apply, but no remaining suspect minutes are excused through Health History. - **Transaction-aware storage.** Zero-transaction minutes are excluded from eligibility rather than counted as unavailable, giving an accurate picture of actual storage service availability. - **Whole-window no-signal exclusion.** If the metric API returns no usable datapoints across the full period, the resource is excluded from availability calculations and shown as `N/A`. +- **Batch API mode.** `--batch` / `-Batch` uses the regional Azure Monitor Metrics Batch API, grouping resources by (subscription, region, kind) and sending them in configurable chunks. This reduces API call count and improves throughput for large inventories. Regional endpoints are validated before fetching. Wave-based processing with GC between waves bounds memory usage. - **`Parallel.ForEachAsync`** (C#) / **`ForEach-Object -Parallel`** (PowerShell) for concurrent metric and health queries with configurable parallelism. - **Ticks-based metric keying** (`long` instead of `DateTime`) — zero-allocation per data point in both versions. - **`System.Text.Json`** for efficient JSON parsing in both versions — avoids large PSObject trees in PowerShell and enables AOT-safe parsing in C#. diff --git a/csharp/GetAvailability/Program.cs b/csharp/GetAvailability/Program.cs index b68a5a7..8d4662e 100644 --- a/csharp/GetAvailability/Program.cs +++ b/csharp/GetAvailability/Program.cs @@ -41,8 +41,10 @@ string[] kinds = ["vm", "sql", "storage"]; string? resourceName = null; string? monthParameter = null; -int parallelism = Math.Max(4, Math.Min(16, Environment.ProcessorCount)); // auto-scale to CPU cores +int parallelism = Math.Clamp(Environment.ProcessorCount, 4, 16); int activityGraceMinutes = 10; +bool useBatch = false; +int batchSize = 10; for (int i = 0; i < args.Length; i++) { @@ -66,6 +68,14 @@ case "--activity-grace-minutes" or "-g": activityGraceMinutes = ParseIntOption(ReadRequiredValue(args, ref i, "--activity-grace-minutes"), "--activity-grace-minutes", minValue: 0); break; + case "--batch" or "-b": + useBatch = true; + break; + case "--batch-size": + batchSize = ParseIntOption(ReadRequiredValue(args, ref i, "--batch-size"), "--batch-size", minValue: 1); + if (batchSize > 50) throw new ArgumentException("--batch-size must be <= 50."); + useBatch = true; + break; case "--version" or "-v": Console.WriteLine($"GetAvailability {typeof(Program).Assembly.GetName().Version?.ToString(3) ?? "0.0.0"}"); return 0; @@ -77,6 +87,8 @@ Console.WriteLine(" --resource, -r Filter to a single resource name."); Console.WriteLine(" --parallelism, -p Max concurrent metric calls (default: auto)."); Console.WriteLine(" --activity-grace-minutes, -g Post-operation grace window for supported Activity Log lifecycle events (default: 10)."); + Console.WriteLine(" --batch, -b Use the regional Metrics Batch API instead of per-resource calls."); + Console.WriteLine(" --batch-size Max resources per batch call (default: 10, max: 50). Implies --batch."); Console.WriteLine(" --version, -v Print version and exit."); return 0; } @@ -96,7 +108,7 @@ try { - await RunAsync(subscriptionNames, kinds, resourceName, monthParameter, parallelism, activityGraceMinutes); + await RunAsync(subscriptionNames, kinds, resourceName, monthParameter, parallelism, activityGraceMinutes, useBatch, batchSize); return 0; } catch (Exception ex) when (ex is AuthenticationFailedException or CredentialUnavailableException) @@ -118,7 +130,9 @@ static async Task RunAsync( string? resourceName, string monthParameter, int parallelism, - int activityGraceMinutes) + int activityGraceMinutes, + bool useBatch, + int batchSize) { var sw = Stopwatch.StartNew(); @@ -179,8 +193,17 @@ static async Task RunAsync( }; } - // Step 4: Fetch Azure Monitor metrics per resource in parallel - var metricResults = await MetricsService.QueryAsync(metricsClient, resources, utcStart, utcEnd, parallelism); + // Step 4: Fetch Azure Monitor metrics (batch or per-resource) + ConcurrentDictionary metricResults; + if (useBatch) + { + Console.WriteLine($"Using Batch API (batch size: {batchSize})."); + metricResults = await BatchMetricsService.QueryAsync(credential, resources, utcStart, utcEnd, parallelism, batchSize); + } + else + { + metricResults = await MetricsService.QueryAsync(metricsClient, resources, utcStart, utcEnd, parallelism); + } // Step 5: For resources with suspect metric minutes, investigate null/0% suspect minutes and // positive degraded datapoints. Activity Log is checked first for supported lifecycle actions, @@ -315,18 +338,16 @@ static async Task RunAsync( } } - int confirmedHealthDowntimeMinutes = suspectResults is not null && suspectResults.TryGetValue(key, out var healthClassification) - ? healthClassification.PlatformFaultGapMinutes + healthClassification.HealthConfirmedDegradedMinutes - : 0; - - int unexplainedPositiveDegradedMinutes = suspectResults is not null && suspectResults.TryGetValue(key, out var unresolvedClassification) - ? Math.Max(0, mr.DegradedMinutes - unresolvedClassification.CustomerExcusedDegradedMinutes - unresolvedClassification.HealthConfirmedDegradedMinutes) - : mr.DegradedMinutes; + int confirmedHealthDowntimeMinutes = 0; + int unexplainedPositiveDegradedMinutes = mr.DegradedMinutes; + int unexplainedSuspectMinutes = mr.SuspectMinutes; - int unexplainedSuspectMinutes = suspectResults is not null && suspectResults.TryGetValue(key, out var unexplainedClassification) - ? unexplainedClassification.UnresolvedZeroDowntimeMinutes - + unexplainedPositiveDegradedMinutes - : mr.SuspectMinutes; + if (suspectResults is not null && suspectResults.TryGetValue(key, out var cls)) + { + confirmedHealthDowntimeMinutes = cls.PlatformFaultGapMinutes + cls.HealthConfirmedDegradedMinutes; + unexplainedPositiveDegradedMinutes = Math.Max(0, mr.DegradedMinutes - cls.CustomerExcusedDegradedMinutes - cls.HealthConfirmedDegradedMinutes); + unexplainedSuspectMinutes = cls.UnresolvedZeroDowntimeMinutes + unexplainedPositiveDegradedMinutes; + } // For Storage Accounts, zero-transaction minutes have no availability signal and are // excluded from eligibility. They count as suspect (no data) and excused (nothing to measure). @@ -360,8 +381,8 @@ static async Task RunAsync( $" [{res.Name}] eligible min = {totalMinutes} - {string.Join(" - ", eligibilityAdjustments)} = {elig.EligibleMinutes}"); } - double customerExcusedAvail = suspectResults is not null && suspectResults.TryGetValue(key, out var availClass) - ? availClass.CustomerExcusedDegradedAvailableSum + double customerExcusedAvail = suspectResults is not null && suspectResults.TryGetValue(key, out var ac) + ? ac.CustomerExcusedDegradedAvailableSum : 0; elig.AvailableMinutes = Math.Round(Math.Max(0, mr.AvailableSum - customerExcusedAvail), 2); } @@ -452,7 +473,16 @@ static string[] ReadRequiredValues(string[] args, ref int index, string optionNa { var values = new List(); while (index + 1 < args.Length && !args[index + 1].StartsWith("-", StringComparison.Ordinal)) - values.Add(args[++index]); + { + // Support comma-separated values: "A, B, C" or "A,B,C" or mixed + var raw = args[++index]; + foreach (var part in raw.Split(',')) + { + var trimmed = part.Trim(); + if (trimmed.Length > 0) + values.Add(trimmed); + } + } return values.Count > 0 ? values.ToArray() diff --git a/csharp/GetAvailability/Services/ActivityLogService.cs b/csharp/GetAvailability/Services/ActivityLogService.cs index b21503b..3fb9e5f 100644 --- a/csharp/GetAvailability/Services/ActivityLogService.cs +++ b/csharp/GetAvailability/Services/ActivityLogService.cs @@ -259,12 +259,17 @@ private static string BuildActivityGroupKey(ActivityLogEvent evt) private static DateTimeOffset ExtendActivityInterval(int graceMinutes, DateTimeOffset currentEnd) => graceMinutes > 0 ? currentEnd.AddMinutes(graceMinutes) : currentEnd; + /// + /// Parses Azure timestamp strings which may use several formats depending on + /// region and API version. Falls back to DateTimeOffset.TryParse for formats + /// not in the explicit list. + /// private static DateTimeOffset? ParseAzureTimestamp(string? timestamp) { if (string.IsNullOrWhiteSpace(timestamp)) return null; - string[] formats = + ReadOnlySpan formats = [ "MM/dd/yyyy HH:mm:ss", "M/d/yyyy H:mm:ss", @@ -274,7 +279,7 @@ private static DateTimeOffset ExtendActivityInterval(int graceMinutes, DateTimeO "yyyy-MM-ddTHH:mm:ss.fffffffZ", ]; - foreach (string format in formats) + foreach (var format in formats) { if (DateTimeOffset.TryParseExact( timestamp, @@ -296,9 +301,11 @@ private static DateTimeOffset ExtendActivityInterval(int graceMinutes, DateTimeO : null; } + /// Truncates to minute boundary for interval grouping. private static DateTimeOffset TruncateToMinute(DateTimeOffset value) => new(value.Year, value.Month, value.Day, value.Hour, value.Minute, 0, TimeSpan.Zero); + /// Reads a nested string property (e.g. operationName.value) from a JsonElement. private static string GetNestedPropString(JsonElement parent, string propertyName, string nestedName) { if (!parent.TryGetProperty(propertyName, out var obj) || obj.ValueKind != JsonValueKind.Object) @@ -306,6 +313,7 @@ private static string GetNestedPropString(JsonElement parent, string propertyNam return GetPropString(obj, nestedName); } + /// Reads a string property from a JsonElement, returning empty if absent or null. private static string GetPropString(JsonElement props, string name) => props.TryGetProperty(name, out var el) ? el.GetString() ?? "" : ""; } diff --git a/csharp/GetAvailability/Services/BatchMetricsService.cs b/csharp/GetAvailability/Services/BatchMetricsService.cs new file mode 100644 index 0000000..fefe83b --- /dev/null +++ b/csharp/GetAvailability/Services/BatchMetricsService.cs @@ -0,0 +1,416 @@ +using Azure.Core; +using GetAvailability.Models; +using System.Collections.Concurrent; +using System.Net.Http.Headers; +using System.Text.Json; + +namespace GetAvailability.Services; + +/// +/// Fetches Azure Monitor metrics using the regional Batch API (metrics:getBatch) instead of +/// per-resource calls. Groups resources by (subscription, region, kind) and sends batched +/// POST requests to reduce total HTTP calls and throttling risk. +/// +/// Batch API constraints: +/// - All resources in a batch must share subscription, region, and resource type. +/// - Max 50 resource IDs per batch call (default 10 for memory safety). +/// - Endpoint: https://{region}.metrics.monitor.azure.com +/// - Auth scope: https://metrics.monitor.azure.com/.default +/// +public static class BatchMetricsService +{ + private static readonly Dictionary KindConfigs = new(StringComparer.OrdinalIgnoreCase) + { + ["VirtualMachine"] = new("Microsoft.Compute/virtualMachines", "VmAvailabilityMetric", "Minimum"), + ["AzureSqlDatabase"] = new("Microsoft.Sql/servers/databases", "Availability", "Minimum"), + ["StorageAccount"] = new("Microsoft.Storage/storageAccounts", "Availability,Transactions", "Minimum,Total"), + }; + + /// + /// Queries metrics for all resources using the batch API. Returns a dictionary keyed by + /// lowercase resource ID, same shape as MetricsService.QueryAsync for drop-in replacement. + /// + public static async Task> QueryAsync( + TokenCredential credential, + IReadOnlyList resources, + DateTimeOffset startDate, + DateTimeOffset endDate, + int parallelism, + int batchSize = 10) + { + int total = resources.Count; + + // Acquire metrics-scoped token + var tokenResponse = await credential.GetTokenAsync( + new TokenRequestContext(["https://metrics.monitor.azure.com/.default"]), default); + + using var http = new HttpClient(); + http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", tokenResponse.Token); + http.Timeout = TimeSpan.FromMinutes(5); + + // Group resources by (subscriptionId, location, kind) + var groups = new Dictionary>(StringComparer.OrdinalIgnoreCase); + foreach (var res in resources) + { + string key = $"{res.SubscriptionId}|{res.Location.ToLowerInvariant()}|{res.Kind}"; + if (!groups.TryGetValue(key, out var list)) + { + list = []; + groups[key] = list; + } + list.Add(res); + } + + // Build batch work items (chunks of batchSize) + var workItems = new List(); + foreach (var (key, resList) in groups) + { + var parts = key.Split('|'); + string subId = parts[0], location = parts[1], kind = parts[2]; + if (!KindConfigs.TryGetValue(kind, out var config)) + { + Console.Error.WriteLine($" WARNING: No batch config for kind '{kind}', skipping."); + continue; + } + + for (int i = 0; i < resList.Count; i += batchSize) + { + var chunk = resList.GetRange(i, Math.Min(batchSize, resList.Count - i)); + workItems.Add(new BatchWorkItem(subId, location, kind, config, chunk)); + } + } + + // Print grouping summary + int uniqueRegions = groups.Keys + .Select(k => k.Split('|')[1]) + .Distinct(StringComparer.OrdinalIgnoreCase) + .Count(); + Console.WriteLine($"Grouped {total} resource(s) into {workItems.Count} batch(es) across {uniqueRegions} region(s) (max {batchSize} per batch)."); + + foreach (var (key, resList) in groups.OrderBy(g => g.Key)) + { + var parts = key.Split('|'); + string subName = resList[0].SubscriptionName; + string kind = ShortKind(parts[2]); + string location = parts[1]; + int chunks = (int)Math.Ceiling((double)resList.Count / batchSize); + Console.WriteLine($" {location} / {kind} / {Truncate(subName, 30)} : {resList.Count} resource(s) -> {chunks} batch(es)"); + } + Console.WriteLine(); + + string startIso = startDate.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffZ"); + string endIso = endDate.ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffZ"); + + var results = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); + int done = 0; + int totalBatches = workItems.Count; + + await Parallel.ForEachAsync(workItems, + new ParallelOptions { MaxDegreeOfParallelism = parallelism }, + async (workItem, ct) => + { + await ProcessBatchWorkItemAsync(http, workItem, startIso, endIso, results, ct); + + int current = Interlocked.Increment(ref done); + if (current % 5 == 0 || current == totalBatches) + Console.Write($"\r Fetching batch metrics: {current}/{totalBatches} batches done"); + }); + + Console.WriteLine($"\r Batch queries completed. Received results for {results.Count} / {total} resource(s). "); + return results; + } + + private static async Task ProcessBatchWorkItemAsync( + HttpClient http, + BatchWorkItem workItem, + string startIso, + string endIso, + ConcurrentDictionary results, + CancellationToken ct) + { + bool isVm = workItem.Kind == "VirtualMachine"; + bool isStorage = workItem.Kind == "StorageAccount"; + + var resourceIds = workItem.Resources.Select(r => r.ResourceId).ToArray(); + + string uri = $"https://{workItem.Location}.metrics.monitor.azure.com" + + $"/subscriptions/{workItem.SubscriptionId}/metrics:getBatch" + + $"?starttime={Uri.EscapeDataString(startIso)}" + + $"&endtime={Uri.EscapeDataString(endIso)}" + + $"&interval=PT1M" + + $"&metricnamespace={Uri.EscapeDataString(workItem.Config.Namespace)}" + + $"&metricnames={Uri.EscapeDataString(workItem.Config.MetricNames)}" + + $"&aggregation={Uri.EscapeDataString(workItem.Config.Aggregation)}" + + $"&api-version=2023-10-01"; + + string bodyJson = BuildResourceIdsJson(resourceIds); + + // Build resource lookup + var resById = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var r in workItem.Resources) + resById[r.ResourceId.ToLowerInvariant()] = r; + + JsonDocument? doc = null; + + for (int attempt = 1; attempt <= 5; attempt++) + { + try + { + using var request = new HttpRequestMessage(HttpMethod.Post, uri) + { + Content = new StringContent(bodyJson, System.Text.Encoding.UTF8, "application/json") + }; + + using var response = await http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, ct); + int statusCode = (int)response.StatusCode; + + if (statusCode >= 200 && statusCode < 300) + { + await using var stream = await response.Content.ReadAsStreamAsync(ct); + doc = await JsonDocument.ParseAsync(stream, cancellationToken: ct); + break; + } + + if ((statusCode == 401 || statusCode == 429 || statusCode >= 500) && attempt < 5) + { + await Task.Delay(TimeSpan.FromSeconds(Math.Min(30, 1 << attempt)), ct); + continue; + } + + string names = string.Join(", ", workItem.Resources.Select(r => r.Name)); + Console.Error.WriteLine($" WARNING: Batch metric query failed for [{names}]: HTTP {statusCode}"); + EmitExcludedResults(workItem.Resources, results); + return; + } + catch (Exception ex) when (attempt < 5 && + (ex.Message.Contains("transport", StringComparison.OrdinalIgnoreCase) || + ex.Message.Contains("connection", StringComparison.OrdinalIgnoreCase) || + ex.Message.Contains("timed out", StringComparison.OrdinalIgnoreCase))) + { + await Task.Delay(TimeSpan.FromSeconds(Math.Min(30, 1 << attempt)), ct); + } + catch (Exception ex) + { + string names = string.Join(", ", workItem.Resources.Select(r => r.Name)); + Console.Error.WriteLine($" WARNING: Batch metric query failed for [{names}]: {ex.Message}"); + EmitExcludedResults(workItem.Resources, results); + return; + } + } + + if (doc is null) + { + EmitExcludedResults(workItem.Resources, results); + return; + } + + try + { + if (!doc.RootElement.TryGetProperty("values", out var valuesArray)) + { + EmitExcludedResults(workItem.Resources, results); + return; + } + + foreach (var resourceEntry in valuesArray.EnumerateArray()) + { + string resId = resourceEntry.GetProperty("resourceid").GetString()!; + string resIdLower = resId.ToLowerInvariant(); + + if (!resourceEntry.TryGetProperty("value", out var metricValueArr)) + { + results[resIdLower] = default; + continue; + } + + if (isStorage) + results[resIdLower] = ProcessStorageBatch(metricValueArr); + else + results[resIdLower] = ProcessVmOrSqlBatch(metricValueArr, isVm); + } + } + finally + { + doc.Dispose(); + } + } + + private static MetricScalars ProcessVmOrSqlBatch(JsonElement metricValueArr, bool isVm) + { + double availSum = 0; + int numericPoints = 0; + var nullTicks = new List(); + var zeroTicks = new List(); + var degraded = new List(); + + foreach (var metricEl in metricValueArr.EnumerateArray()) + { + string mName = metricEl.GetProperty("name").GetProperty("value").GetString()!; + bool isPrimary = mName.Equals("VmAvailabilityMetric", StringComparison.OrdinalIgnoreCase) + || mName.Equals("Availability", StringComparison.OrdinalIgnoreCase); + if (!isPrimary) continue; + + foreach (var tsEl in metricEl.GetProperty("timeseries").EnumerateArray()) + foreach (var dp in tsEl.GetProperty("data").EnumerateArray()) + { + long ticks = DateTime.Parse(dp.GetProperty("timeStamp").GetString()!).ToUniversalTime().Ticks; + double? minVal = TryGetDouble(dp, "minimum"); + + if (minVal.HasValue) + { + numericPoints++; + double v = isVm ? minVal.Value : minVal.Value / 100.0; + if (v == 0.0) + { + zeroTicks.Add(ticks); + } + else + { + availSum += v; + if (v < 1.0) + degraded.Add(new MetricValueSample(ticks, v)); + } + } + else + { + nullTicks.Add(ticks); + } + } + } + + int gapMinutes = nullTicks.Count + zeroTicks.Count; + int degradedMinutes = degraded.Count; + bool exclude = numericPoints == 0 && nullTicks.Count == 0 && zeroTicks.Count == 0 && degradedMinutes == 0; + + return new MetricScalars( + availSum, gapMinutes, 0, exclude, + nullTicks.Count > 0 ? nullTicks.ToArray() : null, + zeroTicks.Count > 0 ? zeroTicks.ToArray() : null, + degradedMinutes, + degraded.Count > 0 ? degraded.ToArray() : null); + } + + private static MetricScalars ProcessStorageBatch(JsonElement metricValueArr) + { + double availSum = 0; + int zeroTxMin = 0; + int numericPoints = 0; + var nullTicks = new List(); + var zeroTicks = new List(); + var degraded = new List(); + + // Build Transactions lookup + var txByTicks = new Dictionary(); + foreach (var metricEl in metricValueArr.EnumerateArray()) + { + string mName = metricEl.GetProperty("name").GetProperty("value").GetString()!; + if (!mName.Equals("Transactions", StringComparison.OrdinalIgnoreCase)) continue; + foreach (var tsEl in metricEl.GetProperty("timeseries").EnumerateArray()) + foreach (var dp in tsEl.GetProperty("data").EnumerateArray()) + { + long ticks = DateTime.Parse(dp.GetProperty("timeStamp").GetString()!).ToUniversalTime().Ticks; + double? tot = TryGetDouble(dp, "total"); + if (tot.HasValue) + txByTicks[ticks] = tot.Value; + } + } + + // Process Availability + foreach (var metricEl in metricValueArr.EnumerateArray()) + { + string mName = metricEl.GetProperty("name").GetProperty("value").GetString()!; + if (!mName.Equals("Availability", StringComparison.OrdinalIgnoreCase)) continue; + foreach (var tsEl in metricEl.GetProperty("timeseries").EnumerateArray()) + foreach (var dp in tsEl.GetProperty("data").EnumerateArray()) + { + long ticks = DateTime.Parse(dp.GetProperty("timeStamp").GetString()!).ToUniversalTime().Ticks; + bool hasTx = txByTicks.TryGetValue(ticks, out double txVal) && txVal > 0; + double? minVal = TryGetDouble(dp, "minimum"); + + if (hasTx && minVal.HasValue) + { + double norm = minVal.Value / 100.0; + numericPoints++; + if (norm == 0.0) + { + zeroTicks.Add(ticks); + } + else + { + availSum += norm; + if (norm < 1.0) + degraded.Add(new MetricValueSample(ticks, norm)); + } + } + else if (hasTx && !minVal.HasValue) + { + nullTicks.Add(ticks); + } + else if (!hasTx) + { + zeroTxMin++; + } + } + } + + int gapMinutes = nullTicks.Count + zeroTicks.Count; + int degradedMinutes = degraded.Count; + bool exclude = numericPoints == 0 && nullTicks.Count == 0 && zeroTicks.Count == 0 && degradedMinutes == 0; + + return new MetricScalars( + availSum, gapMinutes, zeroTxMin, exclude, + nullTicks.Count > 0 ? nullTicks.ToArray() : null, + zeroTicks.Count > 0 ? zeroTicks.ToArray() : null, + degradedMinutes, + degraded.Count > 0 ? degraded.ToArray() : null); + } + + /// Tries to read a double from a JSON data-point element. Returns null if the property is absent or not a number. + private static double? TryGetDouble(JsonElement element, string propertyName) + { + if (element.TryGetProperty(propertyName, out var prop) && prop.ValueKind == JsonValueKind.Number) + return prop.GetDouble(); + return null; + } + + private static void EmitExcludedResults( + List resources, + ConcurrentDictionary results) + { + foreach (var r in resources) + results[r.ResourceId.ToLowerInvariant()] = new MetricScalars(0, 0, 0, ExcludeFromAvailability: true); + } + + private static string ShortKind(string kind) => kind switch + { + "VirtualMachine" => "VM", + "AzureSqlDatabase" => "SQL", + "StorageAccount" => "Storage", + _ => kind, + }; + + private static string Truncate(string s, int max) => + s.Length <= max ? s : string.Concat(s.AsSpan(0, max - 3), "..."); + + /// + /// Builds the JSON body for the batch request without reflection (AOT-safe). + /// Produces: {"resourceids":["id1","id2",...]} + /// + private static string BuildResourceIdsJson(string[] resourceIds) + { + using var ms = new System.IO.MemoryStream(); + using (var writer = new Utf8JsonWriter(ms)) + { + writer.WriteStartObject(); + writer.WriteStartArray("resourceids"); + foreach (var id in resourceIds) + writer.WriteStringValue(id); + writer.WriteEndArray(); + writer.WriteEndObject(); + } + return System.Text.Encoding.UTF8.GetString(ms.ToArray()); + } + + private readonly record struct BatchMetricConfig(string Namespace, string MetricNames, string Aggregation); + private readonly record struct BatchWorkItem(string SubscriptionId, string Location, string Kind, BatchMetricConfig Config, List Resources); +} diff --git a/csharp/GetAvailability/Services/ResourceHealthService.cs b/csharp/GetAvailability/Services/ResourceHealthService.cs index c483af9..a85b0fb 100644 --- a/csharp/GetAvailability/Services/ResourceHealthService.cs +++ b/csharp/GetAvailability/Services/ResourceHealthService.cs @@ -300,6 +300,7 @@ private static async Task> FetchHealthHistoryAsync( return transitions; } + /// Reads a string property from a JsonElement, returning empty if absent or null. private static string GetPropString(System.Text.Json.JsonElement props, string name) => props.TryGetProperty(name, out var el) ? el.GetString() ?? "" : ""; diff --git a/csharp/GetAvailability/Services/ResourceInventoryService.cs b/csharp/GetAvailability/Services/ResourceInventoryService.cs index 3132e2b..fd1fecc 100644 --- a/csharp/GetAvailability/Services/ResourceInventoryService.cs +++ b/csharp/GetAvailability/Services/ResourceInventoryService.cs @@ -9,7 +9,7 @@ namespace GetAvailability.Services; /// Queries Resource Graph resources table for VMs, SQL DBs, and Storage Accounts. public static class ResourceInventoryService { - // Maps CLI kind abbreviations to Resource Graph type filters + /// Maps CLI kind abbreviations to Azure Resource Graph type identifiers. private static readonly Dictionary KindToType = new(StringComparer.OrdinalIgnoreCase) { ["vm"] = "microsoft.compute/virtualmachines", diff --git a/get-availability.ps1 b/get-availability.ps1 index f35f196..69d7828 100644 --- a/get-availability.ps1 +++ b/get-availability.ps1 @@ -49,11 +49,20 @@ .PARAMETER ActivityGraceMinutes Post-operation grace window for Activity Log lifecycle events (default: 10). +.PARAMETER Batch + Use the regional Metrics Batch API instead of per-resource metric calls. + +.PARAMETER BatchSize + Max resources per batch call (default: 10, max 50). Implies -Batch. + .EXAMPLE ./get-availability.ps1 -Subscriptions 'MySubscription' -Month 202506 .EXAMPLE ./get-availability.ps1 -Subscriptions 'Sub1','Sub2' -Month 202505 -Kinds vm,sql + +.EXAMPLE + ./get-availability.ps1 -Subscriptions 'MySub' -Month 202506 -Batch -BatchSize 20 #> [CmdletBinding(DefaultParameterSetName = 'Run')] @@ -81,6 +90,13 @@ param( [ValidateRange(0, 120)] [int]$ActivityGraceMinutes = 10, + [Parameter(ParameterSetName = 'Run')] + [switch]$Batch, + + [Parameter(ParameterSetName = 'Run')] + [ValidateRange(1, 50)] + [int]$BatchSize = 10, + [Parameter(Mandatory, ParameterSetName = 'ShowVersion')] [switch]$Version ) @@ -201,6 +217,7 @@ ${nameFilter}| extend resourceKind = case( # ── Helpers ─────────────────────────────────────────────────────────────────── +## Abbreviates a resource kind for compact table/summary display. function Get-ShortKind([string]$Kind) { switch ($Kind) { 'VirtualMachine' { 'VM' } @@ -210,20 +227,449 @@ function Get-ShortKind([string]$Kind) { } } +## Returns the effective start of the Resource Health 30-day retention window, +## clamped to PeriodStart if Health History covers the full observation period. function Get-HealthCoverageStart([DateTimeOffset]$PeriodStart) { $now = [DateTimeOffset]::UtcNow $cm = [DateTimeOffset]::new($now.Year, $now.Month, $now.Day, $now.Hour, $now.Minute, 0, [TimeSpan]::Zero) $ret = $cm.AddDays(-30) - if ($ret -gt $PeriodStart) { $ret } else { $PeriodStart } + $ret -gt $PeriodStart ? $ret : $PeriodStart } +## Truncates a string to max length with '...' suffix. function Get-TruncatedString([string]$s, [int]$max) { - if ($s.Length -le $max) { $s } else { $s.Substring(0, $max - 3) + '...' } + $s.Length -le $max ? $s : ($s.Substring(0, $max - 3) + '...') +} + +# ── Batch API configuration ────────────────────────────────────────────────── +# Maps resource kind → metric namespace, metric names, and aggregation for the +# regional Metrics Batch API (https://{region}.metrics.monitor.azure.com). + +$KindConfig = @{ + 'VirtualMachine' = @{ + Namespace = 'Microsoft.Compute/virtualMachines' + MetricNames = 'VmAvailabilityMetric' + Aggregation = 'Minimum' + } + 'AzureSqlDatabase' = @{ + Namespace = 'Microsoft.Sql/servers/databases' + MetricNames = 'Availability' + Aggregation = 'Minimum' + } + 'StorageAccount' = @{ + Namespace = 'Microsoft.Storage/storageAccounts' + MetricNames = 'Availability,Transactions' + Aggregation = 'Minimum,Total' + } } -# ── Metrics (parallel) ─────────────────────────────────────────────────────── +# ── Batch endpoint validation ───────────────────────────────────────────────── + +## Probes each regional batch endpoint with an empty payload to verify reachability. +## 400/401/403 are expected for the dummy subscription and treated as OK. +function Test-BatchEndpoints { + param( + [string]$MetricsToken, + [string[]]$Regions + ) + + Write-Host "Validating batch endpoint availability for $($Regions.Count) region(s)..." + $allOk = $true + foreach ($region in $Regions) { + $endpoint = "https://$region.metrics.monitor.azure.com" + $testUri = "$endpoint/subscriptions/00000000-0000-0000-0000-000000000000/metrics:getBatch?api-version=2023-10-01&metricnamespace=Microsoft.Compute/virtualMachines&metricnames=Percentage%20CPU" + $body = '{"resourceids":[]}' + + try { + $oldPref = $ProgressPreference; $ProgressPreference = 'SilentlyContinue' + try { + Invoke-WebRequest -Uri $testUri -Method POST -Body $body -ContentType 'application/json' ` + -Headers @{ Authorization = "Bearer $MetricsToken" } -UseBasicParsing -ErrorAction Stop | Out-Null + } + finally { $ProgressPreference = $oldPref } + Write-Host " $region`: OK" + } + catch { + $statusCode = try { [int]$_.Exception.Response.StatusCode } catch { 0 } + if ($statusCode -in @(400, 401, 403)) { + Write-Host " $region`: OK (got expected $statusCode for probe)" + } + else { + Write-Host " $region`: FAILED (status=$statusCode) - $_" -ForegroundColor Red + $allOk = $false + } + } + } + if (-not $allOk) { + throw 'One or more batch endpoints are unreachable. Aborting.' + } + Write-Host 'All batch endpoints validated successfully.' + Write-Host '' +} + +# ── Batch metric processing ─────────────────────────────────────────────────── + +## Fetches Azure Monitor metrics using the regional Batch API instead of per-resource +## ARM calls. Resources are grouped by (subscription, region, kind), chunked by +## BatchSize, and processed in parallel waves with GC between waves to bound memory. +## Uses HttpClient with ResponseHeadersRead for streaming JSON parsing. +function Get-BatchAvailabilityMetrics { + param( + [object[]]$Resources, + [DateTimeOffset]$StartDate, + [DateTimeOffset]$EndDate, + [int]$ThrottleLimit, + [int]$BatchSize, + [string]$MetricsToken + ) + + $startIso = $StartDate.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fffZ') + $endIso = $EndDate.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ss.fffZ') + + # Group resources by (subscriptionId, location, kind) + $groups = @{} + foreach ($res in $Resources) { + $groupKey = "$($res.SubscriptionId)|$($res.Location.ToLowerInvariant())|$($res.Kind)" + if (-not $groups.ContainsKey($groupKey)) { + $groups[$groupKey] = [System.Collections.Generic.List[object]]::new() + } + $groups[$groupKey].Add($res) + } + + # Build batch work items: split each group into chunks of BatchSize + $batchWorkItems = [System.Collections.Generic.List[object]]::new() + foreach ($entry in $groups.GetEnumerator()) { + $parts = $entry.Key -split '\|' + $subId = $parts[0] + $location = $parts[1] + $kind = $parts[2] + + $config = $KindConfig[$kind] + if (-not $config) { + Write-Warning "No batch config for kind '$kind', skipping." + continue + } + + $chunk = [System.Collections.Generic.List[object]]::new() + foreach ($res in $entry.Value) { + $chunk.Add($res) + if ($chunk.Count -ge $BatchSize) { + $batchWorkItems.Add([PSCustomObject]@{ + SubscriptionId = $subId + Location = $location + Kind = $kind + Config = $config + Resources = @($chunk) + }) + $chunk = [System.Collections.Generic.List[object]]::new() + } + } + if ($chunk.Count -gt 0) { + $batchWorkItems.Add([PSCustomObject]@{ + SubscriptionId = $subId + Location = $location + Kind = $kind + Config = $config + Resources = @($chunk) + }) + } + } + + $totalResources = $Resources.Count + $totalBatches = $batchWorkItems.Count + $uniqueRegions = @($groups.Keys | ForEach-Object { ($_ -split '\|')[1] } | Select-Object -Unique | Sort-Object) + Write-Host "Grouped $totalResources resource(s) into $totalBatches batch(es) across $($uniqueRegions.Count) region(s) (max $BatchSize per batch)." + $sortedEntries = @($groups.GetEnumerator() | Sort-Object { ($_.Key -split '\|')[1] }, { ($_.Key -split '\|')[2] }, { ($_.Key -split '\|')[0] }) + foreach ($entry in $sortedEntries) { + $parts = $entry.Key -split '\|' + $subName = $entry.Value[0].SubscriptionName + $kind = Get-ShortKind $parts[2] + $location = $parts[1] + $count = $entry.Value.Count + $chunks = [math]::Ceiling($count / $BatchSize) + Write-Host " $location / $kind / $(Get-TruncatedString $subName 30) : $count resource(s) -> $chunks batch(es)" + } + Write-Host '' + $resultByRes = @{} + $batchesDone = 0 + + $batchHttpClient = [System.Net.Http.HttpClient]::new() + $batchHttpClient.DefaultRequestHeaders.Add('Authorization', "Bearer $MetricsToken") + $batchHttpClient.Timeout = [TimeSpan]::FromMinutes(5) + + $waveSize = $ThrottleLimit + for ($waveStart = 0; $waveStart -lt $batchWorkItems.Count; $waveStart += $waveSize) { + $waveEnd = [math]::Min($waveStart + $waveSize, $batchWorkItems.Count) + $waveItems = @($batchWorkItems[$waveStart..($waveEnd - 1)]) + $waveNum = [math]::Floor($waveStart / $waveSize) + 1 + $totalWaves = [math]::Ceiling($batchWorkItems.Count / $waveSize) + + $waveProgress = [hashtable]::Synchronized(@{ Done = 0; Total = $waveItems.Count; Base = $batchesDone; Grand = $totalBatches; Waves = $totalWaves; Wave = $waveNum }) + [Console]::Write("`r Fetching batch metrics: wave $waveNum/$totalWaves, batch $batchesDone/$totalBatches done") + + $waveResults = @($waveItems | ForEach-Object -ThrottleLimit $ThrottleLimit -Parallel { + $workItem = $_ + $client = $using:batchHttpClient + $startIso = $using:startIso + $endIso = $using:endIso + $wp = $using:waveProgress + + $subId = $workItem.SubscriptionId + $location = $workItem.Location + $kind = $workItem.Kind + $config = $workItem.Config + $resources = $workItem.Resources + + $isVm = $kind -eq 'VirtualMachine' + $isStorage = $kind -eq 'StorageAccount' + + $resourceIds = @($resources | ForEach-Object { $_.ResourceId }) + + $endpoint = "https://$location.metrics.monitor.azure.com" + + $uri = "$endpoint/subscriptions/$subId/metrics:getBatch" + + "?starttime=$([uri]::EscapeDataString($startIso))" + + "&endtime=$([uri]::EscapeDataString($endIso))" + + "&interval=PT1M" + + "&metricnamespace=$([uri]::EscapeDataString($config.Namespace))" + + "&metricnames=$([uri]::EscapeDataString($config.MetricNames))" + + "&aggregation=$([uri]::EscapeDataString($config.Aggregation))" + + "&api-version=2023-10-01" + + $bodyObj = @{ resourceids = $resourceIds } + $bodyJson = $bodyObj | ConvertTo-Json -Compress -Depth 3 + + $doc = $null + for ($attempt = 1; $attempt -le 5; $attempt++) { + $httpReq = [System.Net.Http.HttpRequestMessage]::new( + [System.Net.Http.HttpMethod]::Post, $uri) + $httpReq.Content = [System.Net.Http.StringContent]::new( + $bodyJson, [System.Text.Encoding]::UTF8, 'application/json') + try { + $httpResp = $client.SendAsync($httpReq, + [System.Net.Http.HttpCompletionOption]::ResponseHeadersRead + ).GetAwaiter().GetResult() + $sc = [int]$httpResp.StatusCode + if ($sc -ge 200 -and $sc -lt 300) { + $respStream = $httpResp.Content.ReadAsStreamAsync().GetAwaiter().GetResult() + try { $doc = [System.Text.Json.JsonDocument]::ParseAsync($respStream).GetAwaiter().GetResult() } + finally { $respStream.Dispose() } + $httpResp.Dispose(); $httpReq.Dispose() + break + } + $httpResp.Dispose(); $httpReq.Dispose() + if (($sc -in @(401, 429) -or $sc -ge 500) -and $attempt -lt 5) { + Start-Sleep -Seconds ([Math]::Min(30, [Math]::Pow(2, $attempt))) + continue + } + $names = ($resources | ForEach-Object { $_.Name }) -join ', ' + Write-Warning "Batch metric query failed for [$names]: HTTP $sc" + break + } + catch { + try { $httpReq.Dispose() } catch {} + $retryable = $_.ToString() -match 'transport|connection.*closed|reset by peer|timed?\s*out' + if ($retryable -and $attempt -lt 5) { + Start-Sleep -Seconds ([Math]::Min(30, [Math]::Pow(2, $attempt))) + continue + } + $names = ($resources | ForEach-Object { $_.Name }) -join ', ' + Write-Warning "Batch metric query failed for [$names]: $_" + break + } + } + $bodyJson = $null + + if (-not $doc) { + foreach ($resource in $resources) { + [PSCustomObject]@{ + ResourceId = $resource.ResourceId + ResourceIdLower = $resource.ResourceId.ToLowerInvariant() + Name = $resource.Name + AvailableSum = 0.0 + GapMinutes = 0 + ZeroTxMin = 0 + ExcludeFromAvailability = $true + GapTicks = @() + ZeroAvailTicks = @() + DegradedMinutes = 0 + DegradedTicks = @() + DegradedValues = @() + SuspectMinutes = 0 + } + } + $wp.Done++ + [Console]::Write("`r Fetching batch metrics: wave $($wp.Wave)/$($wp.Waves), batch $($wp.Base + $wp.Done)/$($wp.Grand) done") + return + } + + $jNum = [System.Text.Json.JsonValueKind]::Number + + function script:GetNum([System.Text.Json.JsonElement]$dp, [string]$prop) { + foreach ($p in $dp.EnumerateObject()) { + if ($p.Name -eq $prop -and $p.Value.ValueKind -eq $jNum) { return $p.Value.GetDouble() } + } + return $null + } + + $resById = @{} + foreach ($r in $resources) { $resById[$r.ResourceId.ToLowerInvariant()] = $r } + + try { + $valuesArr = $doc.RootElement.GetProperty('values') + foreach ($resourceEntry in $valuesArr.EnumerateArray()) { + $resId = $resourceEntry.GetProperty('resourceid').GetString() + $resIdLower = $resId.ToLowerInvariant() + $resObj = if ($resById.ContainsKey($resIdLower)) { $resById[$resIdLower] } else { $null } + $resName = if ($resObj) { $resObj.Name } else { $resId } + + $metricValueArr = $resourceEntry.GetProperty('value') + + [double]$availSum = 0.0 + [int]$zeroTxMin = 0 + [int]$numericPoints = 0 + $nullTicks = [System.Collections.Generic.List[long]]::new() + $zeroTicks = [System.Collections.Generic.List[long]]::new() + $degradedTicks = [System.Collections.Generic.List[long]]::new() + $degradedValues = [System.Collections.Generic.List[double]]::new() + + if ($isStorage) { + $txByTicks = [System.Collections.Generic.Dictionary[long,double]]::new() + foreach ($metricEl in $metricValueArr.EnumerateArray()) { + $mName = $metricEl.GetProperty('name').GetProperty('value').GetString() + if ($mName -ne 'Transactions') { continue } + foreach ($tsEl in $metricEl.GetProperty('timeseries').EnumerateArray()) { + foreach ($dp in $tsEl.GetProperty('data').EnumerateArray()) { + $time = [datetime]::Parse($dp.GetProperty('timeStamp').GetString()).ToUniversalTime() + $tot = GetNum $dp 'total' + if ($null -ne $tot) { $txByTicks[$time.Ticks] = $tot } + } + } + } + foreach ($metricEl in $metricValueArr.EnumerateArray()) { + $mName = $metricEl.GetProperty('name').GetProperty('value').GetString() + if ($mName -ne 'Availability') { continue } + foreach ($tsEl in $metricEl.GetProperty('timeseries').EnumerateArray()) { + foreach ($dp in $tsEl.GetProperty('data').EnumerateArray()) { + $time = [datetime]::Parse($dp.GetProperty('timeStamp').GetString()).ToUniversalTime() + $ticks = $time.Ticks + $txVal = [double]0 + $hasTx = $txByTicks.TryGetValue($ticks, [ref]$txVal) -and $txVal -gt 0 + $minVal = GetNum $dp 'minimum' + + if ($hasTx -and $null -ne $minVal) { + $norm = $minVal / 100.0 + $numericPoints++ + if ($norm -eq 0.0) { + $zeroTicks.Add($ticks) + } else { + $availSum += $norm + if ($norm -lt 1.0) { + $degradedTicks.Add($ticks) + $degradedValues.Add($norm) + } + } + } elseif ($hasTx -and $null -eq $minVal) { + $nullTicks.Add($ticks) + } elseif (-not $hasTx) { + $zeroTxMin++ + } + } + } + } + $txByTicks = $null + } else { + foreach ($metricEl in $metricValueArr.EnumerateArray()) { + $mName = $metricEl.GetProperty('name').GetProperty('value').GetString() + $isPrimary = $mName -eq 'VmAvailabilityMetric' -or $mName -eq 'Availability' + if (-not $isPrimary) { continue } + + foreach ($tsEl in $metricEl.GetProperty('timeseries').EnumerateArray()) { + foreach ($dp in $tsEl.GetProperty('data').EnumerateArray()) { + $time = [datetime]::Parse($dp.GetProperty('timeStamp').GetString()).ToUniversalTime() + $ticks = $time.Ticks + $minVal = GetNum $dp 'minimum' + + if ($null -ne $minVal) { + $numericPoints++ + $v = if ($isVm) { $minVal } else { $minVal / 100.0 } + if ($v -eq 0.0) { + $zeroTicks.Add($ticks) + } else { + $availSum += $v + if ($v -lt 1.0) { + $degradedTicks.Add($ticks) + $degradedValues.Add($v) + } + } + } else { + $nullTicks.Add($ticks) + } + } + } + } + } + + $gapMinutes = $nullTicks.Count + $zeroTicks.Count + $degradedMinutes = $degradedTicks.Count + $excludeFromAvailability = $numericPoints -eq 0 -and $nullTicks.Count -eq 0 -and + $zeroTicks.Count -eq 0 -and $degradedMinutes -eq 0 + + [PSCustomObject]@{ + ResourceId = $resId + ResourceIdLower = $resIdLower + Name = $resName + AvailableSum = $availSum + GapMinutes = $gapMinutes + ZeroTxMin = $zeroTxMin + ExcludeFromAvailability = $excludeFromAvailability + GapTicks = @($nullTicks) + ZeroAvailTicks = @($zeroTicks) + DegradedMinutes = $degradedMinutes + DegradedTicks = @($degradedTicks) + DegradedValues = @($degradedValues) + SuspectMinutes = $gapMinutes + $degradedMinutes + } + } + } + finally { + if ($doc) { $doc.Dispose() } + } + + $wp.Done++ + [Console]::Write("`r Fetching batch metrics: wave $($wp.Wave)/$($wp.Waves), batch $($wp.Base + $wp.Done)/$($wp.Grand) done") + }) + + # Collect this wave's results into the main hashtable + [Console]::Write("`r" + (' ' * 60) + "`r") + foreach ($item in $waveResults) { + $key = if ($item.ResourceIdLower) { $item.ResourceIdLower } else { $item.ResourceId.ToLowerInvariant() } + $resultByRes[$key] = $item + } + $batchesDone += $waveItems.Count + $waveResults = $null + + if ($waveStart + $waveSize -lt $batchWorkItems.Count) { + [GC]::Collect() + [GC]::WaitForPendingFinalizers() + } + } + + $batchHttpClient.Dispose() + + [Console]::Write("`r" + (' ' * 80) + "`r") + Write-Host "Batch queries completed. Received results for $($resultByRes.Count) / $totalResources resource(s)." + + $resultByRes +} + +# ── Per-resource metrics (parallel) ─────────────────────────────────────────── + +## Fetches Azure Monitor metrics one resource at a time via the ARM Metrics API. +## Uses ForEach-Object -Parallel for concurrent calls with System.Text.Json for +## low-allocation parsing. Each resource gets its own HTTP call with retry on 429/5xx. function Get-AvailabilityMetrics { param( [object[]]$Resources, @@ -430,6 +876,16 @@ function Get-AvailabilityMetrics { # ── Suspect gap investigation (parallel) ────────────────────────────────────── +## For each resource with suspect minutes, queries Activity Log (for supported +## lifecycle operations) and Resource Health (for the overlap with the 30-day +## retention window) to classify every suspect minute. +## Classification precedence: +## 1. Platform fault (Resource Health) → stays eligible, counts as downtime +## 2. Lifecycle activity (Activity Log) → excluded from eligibility +## 3. Unknown / customer-initiated (Health) → excluded from eligibility +## 4. Remaining null → metric issue, excluded +## 5. Remaining 0% → trusted as downtime +## 6. Remaining degraded (0% < v < 100%) → trusted as degraded availability function Invoke-SuspectGapInvestigation { param( [object[]]$Candidates, @@ -455,6 +911,7 @@ function Invoke-SuspectGapInvestigation { $hcStart = $using:HealthCoverageStart # ── Local helpers ───────────────────────────────────────────── + ## ARM GET with retry on 429/5xx and exponential backoff. function script:ArmGet([string]$uri) { for ($a = 0; $a -lt 6; $a++) { try { @@ -473,10 +930,12 @@ function Invoke-SuspectGapInvestigation { } } + ## Truncates a DateTimeOffset to the minute boundary (seconds = 0). function script:TruncMin([DateTimeOffset]$v) { [DateTimeOffset]::new($v.Year, $v.Month, $v.Day, $v.Hour, $v.Minute, 0, [TimeSpan]::Zero) } + ## Returns $true if a tick value falls inside any of the given intervals. function script:InInterval([long]$tick, [object[]]$intervals) { foreach ($iv in $intervals) { if ($tick -ge $iv.FromTicks -and $tick -lt $iv.ToTicks) { return $true } @@ -484,6 +943,7 @@ function Invoke-SuspectGapInvestigation { $false } + ## Safely reads a string property from a JsonElement, returning '' if absent. function script:GetJsonStr([System.Text.Json.JsonElement]$el, [string]$name) { $v = [System.Text.Json.JsonElement]::new() if ($el.TryGetProperty($name, [ref]$v) -and @@ -493,6 +953,7 @@ function Invoke-SuspectGapInvestigation { '' } + ## Parses a timestamp string from Azure APIs (multiple formats) into a UTC DateTime. function script:ParseTimestamp([string]$s) { if ([string]::IsNullOrWhiteSpace($s)) { return $null } $dto = [DateTimeOffset]::MinValue @@ -506,6 +967,8 @@ function Invoke-SuspectGapInvestigation { } # ── Activity Log ────────────────────────────────────────────── + # Query Activity Log for supported lifecycle operations (VM: start/deallocate/ + # powerOff/restart; SQL: pause/resume) and build intervals that explain metric gaps. $activityIntervals = @() $supportsActivity = $c.Kind -eq 'VirtualMachine' -or $c.Kind -eq 'AzureSqlDatabase' @@ -645,6 +1108,8 @@ function Invoke-SuspectGapInvestigation { } # ── Resource Health ─────────────────────────────────────────── + # Query the Resource Health REST API for the portion of the window + # covered by the current 30-day retention window. $healthHistoryApplied = [DateTimeOffset]$hcStart -lt [DateTimeOffset]$pEnd $faultIntervals = @() $unknownIntervals = @() @@ -786,6 +1251,7 @@ function Invoke-SuspectGapInvestigation { } # ── Classify each suspect tick ──────────────────────────────── + # Precedence: platform fault > activity log > health unknown/customer > 0% downtime > metric issue $zeroSet = [System.Collections.Generic.HashSet[long]]::new() foreach ($zt in @($c.ZeroTicksArray)) { [void]$zeroSet.Add([long]$zt) } @@ -866,6 +1332,7 @@ function Invoke-SuspectGapInvestigation { # ── Output ──────────────────────────────────────────────────────────────────── +## Prints a fixed-width table with one row per resource showing availability metrics. function Write-ResultsTable([object[]]$Sorted) { $fmt = '{0,-24} {1,-30} {2,-7} {3,-12} {4,7} {5,6} {6,7} {7,10} {8,10} {9,8} {10,10}' Write-Host '' @@ -890,6 +1357,8 @@ function Write-ResultsTable([object[]]$Sorted) { Write-Host '' } +## Prints per-subscription summaries grouped by Kind + Location, plus a cross- +## subscription overall summary when multiple subscriptions are present. function Write-SubscriptionSummaries([object[]]$Sorted) { $eligible = @($Sorted | Where-Object { $_.AvailabilityPct -ne 'N/A' }) @@ -916,9 +1385,9 @@ function Write-SubscriptionSummaries([object[]]$Sorted) { # Cross-subscription summary $subs = @($eligible | ForEach-Object { $_.SubscriptionName } | Select-Object -Unique) if ($subs.Count -gt 1 -and $eligible.Count -gt 0) { - Write-Host ([char]0x2550 * 62) + Write-Host ([string]::new([char]0x2550, 62)) Write-Host ' OVERALL (all subscriptions)' - Write-Host ([char]0x2550 * 62) + Write-Host ([string]::new([char]0x2550, 62)) foreach ($g in ($eligible | Group-Object { "$($_.Kind)|$($_.Location)" } | Sort-Object Name)) { $items = @($g.Group) $n = $items.Count @@ -949,12 +1418,9 @@ $utcEnd = $window.End $totalMinutes = $window.TotalMinutes $healthCoverageStart = Get-HealthCoverageStart $utcStart -$healthCoveredMinutes = if ($healthCoverageStart -lt $utcEnd) { - [int]($utcEnd - $healthCoverageStart).TotalMinutes -} else { 0 } +$healthCoveredMinutes = $healthCoverageStart -lt $utcEnd ? [int]($utcEnd - $healthCoverageStart).TotalMinutes : 0 -$periodLabel = if ($window.IsMonthToDate) { "month $($window.NormalizedMonth) (month-to-date)" } - else { "month $($window.NormalizedMonth)" } +$periodLabel = $window.IsMonthToDate ? "month $($window.NormalizedMonth) (month-to-date)" : "month $($window.NormalizedMonth)" Write-Host "Period: $periodLabel ($($utcStart.ToString('u')) -> $($utcEnd.ToString('u')), $totalMinutes min)" if ($healthCoverageStart -gt $utcStart -and $healthCoveredMinutes -gt 0) { @@ -964,6 +1430,9 @@ if ($healthCoverageStart -gt $utcStart -and $healthCoveredMinutes -gt 0) { } # Step 2: Authenticate and resolve subscriptions +# -BatchSize explicitly set implies -Batch +if ($PSBoundParameters.ContainsKey('BatchSize') -and -not $Batch) { $Batch = [switch]::new($true) } + Write-Host -NoNewline 'Authenticating... ' $allAzSubs = @(Get-AzSubscription) $resolvedSubs = @(foreach ($name in $Subscriptions) { @@ -974,19 +1443,33 @@ $resolvedSubs = @(foreach ($name in $Subscriptions) { }) $subIds = @($resolvedSubs.Id) $subIdToName = @{}; foreach ($s in $resolvedSubs) { $subIdToName[$s.Id] = $s.Name } + +$rawToken = (Get-AzAccessToken -ResourceUrl 'https://management.azure.com').Token +$armToken = ($rawToken -is [securestring]) ? ($rawToken | ConvertFrom-SecureString -AsPlainText) : [string]$rawToken +$rawToken = $null + +if ($Batch) { + $rawMetrics = (Get-AzAccessToken -ResourceUrl 'https://metrics.monitor.azure.com').Token + $metricsToken = ($rawMetrics -is [securestring]) ? ($rawMetrics | ConvertFrom-SecureString -AsPlainText) : [string]$rawMetrics + $rawMetrics = $null +} + Write-Host 'OK' Write-Host "Processing $($resolvedSubs.Count) subscription(s): $($resolvedSubs.Name -join ', ')" Write-Host "Kinds: $($Kinds -join ', ')" +if ($Batch) { Write-Host "Mode: Batch (batch-size=$BatchSize)" } # Step 3: Query Resource Graph inventory Write-Host -NoNewline 'Querying resource inventory... ' $resources = Get-ResourceInventory -SubscriptionIds $subIds -SubIdToName $subIdToName ` -Kinds $Kinds -ResourceNameFilter $Resource -Write-Host "Found $($resources.Count) resource(s) across $($resolvedSubs.Count) subscription(s)." + +$regionCount = @($resources | ForEach-Object { $_.Location } | Select-Object -Unique).Count +Write-Host "Found $($resources.Count) resource(s) across $($resolvedSubs.Count) subscription(s), $regionCount region(s)." if ($resources.Count -eq 0) { Write-Host 'No resources found.'; return } -# Step 4: Build initial eligibility (all minutes eligible) +# Step 4: Build initial eligibility records (all minutes start as eligible) $eligByRes = @{} foreach ($res in $resources) { $eligByRes[$res.ResourceId.ToLowerInvariant()] = [PSCustomObject]@{ @@ -1006,15 +1489,19 @@ foreach ($res in $resources) { } } -# Step 5: Fetch Azure Monitor metrics per resource in parallel -$rawToken = (Get-AzAccessToken -ResourceUrl 'https://management.azure.com').Token -$armToken = ($rawToken -is [securestring]) ? ($rawToken | ConvertFrom-SecureString -AsPlainText) : [string]$rawToken -$rawToken = $null - -$metricResults = Get-AvailabilityMetrics -Resources $resources -StartDate $utcStart ` - -EndDate $utcEnd -ThrottleLimit $Parallelism -ArmToken $armToken +# Step 5: Fetch Azure Monitor metrics +if ($Batch) { + $uniqueRegions = @($resources | ForEach-Object { $_.Location.ToLowerInvariant() } | Select-Object -Unique | Sort-Object) + Test-BatchEndpoints -MetricsToken $metricsToken -Regions $uniqueRegions + $metricResults = Get-BatchAvailabilityMetrics -Resources $resources -StartDate $utcStart ` + -EndDate $utcEnd -ThrottleLimit $Parallelism -BatchSize $BatchSize -MetricsToken $metricsToken + $metricsToken = $null +} else { + $metricResults = Get-AvailabilityMetrics -Resources $resources -StartDate $utcStart ` + -EndDate $utcEnd -ThrottleLimit $Parallelism -ArmToken $armToken +} -# Step 6: Build suspect candidates and investigate +# Step 6: Build suspect candidates and investigate via Activity Log + Resource Health $suspectCandidates = [System.Collections.Generic.List[object]]::new() foreach ($res in $resources) { $key = $res.ResourceId.ToLowerInvariant() @@ -1046,7 +1533,7 @@ if ($suspectCandidates.Count -gt 0) { -HealthCoverageStart $healthCoverageStart } -# Step 7: Assemble final results +# Step 7: Assemble final results — apply investigation outcomes and zero-tx exclusions foreach ($res in $resources) { $key = $res.ResourceId.ToLowerInvariant() $elig = $eligByRes[$key] @@ -1199,6 +1686,3 @@ Write-SubscriptionSummaries $sorted $sw.Stop() Write-Host "Completed in $($sw.Elapsed.ToString('hh\:mm\:ss\.ff'))" - -# Output objects for pipeline use -$sorted