Repository navigation
feat(azure): add azure_endpoint_availability fetcher - #378
bruceweygandt wants to merge 3 commits into
Conversation
3fe2c49 to
d96e260
Compare
Controgon
left a comment
There was a problem hiding this comment.
The metric queries are well built: a per-test name filter, hourly data, and missing hours counted as missing. Coverage reporting is a good idea. I ran the tests (17 passed), the contract tests and paramify list offline, and all pass. I also ran collect with a stub client, and found two bugs that need a fix before merge.
1. Must fix: location: exclusion patterns never match
fetchers/azure/endpoint_availability/fetcher.py:61-75, fetcher.py:78-89
Problem. ResourceExclusions.split reads location from each record. test_record and load_balancer_record do not include location.
Why this is a problem. With resource_exclusions: [location:westus2] and resources in westus2, nothing is excluded, and the pattern is reported as unmatched. Resources that the customer took out of scope stay in the evidence and in every count.
Fix. Add "location": raw.get("location") to both records, and add a test with a location: pattern.
2. Must fix: one load balancer without probes makes the load balancer verdict null
fetcher.py:243-250, fetcher.py:139, fetcher.py:170-171
Problem. The fetcher reads VipAvailability and DipAvailability for every Standard load balancer. A load balancer with no health probes and no load-balancing rules, such as an outbound-only load balancer, has no data for these metrics. It is then "measurement incomplete" in every hour.
Why this is a problem. load_balancers_below_target is null when any load balancer is incomplete. In my stub run, one load balancer was at 100% in every hour and a second, outbound-only load balancer had no data. The result was load_balancers_below_target = None. Outbound-only load balancers are common in Kubernetes subscriptions, so the verdict cannot be calculated there.
Fix. Record the number of load-balancing rules, inbound NAT rules and probes. Mark data_path as not applicable when there are no rules, and health_probe as not applicable when there are no probes. Count these load balancers separately, and add a test.
3. Question: one incomplete item makes the whole count null
fetcher.py:138-142
Each item already gets below_target: null when its hours are incomplete. The summary also sets the whole count to null when one item is incomplete. A test created three days ago stays incomplete for the rest of the 7-day lookback. During that time, availability_tests_below_target is null for all tests. Would it be better to count below-target among the complete items and add an availability_tests_not_judged count?
Smaller items
fetcher.py:71. Classic URL ping tests keep their URL in theConfiguration.WebTestXML, not inRequest.RequestUrl. They do not count inendpoints_tested. Please read the URL from the XML, or count classic tests separately.fetcher.yaml:114.availability_tests_below_target_latest_houris also null when any test is unread or incomplete, not only when no target is set. Please correct the description.tests/test_contracts.py. The instructions pass the plain-text format check, butazure/endpoint_availabilityis not inTEXT_FORMAT_FETCHERS. Please add it.fetcher.yaml:71. Reader already includes metric reads, so the text about Monitoring Reader can cause confusion. Please make it more specific or remove it.fetcher.py:213. The window ends at the start of the last complete hour, so that hour is not included. Please make the Output line match.
…trics without rules or probes, judge complete items and count the rest, classic test URLs
|
Thanks for running it with a stub client. Both bugs are fixed in the latest commit, and I took your suggestion on the question. 1. 2. An outbound-only load balancer made the verdict null. Each load balancer now records its load-balancing rules, inbound NAT rules and probes. The data path is read only when it has load-balancing rules, and the health probe only when it has probes; otherwise that metric is 3. One incomplete item made the whole count null. Agreed. Smaller items
Tests: 22 passed (5 new, 3 updated for the new counts). Contract tests: 884 passed. Ruff is clean on the changed files. |
Controgon
left a comment
There was a problem hiding this comment.
Changes still needed: the main merge broke tests/test_contracts.py. Everything from the first review is handled.
Fixed
-
location:exclusions: fixed. Both records carrylocation, and a full-run test excludes a test and a load balancer and checks that the pattern is not reported as unmatched.
-
- Outbound-only load balancers: fixed. A metric is read only when the load balancer has rules or probes for it. Load balancers with neither are counted in
load_balancers_without_measured_traffic, and no metric call goes to them.
- Outbound-only load balancers: fixed. A metric is read only when the load balancer has rules or probes for it. Load balancers with neither are counted in
-
- One incomplete item: fixed. The below-target counts now cover only the items that can be judged, and the new
_not_judgedcounts show the rest.
- One incomplete item: fixed. The below-target counts now cover only the items that can be judged, and the new
- Classic ping test URLs, the latest-hour description,
TEXT_FORMAT_FETCHERS, the Access line and the Output line: fixed.
Still to fix
1. Must fix: the main merge left the old TEXT_FORMAT_FETCHERS list open
tests/test_contracts.py:227
Problem. The merge (fd9134d) kept the old TEXT_FORMAT_FETCHERS = ( list above main's TEXT_FORMAT_PENDING, without its closing parenthesis. The test module no longer imports: CI's test jobs fail with SyntaxError: '(' was never closed (1 error).
Why this is a problem. All of tests/test_contracts.py is skipped by the error, so no fetcher's contract is checked on this branch.
Fix. Delete lines 227–325 and keep main's version of the block. main now checks every fetcher's text format unless it is on TEXT_FORMAT_PENDING, so endpoint_availability needs no entry.
Summary
Adds
azure_endpoint_availability, an Azure evidence fetcher for customer-facing availability. It reports whether endpoints are tested from outside by Application Insights availability tests and how available they were, and whether Standard load balancers kept their data path and backend health probes up, each against an availability target once one is adopted.Evidence set:
EVD-AZURE-ENDPOINT-AVAILABILITY(Azure Endpoint Availability). KSI: KSI-CNA-OFA.This complements
azure_service_availability, which covers App Service, Front Door and SQL Managed Instance. Offerings that serve customers through their own ingress (for example Kubernetes behind a Standard load balancer) had no availability evidence.How it collects
GET Microsoft.Insights/webtests: every availability test with its URL, locations, frequency and its Application Insights component (from thehidden-linktag).availabilityResults/availabilityPercentageon its component, filtered to the test's name.GET Microsoft.Network/loadBalancers: SKU and public frontends.VipAvailability(data path) andDipAvailability(health probe). Basic load balancers emit no metrics and are reported as not applicable.Settings:
lookback_days(default 7, maximum 30),availability_target_percentageand the sharedresource_exclusions. The target has no default: until an offering adopts an availability objective, every below-target count is null, so nothing is judged against an objective nobody adopted. Access: Reader on the subscription (Monitoring Reader where Reader is not enough for metrics). No new dependencies.Measurement coverage and what tests check
hours_expected,hours_missingandmeasurement_coverage_percentageover the lookback, which ends at the last complete hour because Azure Monitor publishes each hour a few minutes after it closes.null, and while anything in scope is incomplete the summary's below-target counts arenulltoo, so a few successful checks never stand for a whole week.availability_tests_with_missing_measurementsand the lowest coverage show the gaps directly.ValidationRuleson standard tests, rules in the XML of classic ping tests).availability_tests_checking_applicationcounts tests that check both status code and content, which show the application works and not only that it answers.Failure handling
Uses
run_subscription. An unreadable list or metric, or a test with no link to its component, is recorded inapi_failuresand fails the run. Verdicts are null, not 0, when any enabled test's results or any Standard load balancer's metrics could not be read. No availability tests at all reads as 0 endpoints tested, which a validator can fail on.Testing
tests/test_azure_endpoint_availability.py: component parsing, test and load balancer projection, hourly averaging across series, judgement against the target, null-not-zero summaries, the exact metric query (per-test name filter, hourly interval, Standard load balancers only), unreadable metrics, a test without a component link, an invalid target, null verdicts while no target is set, a partial window reported as unknown rather than healthy, coverage counts, status code and content checks from standard and classic tests, andassert_summary_documented.paramify listdiscovers it; the fake-credential smoke test exits 1 with a valid JSON file,auth_failedstatus and null verdicts.test_contracts.py,test_category_requires.pyand all Azure and Entra tests pass (5,419).Not yet run against a live tenant.