Skip to content

feat(azure): add azure_endpoint_availability fetcher - #378

Open
bruceweygandt wants to merge 3 commits into
mainfrom
feat/azure-endpoint-availability
Open

bruceweygandt wants to merge 3 commits into
mainfrom
feat/azure-endpoint-availability

Conversation

@bruceweygandt

@bruceweygandt bruceweygandt commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds azure_endpoint_availability, an Azure evidence fetcher for customer-facing availability. It reports whether endpoints are tested from outside by Application Insights availability tests and how available they were, and whether Standard load balancers kept their data path and backend health probes up, each against an availability target once one is adopted.

Evidence set: EVD-AZURE-ENDPOINT-AVAILABILITY (Azure Endpoint Availability). KSI: KSI-CNA-OFA.

This complements azure_service_availability, which covers App Service, Front Door and SQL Managed Instance. Offerings that serve customers through their own ingress (for example Kubernetes behind a Standard load balancer) had no availability evidence.

How it collects

  • GET Microsoft.Insights/webtests: every availability test with its URL, locations, frequency and its Application Insights component (from the hidden-link tag).
  • Per enabled test: hourly availabilityResults/availabilityPercentage on its component, filtered to the test's name.
  • GET Microsoft.Network/loadBalancers: SKU and public frontends.
  • Per Standard load balancer: hourly VipAvailability (data path) and DipAvailability (health probe). Basic load balancers emit no metrics and are reported as not applicable.

Settings: lookback_days (default 7, maximum 30), availability_target_percentage and the shared resource_exclusions. The target has no default: until an offering adopts an availability objective, every below-target count is null, so nothing is judged against an objective nobody adopted. Access: Reader on the subscription (Monitoring Reader where Reader is not enough for metrics). No new dependencies.

Measurement coverage and what tests check

  • Every test and Standard load balancer metric reports hours_expected, hours_missing and measurement_coverage_percentage over the lookback, which ends at the last complete hour because Azure Monitor publishes each hour a few minutes after it closes.
  • Below target is judged only when every hour was measured. A test or load balancer with a missing hour has below target null, and while anything in scope is incomplete the summary's below-target counts are null too, so a few successful checks never stand for a whole week. availability_tests_with_missing_measurements and the lowest coverage show the gaps directly.
  • Each test reports whether it checks an expected status code, response content and the certificate (ValidationRules on standard tests, rules in the XML of classic ping tests). availability_tests_checking_application counts tests that check both status code and content, which show the application works and not only that it answers.

Failure handling

Uses run_subscription. An unreadable list or metric, or a test with no link to its component, is recorded in api_failures and fails the run. Verdicts are null, not 0, when any enabled test's results or any Standard load balancer's metrics could not be read. No availability tests at all reads as 0 endpoints tested, which a validator can fail on.

Testing

  • 17 tests in tests/test_azure_endpoint_availability.py: component parsing, test and load balancer projection, hourly averaging across series, judgement against the target, null-not-zero summaries, the exact metric query (per-test name filter, hourly interval, Standard load balancers only), unreadable metrics, a test without a component link, an invalid target, null verdicts while no target is set, a partial window reported as unknown rather than healthy, coverage counts, status code and content checks from standard and classic tests, and assert_summary_documented.
  • paramify list discovers it; the fake-credential smoke test exits 1 with a valid JSON file, auth_failed status and null verdicts.
  • test_contracts.py, test_category_requires.py and all Azure and Entra tests pass (5,419).

Not yet run against a live tenant.

@bruceweygandt
bruceweygandt force-pushed the feat/azure-endpoint-availability branch from 3fe2c49 to d96e260 Compare November 17, 2026 17:50

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The metric queries are well built: a per-test name filter, hourly data, and missing hours counted as missing. Coverage reporting is a good idea. I ran the tests (17 passed), the contract tests and paramify list offline, and all pass. I also ran collect with a stub client, and found two bugs that need a fix before merge.

1. Must fix: location: exclusion patterns never match

fetchers/azure/endpoint_availability/fetcher.py:61-75, fetcher.py:78-89

Problem. ResourceExclusions.split reads location from each record. test_record and load_balancer_record do not include location.

Why this is a problem. With resource_exclusions: [location:westus2] and resources in westus2, nothing is excluded, and the pattern is reported as unmatched. Resources that the customer took out of scope stay in the evidence and in every count.

Fix. Add "location": raw.get("location") to both records, and add a test with a location: pattern.

2. Must fix: one load balancer without probes makes the load balancer verdict null

fetcher.py:243-250, fetcher.py:139, fetcher.py:170-171

Problem. The fetcher reads VipAvailability and DipAvailability for every Standard load balancer. A load balancer with no health probes and no load-balancing rules, such as an outbound-only load balancer, has no data for these metrics. It is then "measurement incomplete" in every hour.

Why this is a problem. load_balancers_below_target is null when any load balancer is incomplete. In my stub run, one load balancer was at 100% in every hour and a second, outbound-only load balancer had no data. The result was load_balancers_below_target = None. Outbound-only load balancers are common in Kubernetes subscriptions, so the verdict cannot be calculated there.

Fix. Record the number of load-balancing rules, inbound NAT rules and probes. Mark data_path as not applicable when there are no rules, and health_probe as not applicable when there are no probes. Count these load balancers separately, and add a test.

3. Question: one incomplete item makes the whole count null

fetcher.py:138-142

Each item already gets below_target: null when its hours are incomplete. The summary also sets the whole count to null when one item is incomplete. A test created three days ago stays incomplete for the rest of the 7-day lookback. During that time, availability_tests_below_target is null for all tests. Would it be better to count below-target among the complete items and add an availability_tests_not_judged count?

Smaller items

  • fetcher.py:71. Classic URL ping tests keep their URL in the Configuration.WebTest XML, not in Request.RequestUrl. They do not count in endpoints_tested. Please read the URL from the XML, or count classic tests separately.
  • fetcher.yaml:114. availability_tests_below_target_latest_hour is also null when any test is unread or incomplete, not only when no target is set. Please correct the description.
  • tests/test_contracts.py. The instructions pass the plain-text format check, but azure/endpoint_availability is not in TEXT_FORMAT_FETCHERS. Please add it.
  • fetcher.yaml:71. Reader already includes metric reads, so the text about Monitoring Reader can cause confusion. Please make it more specific or remove it.
  • fetcher.py:213. The window ends at the start of the last complete hour, so that hour is not included. Please make the Output line match.

…trics without rules or probes, judge complete items and count the rest, classic test URLs
@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Thanks for running it with a stub client. Both bugs are fixed in the latest commit, and I took your suggestion on the question.

1. location: patterns never matched. test_record and load_balancer_record now carry location. A full-run test excludes one test and one load balancer with location:westus2 and checks both are counted as excluded and the pattern is not reported as unmatched.

2. An outbound-only load balancer made the verdict null. Each load balancer now records its load-balancing rules, inbound NAT rules and probes. The data path is read only when it has load-balancing rules, and the health probe only when it has probes; otherwise that metric is not_applicable and is not requested. A Standard load balancer with neither is counted in the new load_balancers_without_measured_traffic and is not judged. Your case (one load balancer at 100% and one outbound-only) is a test: load_balancers_below_target is 0, not null, and no metric call goes to the outbound-only one. A test also covers a load balancer with rules but no probes, judged on its data path only.

3. One incomplete item made the whole count null. Agreed. availability_tests_below_target and load_balancers_below_target now count among the items with every hour measured, and the new availability_tests_not_judged and load_balancers_not_judged count the rest (an hour missing, or unread). Both are null only while no target is set or the list could not be read. An unread item is counted as not judged, never as meeting the target.

Smaller items

  • Classic URL ping tests: the URL is read from the Url attribute of the Request element in the WebTest XML, with entities unescaped, so these tests count in endpoints_tested. Tested.
  • availability_tests_below_target_latest_hour: with the change above it is null only while no target is set or the test list could not be read, and the description now says so.
  • TEXT_FORMAT_FETCHERS: azure/endpoint_availability added; the format checks pass.
  • Access line: now says Reader includes reading Azure Monitor metrics; the Monitoring Reader text is gone.
  • Output line: now says the lookback ends at the start of the last complete hour, which is not included because Azure Monitor publishes it a few minutes after it closes.

Tests: 22 passed (5 new, 3 updated for the new counts). Contract tests: 884 passed. Ruff is clean on the changed files.

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes still needed: the main merge broke tests/test_contracts.py. Everything from the first review is handled.

Fixed

    1. location: exclusions: fixed. Both records carry location, and a full-run test excludes a test and a load balancer and checks that the pattern is not reported as unmatched.
    1. Outbound-only load balancers: fixed. A metric is read only when the load balancer has rules or probes for it. Load balancers with neither are counted in load_balancers_without_measured_traffic, and no metric call goes to them.
    1. One incomplete item: fixed. The below-target counts now cover only the items that can be judged, and the new _not_judged counts show the rest.
  • Classic ping test URLs, the latest-hour description, TEXT_FORMAT_FETCHERS, the Access line and the Output line: fixed.

Still to fix

1. Must fix: the main merge left the old TEXT_FORMAT_FETCHERS list open

tests/test_contracts.py:227

Problem. The merge (fd9134d) kept the old TEXT_FORMAT_FETCHERS = ( list above main's TEXT_FORMAT_PENDING, without its closing parenthesis. The test module no longer imports: CI's test jobs fail with SyntaxError: '(' was never closed (1 error).

Why this is a problem. All of tests/test_contracts.py is skipped by the error, so no fetcher's contract is checked on this branch.

Fix. Delete lines 227–325 and keep main's version of the block. main now checks every fetcher's text format unless it is on TEXT_FORMAT_PENDING, so endpoint_availability needs no entry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants