Skip to content

feat(azure): add azure_flexible_server_backups fetcher - #377

Merged
Controgon merged 7 commits into
mainfrom
feat/azure-database-backups
Nov 17, 2026
Merged

Controgon merged 7 commits into
mainfrom
feat/azure-database-backups

Conversation

@bruceweygandt

@bruceweygandt bruceweygandt commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds azure_flexible_server_backups, an Azure evidence fetcher for Azure Database for PostgreSQL and MySQL flexible servers. It reports whether each server's automatic backups actually run (the newest completed automatic backup against a window), with manual (on-demand) backups reported apart so a recent manual backup cannot hide broken automation; backup retention against an optional minimum; geo-redundant backup; stopped servers; and servers created by point-in-time restore.

Evidence set: EVD-AZURE-FLEXIBLE-SERVER-BACKUPS (Azure PostgreSQL and MySQL Backups). KSIs: KSI-RPL-ABO, KSI-RPL-ARP, KSI-RPL-TRC, KSI-SVC-RUD.

The name is scoped to flexible servers on purpose, so Azure SQL Database can get its own fetcher later without a clash. azure_postgresql_configuration and azure_mysql_configuration already report backup settings, including geo-redundant backup; servers_geo_redundant_backup is repeated here so all backup evidence (recency, retention, off-site copy, restores) sits in one evidence set.

How it collects

Resource Manager only, with Reader on the subscription:

  • GET providers/Microsoft.DBforPostgreSQL and providers/Microsoft.DBforMySQL: registration, and the newest stable API version listed for both flexibleServers and flexibleServers/backups (falling back to the flexibleServers list), since Azure Government lags.
  • GET .../flexibleServers for each engine: retention, geo-redundant backup, earliest restore point, state, replication role and restore source.
  • GET flexibleServers/{server}/backups per server that is not a read replica.

Automatic, manual and unknown backups

Each backup is classed by its exact backupType and source against a table per engine, never by substring:

  • PostgreSQL (2024-08-01 enum): Full is automatic; Customer On-Demand is manual.
  • MySQL (free text in 2023-12-30): FULL from Automatic is automatic, except that a backup named customer_... is manual whatever its labels. Microsoft's published on-demand example (BackupPut.json) labels its backup FULL / Automatic, exactly like a scheduled one; only its name sets it apart.
  • Any other label is unknown and never counts as automatic. servers_with_unknown_backup_types shows such servers, and each lists the labels, so a new value appears in the evidence instead of being misread.

Read replicas take no backups and are counted, not judged. A restored server with no creation date counts in restored_servers_date_unknown, never as recent; only restored servers that still exist are seen.

Settings

backup_window_hours (default 26), required_retention_days (a minimum, unset by default), restore_lookback_days (unset by default) and the shared resource_exclusions. Unset values leave their counts null rather than judged against a number nobody adopted. Retention is compared with <, so a longer retention meets the minimum; unknown retention is counted apart.

Failure handling

Uses run_subscription. Any unreadable provider, server list or backup list is recorded in api_failures and fails the run; backup verdicts are null, not 0, unless every backup server's backups were read. An unregistered provider has no servers, which is known and does not fail the run.

Testing

  • 23 tests in tests/test_azure_flexible_server_backups.py, including backup classification against Microsoft's published examples (tests/fixtures/flexible_server_backups/, provenance in its README): the PostgreSQL list, the MySQL list, and the MySQL on-demand example, which must not count as automatic, plus a server whose only recent backup is that on-demand one.
  • test_contracts.py (the description and instructions are checked like every fetcher now), test_category_requires.py and all Azure and Entra tests pass. The tests need no Azure SDK.

Not yet run against a live tenant. A real MySQL on-demand /backups response should still confirm the customer_ rule.

@bruceweygandt
bruceweygandt force-pushed the feat/azure-database-backups branch from e98fb23 to 530f7a4 Compare November 17, 2026 03:59
@bruceweygandt
bruceweygandt force-pushed the feat/azure-database-backups branch from 530f7a4 to 69aa929 Compare November 17, 2026 17:42

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The collection logic is good. Read replicas are counted but not judged, manual backups are kept apart from automatic ones, and the counts stay null when a read fails. I ran the tests (15 passed), the contract tests and paramify list offline, and all pass. Three items need a fix before merge.

1. The description has 33 words. The limit is 30.

fetchers/azure/database_backups/fetcher.yaml:3-4

Problem. The plain-text format check in tests/test_contracts.py reports description has 33 words; at most 30. CI does not show this, because azure/database_backups is not in TEXT_FORMAT_FETCHERS.

Why this is a problem. The description does not follow the Azure format rules. The check will fail when it applies to all fetchers.

Fix. Shorten the description. For example (26 words): "Shows whether each PostgreSQL and MySQL flexible server completed a recent automatic backup, how long backups are kept, whether they are geo-redundant, and recent point-in-time restores." Then add "azure/database_backups" to TEXT_FORMAT_FETCHERS.

2. The name is wider than what the fetcher covers

fetcher.yaml:1, fetcher.yaml:77-78

Problem. The name azure_database_backups and the evidence set "Azure Database Backups" do not tell the reader that only PostgreSQL and MySQL flexible servers are covered.

Why this is a problem. An assessor will expect Azure SQL Database to be included. No fetcher on main reports Azure SQL Database backups yet. If one is added later, it needs a different name next to this one. After merge, a rename also changes the evidence set reference id.

Fix. Choose one before merge. Either rename the fetcher (for example azure_flexible_server_backups, "Azure PostgreSQL and MySQL Backups"), or keep the name and plan to add Azure SQL Database to this fetcher.

3. Automatic and manual backups are told apart by matching substrings

fetcher.py:64-68

Problem. is_manual looks for the words ondemand, customer, manual or user in backupType and source. The test values are not from a real response, and the fetcher has not run against a real tenant.

Why this is a problem. If an engine reports an on-demand backup with a different value, that backup counts as automatic. A recent manual backup then hides broken automatic backups, which is the case this fetcher is designed to find.

Fix. Make one on-demand backup on a PostgreSQL server and on a MySQL server in a sandbox. Save the two /backups responses as test fixtures. Compare against the exact enum values instead of substrings, and count an unknown value as unknown, not as automatic.

Smaller items

  • fetcher.yaml:123. restored_servers_within_lookback counts only restored servers that still exist. A restore test often deletes its server at the end, and then the count is 0. Please state this in the field description.
  • fetcher.py:122-123. servers_retention_not_required uses !=. A server that keeps backups longer than required is counted, and so is a server with unknown retention. If you mean "at least the required days", use <, rename the field to show the direction, and count unknown retention separately.
  • fetcher.py:54. state is recorded but not used. A stopped server takes no backups, so it appears without a recent backup and the reason is not shown. Consider a stopped_servers count.
  • fetcher.py:153-157. The API version comes from the flexibleServers resource type, but the fetcher also uses it for flexibleServers/{name}/backups. Please choose a version that both resource types list.
  • fetcher.yaml:120. azure_postgresql_configuration and azure_mysql_configuration already report geo-redundant backup. If the duplicate is intended, please say so in the PR description.

…er_backups; retention minimum, stopped servers, shared API version
@bruceweygandt bruceweygandt changed the title feat(azure): add azure_database_backups fetcher feat(azure): add azure_flexible_server_backups fetcher Nov 17, 2026
@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Thanks. Items 1 and 2 and the smaller items are fixed in the latest commit. Item 3 needs sandbox data, so I'll push it separately.

1. Description over 30 words. Now your 26-word version, and azure/flexible_server_backups is in TEXT_FORMAT_FETCHERS. The format checks pass.

2. The name was wider than the coverage. Renamed before merge: azure_flexible_server_backups, evidence set EVD-AZURE-FLEXIBLE-SERVER-BACKUPS ("Azure PostgreSQL and MySQL Backups"), and the settings are now AZURE_FLEXIBLE_SERVER_BACKUP_WINDOW_HOURS, AZURE_FLEXIBLE_SERVER_BACKUP_RETENTION_DAYS and AZURE_FLEXIBLE_SERVER_RESTORE_LOOKBACK_DAYS. Azure SQL Database can get its own fetcher later without a name clash.

3. Manual vs automatic by substring. Agreed, and not fixed yet. I'm making one on-demand backup on a PostgreSQL server and one on a MySQL server in a sandbox, then I'll save both /backups responses as fixtures, match the exact enum values, and count an unrecognized value as unknown rather than automatic. That will come as a follow-up commit on this PR.

Smaller items

  • Restored servers that were deleted: the restored_servers_within_lookback description now says only servers that still exist are seen, so a restore test that deletes its server does not count.
  • Retention direction: now servers_retention_below_required, counted with <, so a longer retention meets the requirement. Servers with no retention returned are counted in the new servers_retention_unknown. required_retention_days is described as the minimum. Tested.
  • Stopped servers: new stopped_servers count, so a stopped server's missing backup has a visible reason. Tested.
  • API version for /backups: the version is now the newest one listed for both flexibleServers and flexibleServers/backups (falling back to the flexibleServers list when the backups type is not listed). Tested.
  • Overlap with the configuration fetchers: intended. servers_geo_redundant_backup is repeated here so all of the backup evidence (recency, retention, off-site copy, restores) sits in one evidence set. I'll note it in the PR description.

Tests: 18 passed (3 new). Contract tests: 884 passed. Ruff is clean on the changed files.

…er engine; unknown labels never count as automatic
@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Item 3 is in the latest commit.

Manual vs automatic backups. is_manual and its substring matching are gone. backup_kind now compares the exact backupType and source (case aside) against a table per engine and returns automatic, manual or unknown. Unknown is never counted as automatic, so an unrecognized backup cannot hide broken automation.

  • PostgreSQL: backupType is a defined enum in the 2024-08-01 spec. Full is automatic and Customer On-Demand (source Customer Initiated) is manual.
  • MySQL: backupType and source are free text in the 2023-12-30 spec, so only FULL from Automatic is treated as automatic. No on-demand value is assumed yet: Microsoft's published on-demand example reports source: Automatic, so I want a real response before mapping one. Until then anything else is unknown.

The fixtures are the published list examples from Azure/azure-rest-api-specs (provenance in tests/fixtures/flexible_server_backups/README.md), since I don't have a sandbox with flexible servers. A new servers_with_unknown_backup_types count shows when a server has backups with labels no rule matches, and each server lists those labels, so a new value shows up in the evidence instead of being misread.

Tests: 21 passed (4 new, 1 removed with is_manual). Contract tests: 884 passed.

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes still needed: one item. A MySQL on-demand backup can still count as automatic.

Fixed

    1. Description: fixed. It has 26 words and is in TEXT_FORMAT_FETCHERS.
    1. Name: the rename is complete. The folder, fetcher.yaml name, output file, settings, evidence set, tests, fixtures and CHANGELOG all use the new name, and git grep finds no database_backups left.
    1. Labels: fixed for PostgreSQL, which now matches the exact enum. An unrecognized label is never counted as automatic. MySQL is only partly fixed, see below.
  • Restored servers that were deleted, retention compared with < plus servers_retention_unknown, stopped_servers, and the shared API version: fixed and tested.

Still to fix

1. Must fix: a MySQL on-demand backup still counts as automatic

fetchers/azure/flexible_server_backups/fetcher.py:85

Problem. Microsoft's published MySQL on-demand example (BackupPut.json, 2023-12-30, in the same folder as your list fixture) returns backupType: FULL, source: Automatic. The table maps exactly that pair to automatic. The example's name is customer_20220507t073755_<guid>_mybackup. The automatic backups in BackupsListByServer.json are named daily_….

Why this is a problem. On MySQL, a recent on-demand backup still sets backup_within_window on a server whose automatic backups are broken. That is the case item 3 was about. servers_with_unknown_backup_types does not show it either, because the pair matches a rule.

Fix. Until a live response shows otherwise, count a MySQL backup whose name starts with customer_ as manual, or as unknown, whatever its labels. Add the BackupPut.json response as a fixture, with a test that it is not counted as automatic.

Before merge, please merge main. The branch conflicts in CHANGELOG.md and tests/test_contracts.py, so CI has not run on the last two commits. Please also update the PR description: it still names azure_database_backups.

@bruceweygandt

Copy link
Copy Markdown
Contributor Author

Thanks. The MySQL item is fixed in the latest commit, and main is merged.

1. A MySQL on-demand backup still counted as automatic. A MySQL backup whose name starts with customer_ is now manual, whatever its backupType and source. I chose manual rather than unknown because the name carries the user's own backup name (..._mybackup), which only an on-demand backup has. BackupPut.json (2023-12-30) is a fixture, with provenance in the fixtures README, and two tests use it: it is classed as manual despite FULL / Automatic, and a server whose only recent backup is that on-demand one shows no automatic backup within the window. The /backups line in the instructions explains the rule. A real response should still confirm it, and I've asked for one.

Merging main. CHANGELOG.md keeps both entries, and tests/test_contracts.py takes main's version, since TEXT_FORMAT_PENDING replaced the allow-list and the fetcher is now checked automatically.

PR description. Rewritten for azure_flexible_server_backups; it no longer names azure_database_backups.

Tests: 23 passed (2 new). Contract, category and all Azure and Entra tests: 6,492 passed. No Azure SDK needed.

@bruceweygandt

Copy link
Copy Markdown
Contributor Author

We now have a live response. The customer ran a read-only GET flexibleServers/{server}/backups (api-version 2024-12-30) against a staging MySQL server in Azure Government: seven FULL / Automatic backups, and no provisioningState field.

One difference from Microsoft's examples matters here: live names use hyphens, daily-<time>-<id>, where the published examples use daily_.... If a live on-demand backup follows the same pattern, it would be customer-..., which the customer_ rule would miss, and it would count as automatic. So the rule now accepts both customer_ and customer-.

  • The live response is a fixture (mysql-backups-live-2024-12-30.json) with the subscription, resource group, server name and backup id replaced by placeholders; names, times and labels are as returned. Provenance is in the fixtures README.
  • New tests: all seven live backups are automatic and the server is within its window, with no provisioningState needed; and a customer- on-demand backup is manual.

The live list has no on-demand backup, so the customer prefix itself is still confirmed only by Microsoft's example.

Tests: 25 passed. Contract, category and all Azure and Entra tests: 6,494 passed. No Azure SDK needed.

@Controgon Controgon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. The MySQL item is fixed. CI was green on 8d666c7a; on e092d1c4 the tests are still running.

Fixed

    1. MySQL on-demand backups: a MySQL backup named customer_… is manual whatever its labels (fetcher.py:98). BackupPut.json is now a fixture, and two tests use it: one checks that it is classed as manual, and one checks that a server whose only recent backup is that on-demand one has no automatic backup within the window. The /backups line in the instructions explains the rule.
  • main is merged, and the merge keeps main's TEXT_FORMAT_PENDING block.
  • The PR description now uses the new name and has the geo-redundancy overlap note.
  • e092d1c4 also counts customer-… names as on-demand, and adds a live MySQL backup list (identifiers replaced) with two tests. I read it; it doesn't change anything above.

@Controgon
Controgon merged commit e5251ee into main Nov 17, 2026
16 checks passed
@Controgon
Controgon deleted the feat/azure-database-backups branch November 17, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants