Repository navigation
feat(azure): add azure_flexible_server_backups fetcher - #377
Conversation
e98fb23 to
530f7a4
Compare
530f7a4 to
69aa929
Compare
Controgon
left a comment
There was a problem hiding this comment.
The collection logic is good. Read replicas are counted but not judged, manual backups are kept apart from automatic ones, and the counts stay null when a read fails. I ran the tests (15 passed), the contract tests and paramify list offline, and all pass. Three items need a fix before merge.
1. The description has 33 words. The limit is 30.
fetchers/azure/database_backups/fetcher.yaml:3-4
Problem. The plain-text format check in tests/test_contracts.py reports description has 33 words; at most 30. CI does not show this, because azure/database_backups is not in TEXT_FORMAT_FETCHERS.
Why this is a problem. The description does not follow the Azure format rules. The check will fail when it applies to all fetchers.
Fix. Shorten the description. For example (26 words): "Shows whether each PostgreSQL and MySQL flexible server completed a recent automatic backup, how long backups are kept, whether they are geo-redundant, and recent point-in-time restores." Then add "azure/database_backups" to TEXT_FORMAT_FETCHERS.
2. The name is wider than what the fetcher covers
fetcher.yaml:1, fetcher.yaml:77-78
Problem. The name azure_database_backups and the evidence set "Azure Database Backups" do not tell the reader that only PostgreSQL and MySQL flexible servers are covered.
Why this is a problem. An assessor will expect Azure SQL Database to be included. No fetcher on main reports Azure SQL Database backups yet. If one is added later, it needs a different name next to this one. After merge, a rename also changes the evidence set reference id.
Fix. Choose one before merge. Either rename the fetcher (for example azure_flexible_server_backups, "Azure PostgreSQL and MySQL Backups"), or keep the name and plan to add Azure SQL Database to this fetcher.
3. Automatic and manual backups are told apart by matching substrings
fetcher.py:64-68
Problem. is_manual looks for the words ondemand, customer, manual or user in backupType and source. The test values are not from a real response, and the fetcher has not run against a real tenant.
Why this is a problem. If an engine reports an on-demand backup with a different value, that backup counts as automatic. A recent manual backup then hides broken automatic backups, which is the case this fetcher is designed to find.
Fix. Make one on-demand backup on a PostgreSQL server and on a MySQL server in a sandbox. Save the two /backups responses as test fixtures. Compare against the exact enum values instead of substrings, and count an unknown value as unknown, not as automatic.
Smaller items
fetcher.yaml:123.restored_servers_within_lookbackcounts only restored servers that still exist. A restore test often deletes its server at the end, and then the count is 0. Please state this in the field description.fetcher.py:122-123.servers_retention_not_requireduses!=. A server that keeps backups longer than required is counted, and so is a server with unknown retention. If you mean "at least the required days", use<, rename the field to show the direction, and count unknown retention separately.fetcher.py:54.stateis recorded but not used. A stopped server takes no backups, so it appears without a recent backup and the reason is not shown. Consider astopped_serverscount.fetcher.py:153-157. The API version comes from theflexibleServersresource type, but the fetcher also uses it forflexibleServers/{name}/backups. Please choose a version that both resource types list.fetcher.yaml:120.azure_postgresql_configurationandazure_mysql_configurationalready report geo-redundant backup. If the duplicate is intended, please say so in the PR description.
…er_backups; retention minimum, stopped servers, shared API version
|
Thanks. Items 1 and 2 and the smaller items are fixed in the latest commit. Item 3 needs sandbox data, so I'll push it separately. 1. Description over 30 words. Now your 26-word version, and 2. The name was wider than the coverage. Renamed before merge: 3. Manual vs automatic by substring. Agreed, and not fixed yet. I'm making one on-demand backup on a PostgreSQL server and one on a MySQL server in a sandbox, then I'll save both Smaller items
Tests: 18 passed (3 new). Contract tests: 884 passed. Ruff is clean on the changed files. |
…er engine; unknown labels never count as automatic
|
Item 3 is in the latest commit. Manual vs automatic backups.
The fixtures are the published list examples from Azure/azure-rest-api-specs (provenance in Tests: 21 passed (4 new, 1 removed with |
Controgon
left a comment
There was a problem hiding this comment.
Changes still needed: one item. A MySQL on-demand backup can still count as automatic.
Fixed
-
- Description: fixed. It has 26 words and is in
TEXT_FORMAT_FETCHERS.
- Description: fixed. It has 26 words and is in
-
- Name: the rename is complete. The folder,
fetcher.yamlname, output file, settings, evidence set, tests, fixtures and CHANGELOG all use the new name, andgit grepfinds nodatabase_backupsleft.
- Name: the rename is complete. The folder,
-
- Labels: fixed for PostgreSQL, which now matches the exact enum. An unrecognized label is never counted as automatic. MySQL is only partly fixed, see below.
- Restored servers that were deleted, retention compared with
<plusservers_retention_unknown,stopped_servers, and the shared API version: fixed and tested.
Still to fix
1. Must fix: a MySQL on-demand backup still counts as automatic
fetchers/azure/flexible_server_backups/fetcher.py:85
Problem. Microsoft's published MySQL on-demand example (BackupPut.json, 2023-12-30, in the same folder as your list fixture) returns backupType: FULL, source: Automatic. The table maps exactly that pair to automatic. The example's name is customer_20220507t073755_<guid>_mybackup. The automatic backups in BackupsListByServer.json are named daily_….
Why this is a problem. On MySQL, a recent on-demand backup still sets backup_within_window on a server whose automatic backups are broken. That is the case item 3 was about. servers_with_unknown_backup_types does not show it either, because the pair matches a rule.
Fix. Until a live response shows otherwise, count a MySQL backup whose name starts with customer_ as manual, or as unknown, whatever its labels. Add the BackupPut.json response as a fixture, with a test that it is not counted as automatic.
Before merge, please merge main. The branch conflicts in CHANGELOG.md and tests/test_contracts.py, so CI has not run on the last two commits. Please also update the PR description: it still names azure_database_backups.
…ackups # Conflicts: # CHANGELOG.md # tests/test_contracts.py
…l whatever its labels
|
Thanks. The MySQL item is fixed in the latest commit, and 1. A MySQL on-demand backup still counted as automatic. A MySQL backup whose name starts with Merging PR description. Rewritten for Tests: 23 passed (2 new). Contract, category and all Azure and Entra tests: 6,492 passed. No Azure SDK needed. |
…mer_; add a live MySQL backup list as a fixture
|
We now have a live response. The customer ran a read-only One difference from Microsoft's examples matters here: live names use hyphens,
The live list has no on-demand backup, so the Tests: 25 passed. Contract, category and all Azure and Entra tests: 6,494 passed. No Azure SDK needed. |
Controgon
left a comment
There was a problem hiding this comment.
Approved. The MySQL item is fixed. CI was green on 8d666c7a; on e092d1c4 the tests are still running.
Fixed
-
- MySQL on-demand backups: a MySQL backup named
customer_…is manual whatever its labels (fetcher.py:98).BackupPut.jsonis now a fixture, and two tests use it: one checks that it is classed as manual, and one checks that a server whose only recent backup is that on-demand one has no automatic backup within the window. The/backupsline in the instructions explains the rule.
- MySQL on-demand backups: a MySQL backup named
mainis merged, and the merge keepsmain'sTEXT_FORMAT_PENDINGblock.- The PR description now uses the new name and has the geo-redundancy overlap note.
e092d1c4also countscustomer-…names as on-demand, and adds a live MySQL backup list (identifiers replaced) with two tests. I read it; it doesn't change anything above.
Summary
Adds
azure_flexible_server_backups, an Azure evidence fetcher for Azure Database for PostgreSQL and MySQL flexible servers. It reports whether each server's automatic backups actually run (the newest completed automatic backup against a window), with manual (on-demand) backups reported apart so a recent manual backup cannot hide broken automation; backup retention against an optional minimum; geo-redundant backup; stopped servers; and servers created by point-in-time restore.Evidence set:
EVD-AZURE-FLEXIBLE-SERVER-BACKUPS(Azure PostgreSQL and MySQL Backups). KSIs: KSI-RPL-ABO, KSI-RPL-ARP, KSI-RPL-TRC, KSI-SVC-RUD.The name is scoped to flexible servers on purpose, so Azure SQL Database can get its own fetcher later without a clash.
azure_postgresql_configurationandazure_mysql_configurationalready report backup settings, including geo-redundant backup;servers_geo_redundant_backupis repeated here so all backup evidence (recency, retention, off-site copy, restores) sits in one evidence set.How it collects
Resource Manager only, with Reader on the subscription:
GET providers/Microsoft.DBforPostgreSQLandproviders/Microsoft.DBforMySQL: registration, and the newest stable API version listed for bothflexibleServersandflexibleServers/backups(falling back to theflexibleServerslist), since Azure Government lags.GET .../flexibleServersfor each engine: retention, geo-redundant backup, earliest restore point, state, replication role and restore source.GET flexibleServers/{server}/backupsper server that is not a read replica.Automatic, manual and unknown backups
Each backup is classed by its exact
backupTypeandsourceagainst a table per engine, never by substring:Fullis automatic;Customer On-Demandis manual.FULLfromAutomaticis automatic, except that a backup namedcustomer_...is manual whatever its labels. Microsoft's published on-demand example (BackupPut.json) labels its backupFULL/Automatic, exactly like a scheduled one; only its name sets it apart.servers_with_unknown_backup_typesshows such servers, and each lists the labels, so a new value appears in the evidence instead of being misread.Read replicas take no backups and are counted, not judged. A restored server with no creation date counts in
restored_servers_date_unknown, never as recent; only restored servers that still exist are seen.Settings
backup_window_hours(default 26),required_retention_days(a minimum, unset by default),restore_lookback_days(unset by default) and the sharedresource_exclusions. Unset values leave their counts null rather than judged against a number nobody adopted. Retention is compared with<, so a longer retention meets the minimum; unknown retention is counted apart.Failure handling
Uses
run_subscription. Any unreadable provider, server list or backup list is recorded inapi_failuresand fails the run; backup verdicts are null, not 0, unless every backup server's backups were read. An unregistered provider has no servers, which is known and does not fail the run.Testing
tests/test_azure_flexible_server_backups.py, including backup classification against Microsoft's published examples (tests/fixtures/flexible_server_backups/, provenance in its README): the PostgreSQL list, the MySQL list, and the MySQL on-demand example, which must not count as automatic, plus a server whose only recent backup is that on-demand one.test_contracts.py(the description and instructions are checked like every fetcher now),test_category_requires.pyand all Azure and Entra tests pass. The tests need no Azure SDK.Not yet run against a live tenant. A real MySQL on-demand
/backupsresponse should still confirm thecustomer_rule.