Migrate to DB format 0.47.0-alpha.2 and register column families (#829)

[AcoPiper]

Summary

Moves the database format from 0.47.0-alpha.1 to 0.47.0-alpha.2, registers the two column families the core-component registry and the operation-attempt ledger were already written against, and converts stored agent and external-service values to the layout carrying install state.

Following the documented alpha-to-alpha convention, this extends the existing migrate_0_46_to_0_47 entry rather than adding an alpha.1-to-alpha.2 one: its requirement becomes >=0.46.0,<0.47.0-alpha.2 and its target 0.47.0-alpha.2, so one migration accepts every supported 0.46.x and alpha.1 source. Its old list-and-create_cf body is replaced by a single open of the pinned 39-name MAP_NAMES_V0_47_ALPHA_2 literal with create_missing_column_families(true), which idempotently creates whichever of customer deletion jobs, core components, and operation attempts is absent. That list is a literal-string snapshot rather than live MAP_NAMES, so a later rename or format bump cannot change what this historical migration creates.

CORE_COMPONENTS and OPERATION_ATTEMPTS join MAP_NAMES (37 → 39), with StateDb openers and public Store::core_component_map() / Store::operation_attempt_map() accessors. The now-false registration-deferment comments and #[allow(dead_code)] annotations are gone, and both table test helpers open MAP_NAMES directly instead of pushing a duplicate name. The new tables start empty; nothing is backfilled.

Values in the agents and external services families are migrated in place, keys and existing fields preserved, with installed_version = None, installed_commit = None, lifecycle = 0, and bound_addrs = [] for rows that lack them. Four migration-only layouts are pinned in migration_structures.rs — old and current for each record type — reusing the live component types, with lifecycle stored as u8 so an unrecognized index still reads back. Values are decoded and encoded with bincode::DefaultOptions, the varint encoding src/tables.rs uses, not the fixint event helpers. The current layout is probed first, so a current-shaped row keeps its exact stored bytes and only an old-shaped row is rewritten; a row matching neither layout aborts the migration reporting the raw key and both decoding errors, as migrate_event_country_codes does. Rewrites are batched at EVENT_MIGRATION_BATCH_SIZE through a write_migration_batch helper shared with the country-code migration.

For retry safety, map_names_for_existing_format now returns an owned list derived from existing_map_names, so every intermediate migration opens exactly the non-default families physically present rather than choosing among static lists or inferring a layout from a count. An interrupted run may hold 36 legacy families, all 37 of alpha.1, or those plus either or both new ones, and the retry runs through to the extended migration that repairs the rest. The three historical source layouts (MAP_NAMES_V0_42, MAP_NAMES_V0_43_TO_V0_46, and the new 37-name MAP_NAMES_V0_47_ALPHA_1) are pinned and #[cfg(test)]-gated so the non-test library carries no unused private constants; the alpha.2 target list stays in production because the migration opens it.

The migration body writes only inside data_dir/states.db, and the rollback-snapshot boundary is documented next to the migration chain. The unreleased changelog entries are amended in place rather than duplicated.

Closes #829

Test plan

• cargo fmt -- --check --config group_imports=StdExternalCrate passes
• cargo clippy --bins --tests --all-features -- -D warnings passes
• cargo test --all-features passes
• An alpha.1 fixture built from the pinned 37-name family list, holding old-layout agents and external services with varied preserved fields, migrates: defaults applied, keys and former fields preserved, both new families present and empty, both version markers at alpha.2, and both Store accessors usable
• Fixtures for 0.42.0, 0.43.x, 0.44.x, 0.45.x, and 0.46.x run the full chain, and a stale marker after the customer-deletion-jobs family already exists still completes, with intermediate migrations reopening the exact physical family set
• Stale-marker retries with neither new family, core components only, operation attempts only, and both present all complete without a duplicate-family or open failure
• Mixed old/current values migrate: old rows convert, current rows stay byte-identical — including a current-shaped value carrying an unrecognized lifecycle byte, which reads back as Unknown
• A corrupt or foreign value in each family fails the migration with the raw key and both schema errors and is left unwritten
• Representative Agent and ExternalService values serialized through the live table Value encoding deserialize into the pinned current-layout structs field-for-field and re-encode to identical bytes, guarding against live-type and private-layout drift
• The migration body leaves every file under the data dir outside states.db unchanged

[codecov]

Codecov Report

❌ Patch coverage is 99.55090% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.50%. Comparing base (b81c117) to head (beb6eca).

Files with missing lines	Patch %	Lines
src/migration.rs	99.51%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #861      +/-   ##
==========================================
+ Coverage   84.26%   84.50%   +0.24%     
==========================================
  Files          92       92              
  Lines       36331    36927     +596     
==========================================
+ Hits        30613    31207     +594     
- Misses       5718     5720       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[AcoPiper]

[Reviewer Round 1]

No findings. The alpha compatibility range and single extended migration correctly cover both 0.46.x and alpha.1 sources, while the pinned 39-family list with missing-family creation makes retries safe without coupling this historical migration to live table constants (src/migration.rs:219-279, src/migration.rs:551-598). Agent and external-service values are decoded with the same strict varint options as table storage, probe the current layout before converting the old one, preserve current bytes, and include the key plus both decoder errors when neither schema matches (src/migration.rs:282-340). The migration-only layouts retain the exact field order and defaults required by the issue (src/migration/migration_structures.rs:113-208). The tests meaningfully exercise alpha.1/default conversion, partial-column-family retry states, current bytes with an unknown lifecycle index, corrupt values, layout drift, and the states.db write boundary. PR hygiene is also complete: it closes #829 and includes a test plan.

[AcoPiper]

[Review Verdict Round 1: APPROVED]