Migrate to DB format 0.47.0-alpha.2 and register column families

[sehkone]

Migrate to DB format 0.47.0-alpha.2 and register column families

Context

The current crate and database format are 0.47.0-alpha.1, with COMPATIBLE_VERSION_REQ = >=0.47.0-alpha.1,<0.47.0-alpha.2 and 37 registered RocksDB column families. Its 0.46 → 0.47 migration creates customer deletion jobs. VERSION is written only after the complete migration chain succeeds, so a retry can have an older marker than the column families physically present.

Agent and ExternalService values now persist installed_version, installed_commit, lifecycle, and bound_addrs; values written before those fields cannot decode through the current table layouts. The core-component registry and operation-attempt ledger are implemented, but their core components and operation attempts column families are deliberately absent from MAP_NAMES. That deferral is essential: StateDb::open and StateDb::reboot create every family in MAP_NAMES, while migrate_data_dir returns immediately for an already-compatible marker. Registering either name without this format transition would silently alter an otherwise-compatible directory without recording a version change. Operation-attempt secondary indexes use prefixed key spaces inside operation attempts, so they require no additional families.

This remains an alpha transition rather than finalizing 0.47.0: the storage format is still changing during prerelease. A later stable finalization must follow the documented alpha-to-stable migration rule and consolidate all supported alpha formats.

Scope

• Bump Cargo.toml to 0.47.0-alpha.2 and set COMPATIBLE_VERSION_REQ to >=0.47.0-alpha.2,<0.47.0-alpha.3. Follow the documented alpha-to-alpha format-change convention by extending the existing migrate_0_46_to_0_47 entry, not by adding an alpha.1-to-alpha.2 entry: its requirement becomes >=0.46.0,<0.47.0-alpha.2, its target becomes 0.47.0-alpha.2, and it handles both released 0.46.x and alpha.1 databases. Replace the current list_cf/already_created/explicit create_cf(CUSTOMER_DELETION_JOBS) body. The extended migration opens a pinned 39-name MAP_NAMES_V0_47_ALPHA_2 list with create_missing_column_families(true), which idempotently creates customer deletion jobs, core components, and operation attempts when absent. This production-used list is an explicit literal-string snapshot, like the existing pinned lists, and must not be built from live family-name constants or replaced by live MAP_NAMES; a later rename or format bump therefore cannot cause this historical migration to create an unrelated family.
• Add exactly CORE_COMPONENTS (core components) and OPERATION_ATTEMPTS (operation attempts) to MAP_NAMES, increasing it from 37 to 39 names. Add the matching StateDb openers plus public Store::core_component_map() and Store::operation_attempt_map() accessors. Remove the temporary #[allow(dead_code)] annotations and registration-deferment comments that become false, and update both table test helpers so they open MAP_NAMES directly instead of pushing a duplicate name. The new tables start empty; no registry or ledger rows are backfilled.
• Migrate all values in the AGENTS and EXTERNAL_SERVICES column families, preserving raw keys and every existing field. Values without the install-state fields receive installed_version = None, installed_commit = None, lifecycle = 0, and bound_addrs = []. Do not walk Node records, which hold only references to these values.
• Pin four migration-only stored layouts in src/migration/migration_structures.rs: old and current layouts for each record type. The old Agent layout contains kind: AgentKind, status: AgentStatus, config: Option<AgentConfig>, and draft: Option<AgentConfig>; its current layout adds installed_version, installed_commit, lifecycle: u8, and bound_addrs. The old ExternalService layout contains kind: ExternalServiceKind, status: ExternalServiceStatus, and draft: Option<ExternalServiceConfig>; its current layout adds the same four fields. Reuse the current live component types (AgentKind, AgentStatus, AgentConfig, ExternalServiceKind, ExternalServiceStatus, and ExternalServiceConfig), following the existing migration-structure convention; the current-layout drift test below guards that deliberate coupling. The u8 is required because table storage uses the lifecycle variant index and must continue to tolerate an unrecognized index.
• Decode and encode these table values with bincode::DefaultOptions, the varint encoding used by src/tables.rs; do not use the fixint event-record helpers. Probe the applicable current layout first. A current-shaped row remains byte-identical; only an old-shaped row is decoded and rewritten with defaults. If both current- and old-layout decoding fail, abort the migration without skipping or rewriting the row, reporting the raw key and both decoding errors together, following migrate_event_country_codes.
• Keep the historical column-family layouts explicit: retain the two 36-name source lists, MAP_NAMES_V0_42 and MAP_NAMES_V0_43_TO_V0_46, and add the pinned 37-name MAP_NAMES_V0_47_ALPHA_1 list as snapshots for fixtures. Gate all three test-only source-layout constants with #[cfg(test)] so the non-test library has no unused private constants; retain the pinned alpha.2 target list in production because the extended migration opens it. Refactor map_names_for_existing_format to return an owned list derived from existing_map_names, and make every intermediate migration that uses it open that exact non-default physical-family set. Do not choose among static lists or infer a layout from a count: an interrupted migration can physically hold 36 legacy families, all 37 alpha.1 families, or either one or both new families. This lets an earlier migration such as migrate_event_country_codes reopen a physical 38-family state safely instead of requesting the 39-name target list and failing before the extended migration can repair it.
• Batch agent and external-service rewrites using EVENT_MIGRATION_BATCH_SIZE (100) and the established bounded RocksDB write pattern. The migration body may write only inside data_dir/states.db; document that rollback snapshots cover that database only. The surrounding migration framework continues to update the two VERSION files after a successful chain, which is distinct from the migration body.
• Amend, rather than duplicate, the relevant unreleased changelog entries: replace the closing Store-reachability sentence in both the CoreComponent and OperationAttempt entries; replace the agent/external-service statement that earlier records are not migrated; and update the breaking format entry that currently says 0.47.0-alpha.1 and only mentions customer deletion jobs.

Acceptance criteria

• The crate is 0.47.0-alpha.2, the compatible range is >=0.47.0-alpha.2,<0.47.0-alpha.3, and the single extended migrate_0_46_to_0_47 migration accepts every supported 0.46.x and alpha.1 source format and targets alpha.2. Its old manual customer-deletion-family branch is replaced by one create_missing_column_families(true) open over the pinned 39-name alpha.2 list, not live MAP_NAMES.
• MAP_NAMES contains 39 names, adding only core components and operation attempts; both Store accessors work; obsolete registration comments, dead-code allows, and duplicate test-helper pushes are gone.
• An alpha.1 database with pre-install-state agent and external-service values migrates successfully. Keys and former fields are preserved, the four fields have their defaults (None, None, 0, and []), both new tables are empty and usable, and both version markers become alpha.2.
• The migration has separate old and current layouts for both Agent and ExternalService, uses DefaultOptions for table values, and recognizes current-shaped rows first, leaving their stored bytes unchanged while converting only old-shaped rows. A row matching neither layout fails the migration with its key and both decode errors.
• The two 36-name pre-alpha.1 source layouts (0.42 and 0.43 through 0.46) and the 37-name alpha.1 source layout are pinned and test-gated; the 39-name alpha.2 target is pinned and used by production migration code. Every intermediate migration opens an owned list of the non-default families physically present, obtained from existing_map_names, rather than a count-derived or static list. Retries therefore succeed with legacy, alpha.1, and partial or complete new-family states, including exactly one new family.
• The migration body writes only under states.db, and that rollback-snapshot boundary is documented next to the migration chain.
• The amended unreleased changelog accurately describes the reachable tables, data migration, and alpha.2 format transition without duplicate feature entries.
• cargo fmt -- --check --config group_imports=StdExternalCrate, cargo clippy --bins --tests --all-features -- -D warnings, and cargo test --all-features pass.

Constraints

• Register exactly two families. Do not add index families or redesign the existing core-component, operation-attempt, or secondary-index APIs.
• Preserve the exact historical 36-family 0.42 and 0.43-through-0.46 lists, the alpha.1 37-family list, and the alpha.2 39-family list for their defined formats. An interrupted intermediate migration must instead use its owned physical-family list; do not infer a schema from list_cf().len().
• Keep the migration forward-only. Rollback restores states.db from a pre-update snapshot and must separately restore both version markers; providing the public marker writer or rollback orchestration is not part of this issue.
• Do not infer historical installation state from hosts. Every converted value receives the documented defaults and is updated later by a status report.

Out of scope

• A public version-marker writer, snapshot/rollback orchestration, or changing snapshot coverage outside states.db.
• Stable 0.47.0 finalization and its alpha-consolidating migration range.
• New core-component or operation-attempt behavior beyond making the already-implemented tables available through Store.

Test plan

• Build an alpha.1 fixture from the pinned 37-name family list, containing old-layout agents and external services with varied preserved fields; migrate it and assert the defaults, preserved keys and values, both new families, both version markers, and both Store accessors.
• Build fixtures for every supported pre-alpha.1 source format — 0.42.0, 0.43.x, 0.44.x, 0.45.x, and 0.46.x — and for a stale marker after the customer-deletion-jobs family has appeared; run the full chain and prove that intermediate migrations reopen the exact physical family set.
• Build stale-marker fixtures with neither, only core components, only operation attempts, and both new families present; verify every retry completes without a duplicate-family or open failure.
• Build mixed old/current values, including a current-shaped value carrying an unrecognized lifecycle byte; assert that old rows convert, current rows remain byte-identical, and the migration completes. Add corrupt or foreign values in each family that match neither pinned layout; assert that migration fails with the raw key and both schema errors and leaves that value unwritten.
• Serialize representative Agent and ExternalService values through their table Value encoding, deserialize each with its pinned current-layout migration struct using DefaultOptions, and compare every field to guard against live component-type and private-layout drift.
• Verify the migration body does not modify data outside states.db, then run the required formatting, lint, and test commands.

Dependencies

Part of #831. The prerequisite record and table implementations are already present in the default branch. The separate public version-marker-writer work remains outside this issue and can rely on this format transition once it lands.

Pointers

• src/migration.rs — alpha-version compatibility examples, migration list, migrate_0_46_to_0_47, historical family lists, existing_map_names, map_names_for_existing_format, table-value migration patterns, and test fixtures.
• src/migration/migration_structures.rs — immutable migration-only stored layouts.
• src/tables.rs — family constants, MAP_NAMES, StateDb, and DefaultOptions table serialization.
• src/lib.rs — public Store map accessors.
• src/tables/agent.rs and src/tables/external_service.rs — current persisted layouts and lifecycle index encoding.
• src/tables/core_component.rs and src/tables/operation_attempt.rs — current table openers, obsolete registration-only test setup, and the ledger's shared-family indexes.

[AcoPiper]

Verification summary

• Delivery: COMPLETE — the merged code delivers what the issue asked for. (the auditor's own verdict)
• Extra changes: EXTRA_CHANGES — the merged code also changed things the issue did not ask for. (the auditor's own verdict)
• Defects: NO_DEFECT — the merged pull requests left no defect of their own behind. (the auditor's own verdict)

Delivery decides whether this issue is closed or reopened. Defects widens what the follow-up filing considers. Extra changes gates nothing — it is an observation for a human; the report names what went beyond the issue.

Implementing pull requests, at the merge commits this audit read:

• Add the operation_attempt ledger record and idempotency-keyed CRUD (#827) #853 — bc3190a
• Add operation_attempt secondary indexes, expiry sweep, and retention (#828) #857 — 6feedf3
• Protect operation attempt iteration (#858) #860 — b81c117
• Migrate to DB format 0.47.0-alpha.2 and register column families (#829) #861 — 34bb709 (the tree the audit read)

The full report — the assessment included verbatim — stays local, at /Users/aco/.agentcoop/runs/aicers/review-database/verify/ODI5/report.md.