Repository navigation
feat(migrations): complete the reusable database upgrade framework - #2168
Open
Zxf-xufeng wants to merge 107 commits into
Open
Zxf-xufeng wants to merge 107 commits into
Zxf-xufeng wants to merge 107 commits into
Conversation
Implement issue01 of the Source evidence declarations specification. Bind trusted native and remote declarations to exact Source material and retain per-revision Atomic Memory snapshots in the existing transaction. Preserve neutral fallback, exact identities, scope authorization and public retrieval contracts. Add versioned p0004 migrations and SQLite persistence regressions. Refs #1975. Full suite: 5255 passed, 431 skipped, 7 failed; introduced observability adapter failures fixed and its file passes 18 tests. Topic Memory vector failure also reproduces at the unchanged baseline.
Cover persisted declarations, scope isolation, historical neutrality, synthetic model trust claims, and unchanged recall consumers on SQLite and OceanBase.
Preserve the published p0004 artifact handle revision and introduce evidence as p0005. Retain both schema intents, recompute exact SQLite fingerprints, and update MySQL migration head expectations.
Retain native Source identity independently of adapter aliases and mutable description defaults, preserve legacy read/replay behavior, and keep serialized business fields intact. Capture evidence at the shared Atomic Memory body-commit boundary, including restoration revisions, without retroactively attesting state-only writes. Refs #1975.
Check native serialization compatibility at acceptance without blocking valid captures or Memory writes. Keep previously accepted incompatible declarations neutral while preserving exact material and digest checks. Reject legacy identities that cannot be decoded exactly instead of supplying a default Definition version.
Preserve Source evidence verification and upstream revision metadata initialization in Atomic Memory projection rebuild.
Reject native serialized material that omits an identity field before journal allocation or persistence, rather than accepting Sources that later fail evidence capture. Preserve strict retained identity checks.
Validate serialized identity values as well as key presence before persistence, matching historical evidence validation. Cover round-trippable codecs for all three identity fields without weakening retained-material checks.
Preserve exact evidence inspection alongside upstream bounded list queries and reuse its authorized read snapshot lifecycle.
Canonicalize worker envelope keys and their manifest without changing native persisted schemas. Read retained canonical identities before consulting current adapter aliases, preserving neutral historical evidence.
Only trust retained canonical identity keys when the registered model does not claim those spellings for business fields. Keep headerless business-aliased material on the declared decoder, and format the regression file with the pinned hook version.
Resolve referenced root schemas before canonicalizing transport identity keys while preserving nested serialization aliases. Cover manifest registration, observation submission, and SQLite readback.
Preserve Source evidence persistence alongside upstream Atomic Memory startup vector validation, including both contracts in the observability test index.
Keep remote manifests and payloads on the canonical field-name contract so existing Definition fingerprints remain stable. Preserve native historical identity and retained alias spellings across upgrades, replay, and rebuild.
Generate remote transport schemas with the same alias policy as their payloads so existing serialize_by_alias Definitions keep their fingerprint and observations after upgrade. Recover explicit canonical historical versions beyond version 1 instead of handing accepted material to a newer aliased Definition.
27 tasks
Zxf-xufeng
changed the title
Freeze the three Code tables captured from installed 1.3.0 and extend the existing complete Server chain with their exact schema. Preserve graph rows on upgrade, initialize them for fresh Code-enabled storage, and validate them at maintenance and runtime admission. Include the extension in backup scope and reject implicit managed-schema expansion through the standalone Code service. Add the sixth installed-release profile with immutable cache, graph-row, fingerprint and symbol checks. Initialize the HTTP/CLI integration fixture through the real Server before SDK writes and document the registered boundary in both locales. Constraint: All existing s1300/s1400/s1401 revision checksums remain unchanged. Vector configurations and enabling Code on already managed storage without its tables still require explicit transitions. Scope-risk: Managed seekDB Code initialization now validates rather than creating missing tables or indexes; unmanaged standalone Code behavior is preserved. Tested: Real seekDB migration/admission 3 passed, actual HTTP/CLI sync/restart 1 passed, related framework/release/CI suites 132 passed, full Linux Python 3.11 Ty and scoped hooks passed. Installed-release acceptance remains a separate gate.
Use the automatic recovery point for both installed SQLite profiles, restore it to an independent database, compare the complete source schema and row digests, and verify source readback followed by another target upgrade. Preserve separate Code caches and the original backup checksum. Record the bounded SQLite proof separately from remote-backend restoration. Constraint: Existing revision resources and production runtime are unchanged. Scope-risk: The installed acceptance suite performs extra SQLite lifecycle work. Tested: Installed 1.3.0 and 1.4.1.dev194+g35ed35cb2; both SQLite profiles 11 passed, zero skips; missing-backup red case failed as expected. Tested: Upgrade gate/backup suites 75 passed, 2 existing platform skips; scoped prek and full Linux Python 3.11 Ty passed.
Record VECTOR KEY columns and algorithm parameters as an index instead of a synthetic column. Reject duplicate, unknown and malformed parameter options. Preserve the registered no-vector layouts and fail-closed optional-profile policy. Constraint: No historical revision resources or checksums change; vector upgrade profiles remain disabled. Scope-risk: Normalized inventories of previously unsupported vector layouts change to reflect actual native indexes. Tested: Six regression cases failed before the correction. Full schema suite including real seekDB reflection/recovery: 58 passed, zero skips. Tested: Server baseline, full migration and upgrade-gate suites: 110 passed, zero skips. Full Linux Python 3.11 Ty and scoped hooks passed.
Select frozen SQLite and native vector layouts from the historical embedding profile and preserve embeddings, Topic retrieval shape and existing authority. Register SQLite helper backfill and transactional triggers in maintenance; managed startup rejects profile drift and missing helpers without repair. Constraint: Keep existing revision resources and no-vector checksums unchanged. Scope-risk: Installed vector release acceptance remains required; no full release support is declared and partial processing/projection bundles remain rejected. Tested: SQLite/full-Server/baseline suites 56 passed; actual seekDB/OceanBase vector migrations 4 passed; migration CLI/release suites 141 passed (2 existing async teardown warnings); affected Server E2E 62 passed/28 backend skips. Tested: Full Python 3.11 Ty, scoped prek hooks and git diff --check passed.
Zxf-xufeng
changed the title
Require all twelve SQLite, seekDB and OceanBase combinations with Code and vector storage enabled or disabled. Seed historical Atomic and Topic embeddings through an installed 1.3.0 Server, compare exact embedding digests without re-embedding, and verify semantic reads and fresh-storage equivalence through the installed target. Cover profile drift, missing SQLite helpers and actual SQLite backup restoration without relaxing runtime admission or release policy. Constraint: preserve frozen revision checksums and require zero skipped upgrade phases. Scope-risk: native failures remain release-blocking; no complete acceptance claim. Tested: pytest tests/test_upgrade_release_gate.py -q (62 passed); pytest tests/test_upgrade_publication.py -q (8 passed); scoped Ruff check/format, scoped prek, Python 3.11 Ty, git diff --check.
Prepare the complete registered Server schema before injected access stores write fixture tables, selecting the exact injected embedding profile. Initialize the remote vector Server through the same explicit maintenance path. Allow the OceanBase clean migration suite sixty minutes and its job one hundred minutes. The preceding run passed forty-eight cases before the forty-five-minute step budget expired; per-node process deadlines and all assertions remain. Constraint: Managed startup continues to reject partial or unknown schemas. Scope-risk: Test fixture setup and CI budgets only; no runtime admission change. Tested: Reproduced unknown_baseline before the fix; affected SQLite tests 53 passed; actual OceanBase 3 passed with zero skips; scoped Ruff, format, Python 3.11 Ty and YAML hooks passed. Full Ty reports the existing unrelated fixture verify.py import of task.
Keep managed admission checks in an explicit read snapshot. Unmanaged bootstrap retains BEGIN IMMEDIATE and rechecks schema registration after obtaining it. Regression coverage pauses real startup validation while an existing WAL runtime commits a Scope, including the Topic worker startup path. Constraint: Preserve processing, Atomic Memory and Topic readiness checks. Scope-risk: Read snapshots retain SQLite journal-mode locking semantics; no arbitrary-scale startup latency claim. Tested: Managed startup regression fails twice before the fix and passes afterward; 52 related and 69 embedding/profile/migration cases pass. Scoped Ruff, format and Ty Python 3.11 pass.
Zxf-xufeng
marked this pull request as ready for review
October 23, 2026 10:07
Zxf-xufeng
requested review from
AlexStocks,
PsiACE,
frostming and
zhanghuidinah
as code owners
October 23, 2026 10:07
hidb4ai
reviewed
Oct 23, 2026
hidb4ai
left a comment
hidb4ai
left a comment
Contributor
There was a problem hiding this comment.
Reviewed 1f6c89ce4c7e. I did not confirm an additional implementation defect in this pass, but I would hold the merge until the required acceptance checks are resolved.
- The Upgrade release gate fails for the seekDB
full-server-no-vector,full-server-vector, andfull-server-vector-codeprofiles. Each retained native log recordsCREATE TABLE pc_schema_revisionreturning-6210; publicapplyreturnsinvalid_target. This leaves those upgrade paths unaccepted, without establishing the root cause in this PR's migration code. - The OceanBase Atomic Memory job completes its migration selection, then times out in the subsequent long-query step after ten minutes. That required acceptance still needs to complete.
Separately, I upgraded a disposable copy of the SQLite database created with the actual 1.3.0 wheel to s1401. Automatic backup, repeat apply without writes, and target application startup with historical Scope readback through TestClient passed. Local backend execution in this review covered SQLite; the seekDB and OceanBase findings above come from the retained CI evidence.
Allow complete Server setup within the aggregate long-query CI step. Eight healthy isolated cases can exceed the former ten-minute limit while each remains inside the existing three-minute process deadline. Constraint: Keep all assertions, the 180-second case limit and the 100-minute job limit. Scope-risk: CI aggregate execution budget only; runtime and migration resources are unchanged. Tested: Eight real OceanBase cases passed without skips in 665.9 seconds; YAML structure and git diff --check pass; Ty and other validators pass in the existing environment, while the hook wrapper reports JSON normalizations without a tracked JSON diff.
hireview001
reviewed
Oct 23, 2026
hireview001
left a comment
hireview001
left a comment
Collaborator
There was a problem hiding this comment.
- Resolve merge conflicts with the target branch.
Member
Author
|
Resolved the conflicts with master at 1da9974 and pushed 8f10c560. Both inline implementation findings have separate replies. The merge also requires registering storage already introduced on master. A new s1402 revision adds portable restore receipts and the four native DATETIME(6) columns while preserving the earlier revision resources/checksums. Ordinary managed startup remains read-only and rejects incomplete restores; the existing explicit archive-recovery entry can finish projections after layout and configuration checks. Validation: make check passed; the persistence/managed-maintenance/upgrade-policy selection passed 709 tests (102 skipped, including unavailable native backend cases); CLI/startup selection passed 47; final initialization/policy regressions passed 68; Compose launcher regressions passed 7. SQLite archive restoration was also exercised with and without vectors. These results overlap and are not a combined test count. The earlier native release-gate and long-query acceptance findings remain pending the CI results for this new commit. Local modeled tests do not establish seekDB or OceanBase release acceptance. GitHub now reports MERGEABLE. The new Upgrade release gate is queued. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue or RFC does this PR close?
Refs #2154 and RFC #1771. Includes the upgrade acceptance tests and release gates from #2181 for #2161. Completion of #2154 remains subject to the required acceptance evidence and review.
Rationale for this change
A populated PowerContext 1.3.0 Server database could not be adopted by the partial migration bundle or admitted by the target Server. This PR completes the reusable, registered upgrade path: inspect the historical database, build and apply a target-bound plan, validate schema and domain state, then allow service activation. Subsequent supported changes use the same registration and execution framework.
What changes are included in this PR?
Register the complete
s1300 → s1400 → s1401Server chain and exact historical layouts for SQLite, embedded seekDB and OceanBase. Register supported Code and vector configurations without replacing the shared revision chain.Integrate ordered domain preconditions, transformations, projection maintenance and validators. Preserve Candidate history, Artifact revisions, references, authorization, deployment identity, cursors and Processing ownership. Preserve historical embeddings without calling an embedding provider during migration.
Share full readiness checks between maintenance verification and managed runtime admission. Reject unknown layouts, incomplete projections, missing helpers and configuration drift; normal startup does not repair managed storage. Initialize supported fresh storage through the registered framework.
Coordinate stopped writers, dedicated maintenance connections, target locks, backups and durable recovery evidence. Retain the original recovery point across interruptions and distinguish database migration success from service activation failure.
Include ci(release): require positive upgrades from 1.3.0 on all backends #2181's actual installed 1.3.0-to-target-wheel acceptance and publication gates. Require twelve backend/configuration combinations: three backends, Code enabled or disabled, and vectors enabled or disabled. Validate old-startup rejection, repeated migration, historical data, authorization, Candidate operations, background processing, retrieval and fresh/upgraded equivalence.
Validate SQLite automatic-backup restoration and native service upgrade/recovery scenarios. Add OceanBase client-exit and observer-crash recovery coverage, keeping fixture-based recovery evidence distinct from installed-release and backup-restoration evidence.
Bind publication to reviewed persistence declarations and the accepted wheel/source-distribution set. Fail the gate for failed, skipped or missing required cases; validate source-distribution rebuilding before publishing the exact accepted artifacts.
Are there any user-facing changes?
--bundle serveris the complete default migration bundle. Partial bundles cannot establishserver_ready=true. Stop all writers and drain old queued/running Dream tasks before maintenance. Use explicit maintenance for remote database initialization and unsupported configuration transitions; do not add missing managed tables through startup.Support is limited to registered historical layouts and configuration profiles. Vector profiles bind the historical model, dimensions and retrieval configuration. Code profiles preserve their registered embedded tables or separate caches. Registration alone does not establish release support. Existing Source evidence is preserved; this PR does not introduce the independent Source evidence declaration feature.
The target Candidate routes use
/v1/candidates/*and require contract 2. The offline Memory archive has no public Artifact read path. Backup capacity and offline maintenance time are required; remote backup declarations do not prove successful restoration.The release policy remains
upgrade-required. This PR does not declare all supported upgrades accepted or authorize publication before the required gates pass.How was this change tested?
Current source:
04a33ffc4e64fd9f00f17b1b933dc932bef008d1. Its Mainand E2E workflows are pending.
Full acceptance of this source is not established.
The completed installed-release gate
tested source
747dac3e695f91bd31ab18d98b01590ee30a57f9/ merge1c4107c106ba46deb9a6c4f2b1763f301086d710with actual source 1.3.0 and target1.4.1.dev200+g1c4107c10. All four SQLite and four OceanBase vector/Code configurations passed 11 phases each.All four seekDB configurations passed two phases with nine public-apply setup errors each:
96 passed / 36 errors / zero failures / zero skips; gate failed. Each native seekDB log records
CREATE TABLE pc_schema_revisionwithreturn_code=-6210. All twelve reports and native logs are retained inartifact 12481187672.
Managed SQLite startup commit
04a33ffc4e64: both real WAL concurrency regressions fail withdatabase is lockedbefore the repair and pass afterward (normal Runtime and Topic worker). The relatedsuite passed 52 tests; embedding, SQLite profile and full-Server migration suites passed another 69.
Scoped Ruff, formatting and Python 3.11 Ty passed. Startup retains all readiness checks in a read snapshot;
unmanaged bootstrap still reserves the writer. This does not establish a whole-collection startup latency bound.
Current Head CI confirms the authorization/vector fixture repair: Python 3.11, 3.12, 3.13 and 3.14 E2E
jobs all passed, each with 963 passed / 409 skipped. These general matrices skip environment-dependent
cases and do not replace the separately required zero-skip backend and installed-upgrade gates.
Authorization/vector fixture repair
3ad35d628de6: the selected SQLite case reproducedunknown_baselinebefore thechange; affected SQLite tests passed 53 afterward. Three real OceanBase Topic/Atomic/Server vector cases passed
with zero skips on OceanBase CE 4.3.5.6. These are checkout tests, separate from installed-release acceptance.
Scoped Ruff, format, Python 3.11 Ty and YAML hooks passed. The full Ty hook reports an import diagnostic in
unchanged
tests/e2e/cross_host_work_continuity/fixtures/verify.py:23(from task import increment);it is not reported as a full check pass.
The completed source's actual OceanBase observer-crash job and native Windows/Linux/macOS service checks passed.
Frozen-history crash tests are distinct from installed-release crash or remote backup-restoration proof.
Remaining acceptance includes passing the exact current-artifact twelve-profile gate and completing the
promised interruption/recovery coverage. Remote restoration is not an acceptance prerequisite unless claimed. Independent review remains required. No patched native binding
containing seekdb#1499 has been tested, and remote backup restoration is not advertised as automatic.
AI usage statement
Implemented and validated with OpenAI Codex under maintainer direction.