feat(memory): implement independent atomic memory

[frf12]

Which issue or RFC does this PR close?

Implements RFC #1809 and follows the retrieval contract in RFC #1803.

Rationale for this change

Memory collections couple unrelated entries to one revision and write boundary. This change makes each fact, preference or decision an independent Artifact, so content revision, lifecycle, authorization and exact evidence can be managed per memory.

What changes are included in this PR?

• Introduce atomic-memory content with immutable revisions and separate active, forgotten, merged and retired state. Add explicit writes, exact historical reads, merge, forgetting, restoration preview, restoration and merge undo.
• Replace collection generation with Source extraction and reconciliation (create, revise, merge, noop). Recheck evidence, authorization, read dependencies and cursor generation before publishing memory changes and Source progress together.
• Separate authoritative history/head/state from the rebuildable current search projection. Apply Scope, access, kind and tag eligibility before ranking and limits; use consistent snapshots for short reads. Support text, vector and hybrid retrieval, including full-text retrieval without embeddings.
• Expose Atomic operations through HTTP, MCP and the Python client; update host adapters, generated contracts, dashboard, benchmarks, tests and bilingual workflow/migration documentation. Add offline migration and projection maintenance commands.

Clean migration

Legacy Memory leaves the runtime completely (8b7a953e, e8e6b25b, 6e923f3a):

• Legacy collections are archived to pc_memory_artifact_archive and removed from the public Artifact tables, search and online reads. The legacy entry tables are retained but detached from pc_artifacts.
• Exact entry citations in Experience lineage, Candidates, Handoffs and structured Work Sources become the deterministic Atomic revision. Whole-collection relationships are archived with their original location, then removed; whole-collection addresses in a Handoff receipt's unavailable_evidence remain in the same JSON shape as historical descriptions of unavailable evidence. They are not resolved and cannot support new requests; exact entry citations still become Atomic references. A claim left without required evidence, a Candidate without evidence (unless a --decisions replacement is supplied), or an unfinished Dream whose pinned inputs would change blocks before any write.
• Finished Dream runs move to a historical_data format. Unfinished runs with legacy input snapshots must finish before migration: removing even an empty legacy lineage field changes the pinned Artifact digest. Queued runs without a snapshot can migrate if their selected inputs are unaffected. Empty legacy request fields are removed and request digests are recomputed so identical retries still replay; a historical request that cited legacy entries keeps its original digest.
• Apply accepts import, projection, grant and reference conversion while collections are still public, removes them, and finally drops the emptied memory_citations columns. Startup refuses while those columns exist; rerunning apply completes the remaining steps.
• MemoryService, legacy persistence/indexes, MemoryCitation, every memory_citations field and Handoff kind: memory citations are removed from the runtime, OpenAPI contract, SDK, dashboard, Desktop and host integrations. ArtifactRef rejects family=memory, so new requests cannot reintroduce collection references; generic reads and tag queries of family=memory are unsupported. Frozen legacy decoders exist only inside the offline migration.

Are there any user-facing changes?

This is a breaking Memory upgrade.

• Retained remember/search/list/get(target) routes return Atomic records and exact Artifact references; exact legacy citation reads are unsupported. Flush retains progress/counts with memory: null. Route retention does not preserve the old response models. Collection CAS, citation revise/retire, capacity/compaction, collection Create/Replace, continuous changes and collection rollback are unsupported; HTTP rejects unsupported operations with 422 legacy_memory_operation_unsupported before writes.
• Existing databases require a backup, stopped writers and explicit atomic-memory-migrate --action plan/apply/verify; startup verifies readiness instead of converting data automatically. Migration maps legacy entries to deterministic new Artifact IDs while retaining their version chains, historical evidence, lifecycle, ownership, tags, grants/receipts and Source/task progress. Invalid history/ownership, unsupported grants, unresolved candidates or task/cursor formats can block conversion.
• Custom memory.extract Prompts require explicit migration to atomic_memory.extract and atomic_memory.reconcile. Legacy CandidatePipeline and MemoryWriteGate injection are incompatible with the Atomic runtime.
• Ordinary vector search and related-memory enumeration currently use exact L2 over eligible rows. Related memories are compared in bounded model batches, but enumeration is uncapped. This PR makes no ANN or production-scale performance claim. Merge/restoration do not provide a durable idempotency receipt.

How was this change tested?

Current validation at 3be389ce:

• Full local default pytest run: 4,881 passed, 279 skipped, 5 deselected, 0 failed (Node 22). One OpenCode host test was explicitly deselected; the other four deselections are the default-disabled Zcode host acceptance cases. Skipped cases are not counted as passed.
• Migration, Dream and Work receipt regressions: 83 passed / 41 skipped; API contract tests: 49 passed. Dify's full check: 80 tests plus 3 real HTTP/SQLite E2E tests passed, including the pinned official host helper.
• All pre-commit hooks and full type checking passed. The five Linux migration failures were reproduced under SQLite's BLOB/LIKE semantics and fixed with explicit text casts. Legacy Dream snapshots are rejected during preflight, while unaffected current snapshots and queued requests remain supported.
• CI for this exact head: 28 successful checks, 0 failed, 0 pending, 1 configured skip; all 6 workflows succeeded. The skipped Desktop installed-package job requires manual dispatch with an installer run; the regular Windows Desktop validation passed. Python 3.11–3.14, SQLite/OceanBase acceptance and SeekDB checks passed.
• Design documents are retained locally and excluded from this PR.

Focused validation for f97529c5:

• New opt-in real-model temporal-conflict acceptance passed on SQLite with A NewAPI GPT-5.6 Luna: 1 passed, 0 failed, 0 skipped, 1 configured-backend case deselected. The public Source/flush/list/search flow preserved the May fact when March evidence arrived later, then accepted a July correction. The default extraction/reconciliation Prompts were used. A per-memory claim interpreter checks current values without being given the expected answer or other memories; calibration verifies that it preserves unresolved contradictions.
• Focused local regression: 16 passed, 2 intentionally skipped real-model cases. Full pre-commit validation passed all 11 hooks, including Ruff and ty.
• Three earlier acceptance attempts failed: two test request/Prompt-comparison mistakes and one assertion that incorrectly rejected explicitly superseded historical values. These were corrected before the passing run; production code was unchanged.

Submission checks: uv run --locked prek run -a passed all 11 hooks, including Ruff and ty; git diff --check passed. The seven follow-up files match the recorded validation material.

Recorded validation used isolated, locked environments:

• DSH: pnpm --dir integrations/dsh/plugins/powercontext test and test:e2e — 266 unit tests and 9 E2E tests passed on each of macOS and Linux. Formal builds reproduced the checked-in bundles. OpenCode test — 72 passed; typecheck and build passed.
• Selected pytest coverage: 84 prompt/Atomic checks passed; 49 distinct SeekDB cases and 37 supported SQLite Dream cases passed; the seven host dependency checks passed. Host coverage includes adapter fixtures, native discovery and one real CLI hook, rather than complete acceptance of every host.
• Real-model Tags/Prompt acceptance: six distinct SQLite/OceanBase nodes passed with GPT-5.6 Luna and text-embedding-3-small. SQLite's three raw XML reports were checked locally. OceanBase's three complete launcher records exited 0; their raw XML/events could not be exported after connectivity was lost. Across all model attempts, results were 9 passed / 8 failed over 17 executions of those six nodes: two provider errors, three model-output/decision failures and three configuration failures. These earlier failures remain recorded.
• Earlier full default suite at 2b903293: 3,798 passed / 152 skipped / 4 deselected. Across recorded implementation commits, all 90 distinct official OceanBase nodes have passing records after targeted fixes/replays. This is not a full-suite run of the submission's seven follow-up files; that full suite was not rerun.

A historical Zcode Stop unknown and one OceanBase FULLTEXT initialization lock timeout were not reproduced in bounded diagnostics; their causes remain unknown. Windows/macOS-specific cases, isolated native service checks and four controlled/live Zcode acceptance nodes remain deferred. Real-model OceanBase raw evidence export and the new validation container's stop operation remain pending connectivity. Migration evidence covers the tested schemas/scenarios, not production deployment acceptance.

Clean migration commits: the full default suite at e8e6b25b passed (4,853 passed / 291 skipped) except one assertion on the startup rejection message, which was updated and rerun; tests/e2e/test_opencode_plugin_host.py was deselected because the local OpenCode installation had no usable model. prek run -a, ty check and make contract-test passed. Network-dependent packaging tests that failed during one local run on DNS errors passed when rerun.

Runtime removal (6e923f3a, merged with master at f69c2f92): the full default suite passed at f69c2f92 (4,867 passed / 279 skipped; tests/e2e/test_opencode_plugin_host.py deselected; Node 22 for native adapter checks). prek run -a, ty check, make contract-test, the Dify contract check and Desktop generate:check passed. OpenClaw, Pi, OpenCode and dsh plugin tests passed locally; Desktop UI tests and Rust tests passed on macOS except three tests/ipc.rs origin checks that depend on Windows. The column drop on seekDB/OceanBase and Windows-specific Desktop checks rely on CI. Merging #1918 makes the recall gate read an Atomic Memory hit's vector distance as cosine relevance; its new e2e cases were adapted to Atomic search (kind+body embedding input, per-channel limits before fusion, recoverability as an existence probe).

AI usage statement

GPT-6.1 Sol Ultra subagents performed implementation, validation and PR preparation; the Codex root agent reviewed the result. The runtime removal was implemented and validated by Claude Code with subagents. GPT-6 Astra Ultra subagents completed the CI and receipt migration fixes; the Codex root agent independently reviewed the combined changes and ran the final local suite. GPT-5.6 Luna was used as the generation/reranking model in the final real-model acceptance run. Validation scope and evidence limits are stated above.

[hidb4ai]

Reviewed 8b7a953ed2b7840e9e5349b8a24bfc9e1dca5274. Three reproduced issues remain in the inline comments: a custom Builtin authorization bypass, SQLite migration falsely reporting readiness, and non-resumable migration decisions. These should be addressed before merging.

Validation used the configured Qwen generation/embedding services, SQLite, and real OceanBase CE 4.3.5.4, including HTTP/MCP, lifecycle/restart, migration, and authorization snapshots. Acceptance is not fully green: OceanBase temporal reconciliation hit the configured 60-second timeout; Prompt acceptance failed initially and passed on a fresh-database replay, with the initial cause unresolved. The supplementary local full-suite run was stopped before completion. No installed Desktop/native-host acceptance is claimed.

[hidb4ai]

please resolve conflicts

[hidb4ai]

Re-reviewed 400602e827fdc91eab5ba72c3b515a6423465b6f. The previous decision-replay issue is fixed in the real OceanBase CLI interruption/retry probe. The Builtin authorization bypass and SQLite BLOB migration issue remain reproducible. Two additional P2 findings are below: silently discarded Python Runtime evidence and stale pinned Dream evidence after migration.

Real-service checks used the original .env Qwen generation/embedding models and isolated OceanBase CE 4.3.5.4 databases. HTTP/MCP extraction, text/vector/hybrid retrieval, lifecycle/restoration, restart readback, tagged retrieval (access disabled/enforced), and Prompt/Experience/Skill/Handoff generation passed on SQLite and OceanBase. OceanBase temporal reconciliation and migration/DDL-interruption recovery also passed. All test databases were cleaned up. These results do not establish installed native-host acceptance; the pinned-Dream finding was reproduced at the resolver boundary before generation.

The Dify tools check also needs its stale selector-count assertion updated: the probe resolves 17 remaining selectors while the test still expects 20. Correctness findings above remain blocking.

[hidb4ai]

[P2] Reject removed evidence fields at the Python Runtime boundary

Removing memory_citations leaves these public request models with Pydantic's default extra="ignore". Through the embedded Runtime, I passed a valid Source plus an invalid legacy citation to Experience propose and Candidate revise; both succeeded, and approval plus restart preserved only the Source. The same proposal on the previous head fails with InvalidMemoryCitationError before creating a Candidate.

HTTP/SDK models already reject this field, and the clean-migration contract requires rejecting legacy Candidate evidence before persistence. Please enforce that rejection for ProposeExperienceRequest and ReviseArtifactCandidateRequest too, so a successful write cannot silently discard supplied provenance.

[hidb4ai]

Verified fixed in the current Python Runtime. The original proposal now raises a validation error for memory_citations before creating a Candidate. The focused checks also verify rejection of populated, empty and null legacy fields for both ProposeExperienceRequest and ReviseArtifactCandidateRequest, while preserving current evidence references.

[hidb4ai]

Re-reviewed 60eb0095bfe7c25fc1c7f7ce7c1864106d1879c1. The Builtin authorization bypass remains reproducible. One additional P2 is below: ordinary Work message text can stop migration after writes and prevent Runtime startup. The SQLite BLOB filtering, removed Python evidence field and unfinished Dream snapshot issues are verified fixed; I have replied on their existing threads. The latest commit also fixes Desktop contract generation drift.

Validation: 95 focused tests, 55 authorization/lifecycle tests and five native SQLite BLOB regressions passed, plus a standalone public API reproduction without the pytest LIKE fixture. Using the original .env Qwen generation and 1024-dimensional embedding models, real HTTP/MCP journeys passed on SQLite and OceanBase CE 4.3.5.4, including extraction, three search modes, lifecycle changes, history and restart. Real-embedding CLI migration and repeated projection rebuilds also passed on both backends after deleting the disposable legacy tables/archive, preserving authoritative data and search results. OceanBase migration probes additionally covered pinned/queued Dream runs and recovery after committed Candidate conversion or column removal. All ten disposable remote databases were deleted and their absence independently verified.

Runtime tests ran at 38432c5a; local Git comparison confirms that the two subsequent commits change only Skill vendor/lock files and Desktop generated outputs. Their corresponding local validation checks pass. CI is not fully green: the current OceanBase migration job failed during fixture setup while creating a full-text index (SQL error 1205, lock wait timeout), before migration started; the cause remains unresolved, and other checks are still running. Native installed hosts and SeekDB were not exercised locally.

[hidb4ai]

LGTM