[MaoMengww]

Which issue or RFC does this PR close?

Closes #1975.

Implements the Source authority and verification prerequisite in RFC #1652, while preserving the Source access and lineage_only semantics in RFC #1458.

Rationale for this change

Source generation eligibility does not describe the authority or verification of its material. Atomic Memory needs explicit, versioned evidence declarations that remain inspectable after registration changes, restarts, and projection rebuilds.

This change records those facts without introducing trust scores, ranking bonuses, or additional write authority.

What changes are included in this PR?

• Add MemoryEvidenceDeclaration and exact Source material bindings through the existing Source Definition registry.
• Accept declarations only from trusted native adapters or deployment-owned configuration for exact remote Definition manifest identities.
• Retain historical Definition manifests, Source declaration bindings, and evidence snapshots keyed to exact Atomic Memory revisions.
• Add an authorized Python evidence-read API and reuse the existing projection rebuild path to verify retained historical material.
• Resolve missing, invalid, or unverifiable declaration records to neutral untrusted / unknown values. Missing or corrupt authoritative Source material and database failures remain errors.
• Add the additive p0005 evidence schema migration for SQLite and MySQL-compatible backends, without backfilling historical trust.
• Add compact SQLite/OceanBase acceptance scenarios covering revision history, write rollback, registration drift, cross-Scope isolation, missing-snapshot neutrality, synthetic model trust claims, and unchanged recall consumers.

Are there any user-facing changes?

This adds optional Source Definition evidence declarations and a Python API for inspecting an exact Atomic Memory revision's evidence.

It does not change Atomic Memory bodies or content hashes, Artifact revision identities, HTTP/OpenAPI search schemas, public search scores, or citation identities. It introduces no quality ranking or new generation eligibility rules.

Persistence gains additive evidence tables. Existing Sources and Memory revisions without accepted declarations remain neutral; current configuration does not retroactively attest them.

How was this change tested?

• make check: passed, including lock consistency, pre-commit checks, generated guidance checks, and type checking.
• make contract-test: 52 passed.
• make opendal-test: 8 passed; connector lint, formatting, and type checks passed.
• Full suite with doctests and the required DSH configuration runtime, using Python 3.12.13 / SQLite 3.50.4:
  • 5294 passed, 385 skipped, 0 failures.
  • Command: uv run --frozen --python 3.12 python -m pytest -n 6 --doctest-modules --require-dsh-config-runtime
• Focused Source, persistence, migration, lifecycle, and recall regressions: 128 passed.
• Real OceanBase CE 4.4.2.1 evidence acceptance: 3 passed, 0 skipped.
• Wheel and source distribution builds passed; the wheel contains the new evidence migration resources.

The initial full run under the local Python 3.11 / SQLite 3.37.2 environment exposed an existing Topic Memory vector-prefilter compatibility failure. A standalone SQLite reproducer isolated the cause: sqlite-vec's rowid IN KNN prefilter requires SQLite 3.38 or newer. The original test and full suite passed with the newer SQLite runtime, without changing code, assertions, or skip conditions.

The complete GitHub Actions Python-version matrix has not been run locally.

AI usage statement

AI assistance was used through Oh My Pi with gpt-6.1sol for implementation, test design and consolidation, code inspection, and validation analysis.

[hidb4ai]

Please address the two native Source evidence issues below before merging. Both reproduce through public SQLite flows and break evidence inspection and projection rebuild after successful Memory generation.

[hidb4ai]

Native Source evidence inspection and projection rebuild pass the input-alias and model-default scenarios. Remote observation ingestion still fails for serialization aliases; please address the inline P2 before merging.

[hidb4ai]

The three reported reproduction scenarios now pass. One remote Source regression remains when aliases apply to inherited envelope fields; please address the inline P2 before merging.

[hidb4ai]

One native Source acceptance issue remains: a round-trippable identity serializer can be durably accepted but fail later Memory extraction. Please address the inline P2 before merging.

[hireview001]

• CI failed: validate (failure).
• CI failed: E2E harness (failure). Failed steps: validate / Validate harness and Compose definitions.

[hireview001]

[P2] Decode canonical historical identity before consulting current aliases

For a pre-upgrade canonical native envelope without a source header, registering a newer Definition with a name alias sends the old material through today's decoder, causing evidence reads and rebuilds using that registry to fail on valid historical data. Resolve retained canonical identity independently of the current decoder and keep missing historical declarations neutral.

[hidb4ai]

Independently reproduced on 88cd8f23 with SQLite data captured through the public Source and Memory workflow on base e4efb39b. The retained Source has canonical identity keys and no source header. With no current native decoder registered, both memory.evidence(..., revision=1) and projection rebuild succeed. Registering Definition version 2 with name: str = Field(alias="source_name") makes both operations fail with InvalidStoredPayloadError because the old payload is passed to today's model. Exact Memory reads still succeed. Historical identity must remain readable independently of the current Definition, with absent historical declarations treated as neutral.

[hidb4ai]

[P2] This remains reproducible for non-default historical versions on 2b95df67. I captured a version 2 Source and generated its Memory through public APIs on base 0da4436d, then reopened the same SQLite database on this head. Without a native decoder, evidence inspection and rebuild both succeed. Registering version 3 with name: str = Field(alias="source_name") and a business label aliased to name makes both fail with InvalidSourceDefinitionError: Source declares '2', expected '3'; exact Memory reads still succeed. The unbound fallback in _historical_canonical_source() only recognizes version 1 (sources.py:455-458), so this case still reaches today's decoder. Historical evidence must preserve the retained explicit identity independently of the current registration.

[MaoMengww]

Fixed in 13517fdd.

_historical_canonical_source() now:

• when no evidence binding exists, accepts the explicit string definition_version from the retained canonical material instead of only "1";
• when a binding exists, requires the material version to equal the binding version; a mismatch is no longer guessed and falls through to the strict decoder path;
• still rebuilds only a neutral Source identity, so the declaration remains untrusted/unknown and the current aliased Definition is never used to decode historical material.

Regressions:

• test_unbound_versioned_canonical_history_ignores_new_definition_aliases — version 2 canonical material with a version 3 Definition where name is occupied by a business alias; evidence matches the neutral baseline and reports version "2".
• test_canonical_historical_evidence_survives_business_alias_rebuild — now parameterized over historical versions 1 and 2, and covers public evidence() plus rebuild_atomic_memory_projection() before/after equality.

Affected tests: 63 passed, 10 skipped; make check passed. OceanBase parameterizations were skipped locally because no instance is available.

[hidb4ai]

Re-reviewed 88cd8f23. The previous identity-serializer finding is fixed: the unsupported representation is rejected before persistence.

Two P2s remain independently reproducible: remote transport with an explicit identity serialization alias, and historical evidence/rebuild failures after registering a newer aliased Definition. Added reproduction details to the existing threads rather than opening duplicates. Please address both before merging.

The two failing Kimi ARM harness consistency tests also fail on base e4efb39b; I have not attributed that CI failure to this PR.

[hidb4ai]

Re-reviewed 25d9e7a7. The previous two P2s are fixed. One remote-ingestion regression remains for recursive Source schemas with identity aliases; please address the inline P2 before merging.

[hidb4ai]

Re-reviewed 2b95df67. The recursive Source transport regression is fixed. Two P2s remain: an unchanged aliased remote Definition conflicts with its existing manifest after upgrade, and version-2 canonical history still fails evidence inspection and rebuild under a newer aliased Definition. See the inline finding and my follow-up in the existing historical-identity thread. Please address both before merging.

[hidb4ai]

[P2] Preserve registered remote manifest identities when upgrading

An unchanged remote Definition with model_config = ConfigDict(serialize_by_alias=True) and body: str = Field(alias="wire_text") registers and submits successfully on base 0da4436d. After upgrading the same SQLite database, forcing by_alias=False changes its schema from wire_text to body and changes its fingerprint without changing the Definition name/version. register_source_definition() then raises SourceConflictError against the previously accepted manifest, blocking the worker. Preserve existing supported manifest/payload identities or provide an explicit version transition rather than silently redefining the same version.

[MaoMengww]

Fixed in 13517fdd.

The transport contract now derives both sides from the same alias policy:

• _manifest_for_definition(transport=True) uses model_json_schema(mode="serialization", by_alias=bool(model_config.get("serialize_by_alias")));
• project_source_for_transport() uses model_dump(mode="json"), whose default by_alias=None follows the same serialize_by_alias config.

For the reported case, serialize_by_alias=True with body: Field(alias="wire_text") now keeps wire_text in both the schema and the payload, and the fingerprint matches the base model_json_schema() result. Definitions that only declare serialization_alias still use field names, also matching base.

test_alias_serialization_remote_manifest_and_payload_remain_upgrade_stable reproduces the upgrade path: register on a fresh SQLite database, reopen the same database, re-register the unchanged manifest, and submit the observation. It asserts the schema equals the base-generated schema, the fingerprint is recomputed from it, and the payload retains wire_text.

Affected tests: 63 passed, 10 skipped; make check passed. OceanBase parameterizations were skipped locally because no instance is available.

[hidb4ai]

[P2] The original serialize_by_alias=True case is fixed on 3c5c0ecc, but fingerprint compatibility still fails for an ordinary amount: Decimal field with no aliases. On base 2d11abd6, public manifest registration and observation submission succeed: the validation schema allows number/string, and the payload contains "1.25". On this head, mode="serialization" at observations.py:178 produces a string-only schema and changes the fingerprint for the unchanged Definition name/version. Re-registering it on the upgraded SQLite database raises SourceConflictError, blocking the worker even though the payload is identical. Preserve supported pre-upgrade manifest identities or provide an explicit version transition.

[MaoMengww]

Fixed in 3a8b38be.

The transport schema now keeps the pre-upgrade validation schema unless an alias renames a required payload key, checked recursively for nested models. Optional aliased fields keep the base schema because the canonical key is an allowed extra property while the alias key is not required, which matches pre-upgrade behaviour and preserves the fingerprint. AliasChoices resolves to its first string choice; AliasPath keeps the field name.

Regressions:

• test_decimal_source_manifest_fingerprint_survives_upgrade asserts the Decimal Definition keeps the base model_json_schema() and fingerprint, and that re-registration and observation submission still succeed after reopening the same SQLite database.
• test_optional_alias_keeps_the_preexisting_manifest_fingerprint covers an aliased optional version field and asserts the same fingerprint preservation.

[hidb4ai]

Re-reviewed 3c5c0ecc. The original alias-configuration and version-2 historical-read reproductions now pass. One P2 remains: changing transport schema mode changes the fingerprint of an unchanged Decimal-bearing Definition and prevents re-registration after upgrade. Added the reproduction to the existing manifest-compatibility thread. Please address this before merging.