Skip to content

[P0][Graph PR B] Add deterministic semantic multigraph storage and artifact v2 #657

Description

@mohanagy

Parent: #649
ADR sequence: 2 of 4
Depends on: #656
Merge blocked by: #654
Qualification baseline: #655


Maintainer amendment after Stage 0 endpoint-identity audit

This section supersedes the absolute movement-stability invariant stated below. Read it first. Every clause elsewhere in this body that asserts the absolute rule is qualified to require two stable endpoints.

A mandatory adversarial review disproved the premise that a bounded set of route node IDs could be corrected to make fact identity universally movement-stable. Route-only partial re-keying is prohibited in this issue — it risks silent node collisions and an inconsistent identity policy. Making semantic node identities actually movement-stable is #704.

Endpoint identity is explicit, with four states

type EndpointIdentityStatus = 'stable' | 'context_bound' | 'unknown' | 'legacy'

interface EndpointIdentityQualification {
  readonly source: { readonly status: EndpointIdentityStatus; readonly reasons: readonly EndpointIdentityReason[] }
  readonly target: { readonly status: EndpointIdentityStatus; readonly reasons: readonly EndpointIdentityReason[] }
}
Status Meaning
stable positively audited: deterministic under the declared identity-policy version, repository-relative rather than checkout-root dependent, independent of incidental line/column/range, independent of unstable ordinals and collision order, collision-safe for the node kind. Absence of a known problem does not prove stable.
context_bound known to depend on non-semantic context — source line/column/range, absolute workspace path, source ordinal or rawIndex, collision suffix, or adapter-local synthetic identity. Exact reasons must name the dependency.
unknown the producer or compatibility caller does not supply enough information to classify. Not an alias for context_bound, and never silently stable.
legacy v1-loaded endpoints whose historical identity policy cannot prove the current v2 guarantees.

Reason codes, at minimum: source_location_derived, absolute_workspace_path_derived, source_ordinal_derived, collision_suffix_derived, adapter_local_synthetic_identity, identity_policy_not_declared, identity_policy_not_audited, legacy_identity_policy. A status and its reasons must be internally consistent — stable carries no degradation reason; context_bound carries at least one context-dependency reason; unknown carries identity_policy_not_declared or identity_policy_not_audited; legacy carries legacy_identity_policy. Malformed combinations fail invariant validation.

This dimension is separate from discriminator completeness and must not be hidden in arbitrary metadata.

Admission policy

Status Normal-mode behaviour
stable retain
context_bound retain, visibly degraded
unknown retain, visibly degraded
legacy retain only through the legacy compatibility path, visibly degraded

A fact is never rejected solely because an endpoint is unknown. Discarding real facts because the identity audit is incomplete would be worse than recording the uncertainty.

Hard failure remains correct for: missing source endpoint, missing target endpoint, malformed endpoint qualification, impossible status/reason combination, hash/payload invariant failure. A missing endpoint is not unknown; a malformed qualification is not unknown. Missing internal endpoints must not be auto-created.

Every admitted node carries one normalized status: audited producers attach stable or context_bound; the v1 loader attaches legacy; unmigrated or compatibility producers attach unknown with an explicit reason. addNode() stores it, addEdge() verifies both endpoints exist with valid stored qualifications, and copy/subgraph/load preserve them exactly. A compatibility API that omits qualification normalizes to unknown + identity_policy_not_declared — never to stable, and never by dropping the node.

The truthful movement rule

Moving an evidence location changes the occurrence ID, and does not change the fact ID, when both semantic endpoint identities are stable and the behaviour discriminator is unchanged.

For any other status the fact ID may change when the endpoint NodeId changes; the fact stays deterministic for one exact snapshot; the limitation is machine-visible; and it may not be cited as proof of cross-movement stability.

  • IDs deterministic — for the exact source snapshot and declared identity policy.
  • Movement-stable — a stronger, qualified property holding only for stable+stable facts.

Receipt: an exact 4 × 4 partition

The storage-only receipt carries an ordered source-status × target-status matrix. Outer key is source status, inner key is target status. For undirected facts, use the same canonical endpoint ordering as fact identity, and document it.

{
  "accounting_scope": "storage_only",
  "status": "degraded",
  "reasons": ["full_emission_accounting_not_available"],
  "endpoint_identity": {
    "statuses": ["stable", "context_bound", "unknown", "legacy"],
    "fact_pair_counts": { "stable": { "stable": 0, "context_bound": 0, "unknown": 0, "legacy": 0 }, "context_bound": {}, "unknown": {}, "legacy": {} },
    "reason_fact_counts": {}
  }
}

Partition invariant: the sum of all sixteen cells equals the number of facts retained in the artifact. No retained fact is absent from the matrix; none occupies more than one cell. There is no "worst status wins" precedence, because no single precedence preserves both endpoint statuses.

reason_fact_counts counts distinct retained facts carrying each reason. A fact with several reasons contributes to several counters, so reason counts are not a partition and their sum need not equal the fact count. This must be documented, not implied. Derived summaries such as fully_stable_facts should be computed from the matrix; if serialized redundantly they must be validated against it and disagreement must fail.

A missing endpoint produces no fact and therefore no cell — admission fails with a typed error. Durable accounting for the rejected candidate belongs to #658. Do not add missing, failed or rejected to the status union.

The receipt never claims valid.

Propagation is staged — PR B stores, it does not project

#657 makes the qualification available in the graph domain only: SemanticFact, KnowledgeGraph fact queries, artifact v2 and its loader, the storage-only receipt, and graph-specific inspection APIs directly required by artifact v2.

#657 must not change Pack v1 relationships or coverage fields, Pack answerability, context-pack governance, MCP response answerability, existing ready / verify_targets / insufficient transitions, or text renderers merely to expose this limitation. No optional Pack fields, no answerability cap, no partial PR-D implementation.

#657  store and serialize the limitation
#658  preserve it in the normalized-boundary integrity receipt
#659  project it compactly into Pack/MCP and cap answerability

The intermediate next branch may carry stored degradation before PR D consumes it. No beta may be published in that state.

Additional acceptance criteria

Additional non-goals

Unchanged and still binding: multigraph storage, artifact v2, old-reader rejection, topology uniqueness, Pack required-field compatibility, and the performance gate.


Outcome

Replace endpoint-only relationship storage with a deterministic semantic multigraph that preserves behaviorally distinct facts, separate evidence occurrences, and unique endpoint-pair topology.

Introduce artifact v2 with deterministic round-trip behavior, safe v1 loading, and a proven mechanism preventing v0.32.1 from silently down-reading v2.

This issue corrects the confirmed silent-overwrite defect. It does not yet implement full normalized-candidate integrity accounting or answerability propagation.

Confirmed defect

At v0.32.1:

  • KnowledgeGraph.edgeMap is keyed by source + target only;
  • Map.set() makes insertion last-write-wins;
  • calls followed by injects between the same endpoints leaves only injects;
  • reversing insertion order leaves only calls;
  • same-relation facts at different locations also collapse;
  • serialization exports only the survivor;
  • deserialization routes input through the same collapsing behavior.

This is P0 because valid semantic and provenance evidence can disappear deterministically while downstream retrieval and context compilation appear successful.

Architecture decision

Implement separate canonical concepts:

  1. Semantic fact — one behaviorally distinct relationship.
  2. Evidence occurrence — one adapter-owned observation supporting a fact.
  3. Topology pair — one unique directed/undirected endpoint pair used by topology algorithms.
  4. Storage-only integrity status — an honest temporary status until PR C adds full normalized-boundary accounting.

Conceptual semantic fact

interface SemanticFact {
  readonly id: SemanticFactId
  readonly direction: 'directed' | 'undirected'
  readonly source: NodeId
  readonly target: NodeId
  readonly relation: RelationType
  readonly discriminator: SemanticDiscriminator
  readonly endpointIdentity: EndpointIdentityQualification
  readonly occurrenceIds: readonly EvidenceOccurrenceId[]
  readonly annotations: Readonly<Record<string, CanonicalJson>>
}

Conceptual occurrence

interface EvidenceOccurrence {
  readonly id: EvidenceOccurrenceId
  readonly factId: SemanticFactId
  readonly owner: {
    readonly adapterId: string
    readonly strategy: string
    readonly sourceFile?: string
    readonly adapterVersion?: string
  }
  readonly sourceFile?: string
  readonly sourceRange?: SourceRange
  readonly targetFile?: string
  readonly targetRange?: SourceRange
  readonly siteKind?: string
  readonly adapterEvidenceKey?: string
  readonly provenance: readonly Provenance[]
  readonly confidenceObservations: readonly ConfidenceObservation[]
  readonly metadata: Readonly<Record<string, CanonicalJson>>
}

Adapt exact types to repository conventions without weakening the fact/occurrence separation.

Fact identity

Fact ID must be deterministic and content-derived from canonical semantic content:

"sf_" + SHA-256(canonical JSON {
  identitySchemaVersion,
  direction,
  source,
  target,
  relation,
  discriminator registry/policy version,
  discriminator completeness,
  canonical discriminator value
})

Requirements:

  • full lowercase SHA-256 hex; no truncation;
  • UTF-8 canonical input;
  • NFC string normalization;
  • lexically ordered object keys;
  • finite JSON numbers only;
  • explicit ordered versus set-like array handling;
  • source ranges, adapter names, confidence, timestamps, scores, and arbitrary metadata excluded;
  • directed endpoint order preserved;
  • undirected endpoints canonically sorted;
  • different canonical payloads producing one hash is a fatal invariant failure.

Renaming a node may change its node/fact ID. Moving an evidence location must not change fact identity when both endpoint identities are classified stable and the discriminator is unchanged. When either endpoint is context_bound, the fact ID may change and the fact is classified partial/degraded — see the maintainer amendment above.

Occurrence identity

Occurrence ID must be deterministic and adapter/evidence-site specific:

"eo_" + SHA-256(canonical JSON {
  occurrenceIdentitySchemaVersion,
  factId,
  adapterId,
  strategy,
  repositoryRelativeSourceFile,
  sourceRange,
  repositoryRelativeTargetFile,
  targetRange,
  siteKind,
  adapterEvidenceKey
})

Exclude confidence, mutable scores, timestamps, adapter version, absolute paths, and provenance ordering.

Required behavior:

  • same adapter/site repeat merges deterministically;
  • different adapters remain separate occurrences;
  • different call/registration sites remain separate occurrences;
  • moving a site changes occurrence ID but not fact ID when both endpoints are stable; when either endpoint is context_bound, the fact ID may also change and the fact is degraded;
  • unchanged full generation reproduces IDs exactly;
  • provenance and confidence disagreement remain observable.

When no range or stable evidence key exists, indistinguishable emissions may merge only with an explicit diagnostic such as occurrence_multiplicity_unknown; do not invent nondeterministic sequence IDs.

Relation discriminator registry v1

Add an exhaustive, static, versioned registry. There is no implicit endpoint-only fallback for unknown relations.

At minimum define policies for:

Calls

Behavior fields may include dispatch kind, invocation kind, optionality, overload identity, and resolution state.

Multiple sites with identical behavior are occurrences of one fact. Optional versus ordinary calls, construct versus call, or valid different overload targets may be separate facts.

Dependency injection and registrations

Cover current relations such as injects, provides, module/controller registration relations, with token, qualifier, scope, binding kind, resolved implementation, and resolution state where available.

Routes/middleware

Cover route/handler/controller/mount/middleware relations using HTTP method, normalized path, registration type, router scope, order where behaviorally relevant, and direct/decorator/config/generated origin.

Producers/consumers/jobs

Cover queue/topic/event/job identity, channel name, job type, routing key, role, and resolution state.

Imports/exports

Cover binding kind, imported binding, runtime versus type phase, static versus dynamic form, module specifier, and internal/external/unresolved state.

Explicit endpoint-only relations

Known structural relations may use a registered endpoint-only policy. The registry must list them explicitly.

Fallback behavior

  • registered relation missing data: retain a partial/coarse fact only when endpoint-level meaning remains valid and record reasons;
  • unregistered relation: do not invent endpoint-only identity; produce storage-boundary unresolved/degraded handling sufficient for this slice, with full accounting deferred to PR C;
  • unknown metadata remains on occurrences and never enters identity blindly;
  • conflicting valid discriminators create separate facts and an explicit conflict diagnostic, or are withheld when unsafe to canonicalize.

A registry version change invalidates identity caches and requires regeneration.

Duplicate/conflict policy

Implement and test:

Case Required behavior
Same fact and same occurrence twice from one adapter Merge occurrence; union provenance/confidence observations deterministically
Same fact/site from different adapters One fact, separate occurrences
Same fact at different locations One fact, separate occurrences when both endpoints are stable. With a context_bound endpoint the sites may yield separate facts; each is deterministic at one snapshot and classified degraded.
Same endpoints/relation, different valid discriminator Separate facts
Same endpoint/relation with partial registered discriminator Retain partial coarse fact with reasons when safe
Same occurrence, different confidence Preserve all observations; no winner erases disagreement
Conflicting behavior metadata Separate valid facts plus conflict diagnostic, or withhold as conflict; never last-write-wins
Legacy v1 fact One legacy/partial fact and legacy occurrence; warn multiplicity is historically unrecoverable

No case permits arbitrary first-writer or last-writer selection.

Storage and indexes

Store facts by SemanticFactId, occurrences by EvidenceOccurrenceId, with deterministic secondary indexes for:

  • source;
  • target;
  • relation;
  • endpoint pair;
  • fact-to-occurrence lookup.

The plural APIs introduced by #656 must now return all facts deterministically.

Topology remains unique-neighbor/unique-endpoint based:

  • BFS/DFS/path existence: endpoint pairs;
  • centrality: endpoint pairs;
  • communities: endpoint pairs;
  • bridges: endpoint pairs;
  • relation-aware traversal: matching facts with deduplicated endpoint visits;
  • semantic analysis/diff: facts;
  • evidence strength/provenance: occurrences.

Occurrences must never create extra traversal steps or centrality weight. Initial ranking must not sum fact/occurrence multiplicity.

edgeAttributes(source, target) compatibility behavior after this issue:

  • zero facts: existing unknown-edge behavior;
  • exactly one fact: compatibility projection;
  • multiple facts: typed AmbiguousEdgeError.

Internal code must already be migrated by #656.

Artifact v2

Canonical artifact:

out/graph.madar

Format:

MADAR_GRAPH_ARTIFACT/2
<canonical UTF-8 JSON payload>\n

Payload must include explicit versions for artifact, fact identity, occurrence identity, discriminator registry, repository revision, generation mode, nodes, facts, occurrences, hyperedges, community labels, and a storage-only integrity status.

Deterministic ordering:

  • nodes by node ID;
  • facts by fact ID;
  • occurrences by occurrence ID;
  • occurrence IDs within facts sorted;
  • canonical object keys;
  • deterministic hyperedge/label ordering.

A generated timestamp must not invalidate deterministic tests; inject/fix it in deterministic fixtures.

During the PR B → PR C interval, integrity must state honestly:

{
  "accounting_scope": "storage_only",
  "status": "degraded",
  "reasons": ["full_emission_accounting_not_available"]
}

It must not say valid before PR C.

Legacy artifact compatibility and old-reader rejection

Simply adding an unknown JSON field is unsafe because the v0.32.1 loader can ignore it.

Required migration behavior:

  1. v2 uses the non-JSON magic header and distinct out/graph.madar path.
  2. When v2 activates, preserve a valid existing v1 artifact as out/graph.v1.json where applicable.
  3. Atomically replace default out/graph.json with a non-JSON tombstone:
MADAR_GRAPH_MOVED/2
Use out/graph.madar with Madar >= the v2-supporting version.
  1. Prove using the exact v0.32.1 loader behavior:
    • old loader rejects graph.madar;
    • old loader rejects the tombstone;
    • new loader reads v2;
    • new loader reads legacy v1 as degraded.
  2. Never down-convert v2 to v1 while claiming equivalence.
  3. Rollback requires source regeneration using the prior binary.

New loader behavior for v1:

  • recognize pure legacy JSON;
  • convert each surviving link into one deterministic legacy fact and occurrence;
  • mark discriminator completeness legacy/partial;
  • emit legacy_parallel_facts_unrecoverable;
  • permit bounded normal-mode use as degraded;
  • reject qualification/strict use;
  • recommend full regeneration.

Scope

Likely affected modules:

  • src/contracts/graph.ts;
  • src/contracts/types.ts;
  • new focused semantic-graph/identity/canonicalization/discriminator modules;
  • src/pipeline/build.ts;
  • src/pipeline/export.ts;
  • src/runtime/serve.ts;
  • src/shared/workspace.ts;
  • graph-copy/subgraph helpers;
  • CLI graph-path resolution/help directly required by graph.madar.

Explicit non-goals

  • Full normalized-candidate integrity receipt/state machine — PR C.
  • Graph-integrity answerability cap — PR D.
  • Retrieval/ranking redesign.
  • SPI/legacy consolidation.
  • Incremental indexing.
  • MCP profiles/installers.
  • Pack required-schema redesign.
  • New languages/frameworks.
  • Broad CLI refactor.

Required tests

Semantic preservation

  • calls and injects on identical endpoints → two facts, one endpoint pair;
  • same relation/different valid discriminator → separate facts;
  • same fact/different sites → one fact, multiple occurrences;
  • compiler and heuristic corroboration → separate occurrences;
  • exact duplicate → deterministic merge;
  • conflicting metadata → explicit conflict/no winner;
  • unknown relation → unresolved/degraded, not topology.

Graph APIs/topology

  • all facts returned deterministically;
  • relationsBetween() stable;
  • ambiguous compatibility API throws;
  • successors/predecessors remain unique;
  • fact/pair/occurrence counts differ correctly;
  • centrality/community/bridges/path steps do not multiply;
  • impact visits each node once;
  • relation filters preserve all qualifying fact IDs.

Identity

  • deterministic canonical JSON;
  • stable fact and occurrence IDs;
  • line move changes occurrence, not fact, for stable-endpoint facts; for a context_bound endpoint, assert instead that the fact is deterministic at one snapshot, classified context_bound, and contributes a degraded receipt reason;
  • adapter change produces separate occurrence;
  • canonical payload collision check is fatal;
  • absolute paths/timestamps/scores do not affect identity.

Serialization/compatibility

  • byte-deterministic v2 round trip;
  • facts, discriminators, occurrences, provenance and storage-only receipt survive;
  • v1 loads degraded;
  • historical multiplicity warning;
  • exact v0.32.1 loader rejects v2 and tombstone;
  • corrupt header/body invalid;
  • unsupported schema/identity/registry version incompatible;
  • cache invalidation on version/revision changes.

Differential safety

  • centrality, communities, bridges, and path topology match unique-pair baseline;
  • Pack v1 external required schema remains unchanged;
  • every changed Pack/retrieval output caused by newly preserved facts is reviewed;
  • generated demo/eval graph regression;
  • exporter/report/Neo4j representations distinguish facts from occurrences where exposed.

Performance and artifact gate

Measure representative before/after:

  • generation time;
  • serialization/loading time;
  • peak RSS;
  • artifact size;
  • fact/occurrence/pair counts.

More than 2× cost in a material dimension requires explicit review and an accepted rationale; do not silently accept it.

Acceptance criteria

  • Every valid fact admitted at the storage boundary survives.
  • No Map.set() path can overwrite another semantic fact.
  • Exact duplicate and conflict behavior follows the documented policy.
  • Evidence/provenance survives merging.
  • Topology is not multiplied by facts or occurrences.
  • IDs and artifact bytes are deterministic.
  • Artifact v2 cannot be silently down-read by v0.32.1.
  • v1 loads only as visibly degraded and cannot qualify strict runs.
  • Artifact/cache migration and rollback paths are tested.
  • Pack v1 required fields remain compatible.
  • Storage-only receipt never claims full validity.
  • Performance/artifact measurements are published and reviewed.
  • [P0] Stabilize the complete Vitest suite and protected CI merge gate #654 is complete before merge.
  • All protected gates pass.

Preconditions carried from the #680 plural-projection audit

These are not new scope. They are the four findings the audit deliberately did not fix in PR A,
because none can be triggered or tested until this issue introduces fact multiplicity, plus the
testing gap that follows from the same cause. Each is recorded as a comment at its site in the
merged code; they are listed here so they gate the work rather than annotate it.
See #657 (comment) for the full audit.

  • endpointEntries() still yields each endpoint pair exactly once under a fact-keyed store, and the Louvain accumulation in cluster.ts is re-verified. _edgeWeight is correctly non-additive, but the loop consuming it accumulates per entry (totalWeight +=, neighborWeights, nodeDegree). If this issue makes the store fact-keyed and that uniqueness lapses, Louvain weights inflate silently and the suite stays green.
  • graphDiff() distinguishes facts that differ only by discriminator. Today it keys on endpoints + relation, so adding such a fact reports as no changes.
  • N-order stability of relationsBetween() is proven against this issue's fact-ordering contract, not inherited from it. It currently delegates to factsBetween() order and does not normalise independently, so property 4 of [P0][Graph PR A] Introduce plural relationship queries and explicit topology APIs #656 must be re-established here.
  • SVG export does not render multiplicity as visual weight. N facts currently draw N coincident lines, so alpha overdraw makes a connection look stronger purely because more facts share its endpoints — the same non-additive principle already corrected in cluster.ts.
  • The plural-fact consumers enumerated in the feat(graph): PR A — plural relationship queries and explicit topology APIs (#656) #680 audit each have a test exercising N > 1. All of them are correct today only because N is always 1; none is currently characterised at N > 1.

Rollback

Revert this issue’s PR, restore/regenerate a real v1 out/graph.json from source using the prior binary, and retain graph.madar only as an ignored diagnostic backup. Never convert v2 to v1 while claiming semantic equivalence.

Agent handoff

Implement only PR B. Stop before full integrity accounting or answerability changes. The final report must include architecture conformance, exact model/index/serializer changes, compatibility fixture evidence, all output differences, performance measurements, complete validation, and rollback instructions.


Implementation amendment — PR B split

The conceptual ADR slice is unchanged. PR B is implemented as two sequential mergeable PRs because the legacy-path cutover changes a large public and test contract unrelated to graph storage (618 graph.json mentions across 60 test files, 148 of them hard JSON.parse assumptions, plus 130 in src, 366 in docs, and three public surfaces).

B1 — core multigraph and transitional artifact activation

  • semantic fact / occurrence storage
  • endpoint-pair topology
  • relation registry
  • storage-boundary degradation
  • deterministic artifact v2
  • canonical graph.madar generation
  • new-reader preference for graph.madar
  • fresh v1 graph.json compatibility mirror
  • no tombstone activation
  • no public graph resource rename
  • no claim that the old default loader rejects

B2 — legacy graph-path cutover (#705)

  • stop active v1 mirror generation
  • preserve valid v1 as graph.v1.json
  • write the graph.json tombstone
  • migrate the public HTTP / MCP / installer surfaces
  • migrate all tests, docs and fixtures
  • exact old-default-reader rejection
  • rollback and source regeneration

Dependency and release gates

#657 is completed only after B1 and B2.
#658 depends on B2.
No release is permitted between B1 and B2:
  no v0.33.0* tag, no npm next dist-tag,
  no GitHub beta/RC, no MCP Registry prerelease.

The transitional dual-artifact state is allowed on next only. It is not an approved release state.

Every acceptance criterion below concerning the graph.json tombstone, the default-path old-reader rejection, or the public graph resource contract is Owned by B2. Those criteria are unchanged, not removed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0architectureCross-cutting design or substrate decisionsarea:qualitybugSomething isn't workingindexingIndex build / extraction / SPI

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions