fix(store): relations missing a sync_mutations row never reach cloud (no relation backfill)

[forNerzul]

📋 Pre-flight Checks

• I have searched existing issues and this is not a duplicate
• I understand this issue needs status:approved before a PR can be opened

📝 Bug Description

A relation can exist in the local memory_relations table with no corresponding sync_mutations row, and there is no mechanism to ever create one. Such relations are silently never replicated to Engram Cloud — even though the local chunk sync path (post-#489/#494) exports every non-orphaned relation directly from the table.

This is distinct from the already-resolved relation cloud issues, all of which operate on relation mutations that already exist in the journal:

• fix(store,chunkcodec): cloud sync blocked by unsupported relation mutations #313 / fix(sync): accept cloud relation mutations #379 — cloud rejected enqueued relation mutations (chunkcodec/server). Fixed.
• cloud upgrade legacy evaluator does not whitelist relation entity (regression after #379) #383 — the legacy upgrade evaluator blocked enqueued relation mutations. Fixed (SyncEntityRelation is now whitelisted in evaluateCloudUpgradeLegacyMutationTx, internal/store/store.go:1514).
• Cloud repair should backfill and report legacy mutation required fields #340 — repairs missing required fields inside existing mutation payloads (session/observation/prompt). Does not create missing rows, and cloud upgrade legacy evaluator does not whitelist relation entity (regression after #379) #383 itself notes it "does not call out the relation entity gap specifically".

None of these create a sync_mutations row for a relation that never had one. The cloud's own row-creation/repair mechanism, backfillProjectSyncMutationsTx, backfills sessions/observations/prompts but never relations.

Root cause

Three confluent code paths, all verified against engram 1.16.3:

1. No relation backfill. backfillProjectSyncMutationsTx (internal/store/store.go:4996) calls only backfillSessionSyncMutationsTx, backfillObservationSyncMutationsTx, and backfillPromptSyncMutationsTx. projectNeedsBackfill (store.go:5009) only counts missing sessions/observations/prompts. There is no backfillRelationSyncMutationsTx anywhere in the repo. These run at startup (New → repairEnrolledProjectSyncMutations, store.go:643) and on enrollment/rename/merge — the exact "row exists in source table but not in journal" repair that relations are excluded from.

2. JudgeBySemantic never enqueues. JudgeBySemantic (internal/store/relations.go:732) inserts/updates memory_relations but never calls enqueueSyncMutationTx. The only relation enqueue in the codebase is relations.go:687, inside JudgeRelation. JudgeBySemantic is live: it backs mem_compare (internal/server/server.go:1315, internal/mcp/mcp.go:2134) and ScanProject (relations.go:1400). Every semantic/system verdict therefore lands in the table with no journal row.

3. JudgeRelation skips the enqueue when the project is not enrolled. relations.go:653: if enrolled == 0 { return nil // not enrolled — no mutation enqueued }. By contrast, enqueueSyncMutationTx (store.go:5364) does not gate on enrollment — observations/sessions/prompts are enqueued unconditionally and the backfill catches legacy ones. Relations have neither: judged-before-enrollment relations get no row, and enrolling later does not backfill them.

The net asymmetry: the local chunk export reads relations straight from memory_relations (query at store.go:~3380, used by engram sync), so local sync carries them regardless of how they were judged. The cloud path is journal-based and only ever carries what was enqueued — so the same table yields different replication outcomes for local vs cloud, and for JudgeRelation vs JudgeBySemantic.

🔄 Steps to Reproduce

Path A — JudgeBySemantic (no upgrade required):

1. Enroll a project in cloud sync.
2. Trigger a semantic verdict that is not not_conflict — e.g. an agent calls mem_compare on two observations, or ScanProject runs.
3. Inspect the journal:

   -- a row exists in memory_relations:
   SELECT sync_id, relation FROM memory_relations ORDER BY created_at DESC LIMIT 1;
   -- but no matching mutation was enqueued:
   SELECT COUNT(*) FROM sync_mutations WHERE entity='relation' AND entity_key='<that sync_id>';  -- 0

4. The relation is never pushed to cloud; local sync (engram sync) would export it.

Path B — pre-enrollment JudgeRelation:

1. With a project not enrolled, resolve a conflict candidate via mem_judge (→ JudgeRelation). relations.go:653 returns without enqueuing.
2. Enroll the project. repairEnrolledProjectSyncMutations backfills sessions/observations/prompts but not the relation.
3. The relation is permanently absent from the cloud journal.

✅ Expected Behavior

• A relation present in memory_relations (non-orphaned) is eventually represented by a sync_mutations row and replicated to cloud, regardless of the judge path or enrollment ordering — matching how local chunk sync already treats relations and how the cloud already treats sessions/observations/prompts.
• Concretely: backfillProjectSyncMutationsTx / projectNeedsBackfill cover relations, and/or JudgeBySemantic enqueues, and/or JudgeRelation enqueues even when not-yet-enrolled (so backfill can pick it up later).

❌ Actual Behavior

Relations created via JudgeBySemantic, or via JudgeRelation before enrollment, have no sync_mutations row and no repair path. They never reach Engram Cloud and fail silently — there is no warning, and the local sync path replicates them, masking the loss until a cloud-only consumer is compared.

Operating System

macOS

Engram Version

1.16.3

Agent / Client

Claude Code

📋 Relevant Logs

# Verified by source inspection (engram 1.16.3):
internal/store/store.go:4996  backfillProjectSyncMutationsTx -> session/observation/prompt only
internal/store/store.go:5009  projectNeedsBackfill           -> counts session/observation/prompt only
internal/store/store.go:5364  enqueueSyncMutationTx          -> no enrollment gate (obs/sessions/prompts always enqueue)
internal/store/relations.go:653  JudgeRelation               -> "return nil // not enrolled — no mutation enqueued"
internal/store/relations.go:687  JudgeRelation               -> the ONLY relation enqueue in the codebase
internal/store/relations.go:732  JudgeBySemantic             -> inserts/updates memory_relations, never enqueues
grep -rn "backfillRelation" internal/ --include="*.go"  -> no match (no relation backfill exists)

💡 Additional Context

• Sibling of fix(sync): local chunk sync silently drops memory_relations without warning #353 (fix(sync): local chunk sync silently drops memory_relations) but for the cloud journal path. The local fix (PR fix(sync): export pre-existing relations missing from local chunks #494) reads relations from the table on export; the cloud path cannot, because it syncs from the journal.
• Open question on intent: a Phase-1-era test documents REQ-009: memory_relations are local-only in Phase 1 (internal/mcp/mcp_conflict_loop_test.go:621). Since JudgeRelation now enqueues relations and fix(store,chunkcodec): cloud sync blocked by unsupported relation mutations #313/fix(sync): accept cloud relation mutations #379/cloud upgrade legacy evaluator does not whitelist relation entity (regression after #379) #383 explicitly enabled relation cloud sync, that constraint appears superseded — but please confirm whether full relation cloud replication is the intended end state before this is implemented.
• Suggested fix direction: add a backfillRelationSyncMutationsTx mirroring the observation backfill (scan memory_relations where judgment_status != orphaned and no matching sync_mutations row for the enrolled project), include relations in projectNeedsBackfill, and have JudgeBySemantic enqueue on the enrolled path (or rely on the new backfill). Add regression coverage analogous to the fix(sync): local chunk sync silently drops memory_relations without warning #353 export test (internal/sync/sync_test.go:693).
• Workaround today: none clean for cloud. engram sync (local chunk) does replicate these relations, so cross-machine continuity via local sync is unaffected; only cloud replication loses them.

[forNerzul]

@Alan-TheGentleman as an external contributor I can't add labels. Could you add bug and, if you agree it's in scope, status:approved so I can open the PR? This is the cloud-journal sibling of #353/#494 — the fix I have in mind mirrors the existing observation/session/prompt backfill for relations. Happy to adjust direction based on the REQ-009 intent question in the body before I start.

[Alan-TheGentleman]

@forNerzul I'm about to create a PR for it, can you please compare it with yours and do all the needed changes? I'm trying to fix the issue together as this is really important for me :) And THANKS for all your hard work!!!!

[Alan-TheGentleman]

Merged to main, and genuinely, thank you. The hard part was yours: you traced this to its root cause and sketched the fix direction (mirror the observation/session/prompt backfill for relations), which made the implementation almost mechanical. It built right on your earlier #494 work.

When you can, please test it as soon as possible. You have the real cloud-enrolled cognicion-modular setup that reproduces the silent drop, so your validation matters more than anything CI can assert:

git fetch origin main && git checkout main && git pull
go build -o ./engram ./cmd/engram

On an enrolled project, trigger a JudgeBySemantic verdict (mem_compare / ScanProject), or start engram on a project that already has relations, and confirm a relation row now appears in sync_mutations and reaches cloud (it was 0 before).

I left this issue open with its labels on purpose: it is yours to close once you confirm it works. And if anything is off, or you want to take the next one, the door is open. Please keep contributing.

[forNerzul]

Confirmed on a self-hosted engram-cloud built from main @ 36c0819 (which includes #497) — the #496 fix works. Closing as resolved.

I didn't have a pre-existing cloud-enrolled cognicion-modular, so rather than asserting the fix from the diff, I built and ran the cloud server from this same main and validated end-to-end against it. It's the same server code a deploy runs — only auth differs, which doesn't touch the mutation/materialization path.

Setup:

• Client + server both built from main @ 36c0819 (includes fix(store): backfill relation sync mutations so relations reach cloud #497).
• Server: docker-compose.cloud.yml smoke stack — Postgres + cloud server, dev mode (ENGRAM_CLOUD_INSECURE_NO_AUTH=1).
• Client (1.16.4-dev): imported a copy of the real cognicion-modular project (63 observations) into an isolated client DB so the test data is real, not synthetic.

What I exercised — the real caller, not a synthetic mutation insert:
I drove a real mem_compare verdict over MCP between two real observations — "El descenso" ↔ "Patrones defensivos contra el descenso" — so the path went through JudgeBySemantic and hit the in-line enqueue at relations.go:897.

Results:

• sync_mutations relation row went 0 → 1 (it was 0 before the verdict).
• It reached the server: cloud_mutations now has entity=relation, entity_key=rel-c4ec07c0f6d4513d, op=upsert, project=cognicion-modular, via the autosync /sync/mutations/push path.
• I also validated the backfill leg on a separate isolated DB: a pre-existing journal-orphaned relation → enroll → backfillRelationSyncMutationsTx → autosync → cloud. Both the new-verdict path and the backfill path land.

Thanks, Alan — both for the fast turnaround on this and the generous credit. Really appreciate the collaboration.

One heads-up, concrete now: while validating end-to-end I ran into two adjacent gaps that #497 does not touch, so I'll file them separately rather than muddy this thread:

1. The server chunk-ingest path does not materialize relations into cloud_mutations — a parity gap with the mutation-push path (fix(store,chunkcodec): cloud sync blocked by unsupported relation mutations #313/fix(sync): accept cloud relation mutations #379/cloud upgrade legacy evaluator does not whitelist relation entity (regression after #379) #383), the leg the fix(store): relations missing a sync_mutations row never reach cloud (no relation backfill) #496/fix(store): backfill relation sync mutations so relations reach cloud #497 work built on. Same destination (cloud_mutations), different ingest leg: handlePushChunk → WriteChunk writes only cloud_chunks, never cloud_mutations.
2. The mutation-push returns an opaque 500 that fails the whole project batch when a single observation has empty content (validation-vs-canonicalization asymmetry, no push-side dead-letter/repair).

None of these block closing #496 — the row-creation + push path you fixed here is solid. Thanks again.