claims sidecar over markdown vs derived_from in postgres: where does braindb fork?

[kiluazen]

Markdown wikis are flat and unstructured: the LLM has to grep, read whole files into context, and manage linking by hand.

Recently, yologdev/yopedia #139 committed to a 3-phase adoption of this exact decomposition (Phase A hybrid raw anchors, Phase B claims sidecar with per-claim source plus lineage, Phase C anchor-verifier lint), tracked at #140 against the schema sketch at this gist. Three independent agent-wiki builders converged on the same three primitives: hybrid raw anchors, a commit-keyed ingest ledger, and post-ingest completeness plus staleness checks.

The interesting fork is where braindb has already gone further. BrainDB lifts provenance into a typed Postgres schema: supports, contradicts, derived_from, with markdown reduced to fragment input and the DB as substrate. Yopedia's bet keeps markdown as the substrate and lifts only the claims layer into a sidecar the agent can read and lint against. Same diagnosis (markdown-is-flat), opposite architectural conclusion (DB-as-substrate vs. claims-sidecar-over-markdown).

Two questions on the seam:

1. With derived_from already in place, what stopped braindb from also storing the in-source span (offset plus quote hash) rather than only the source-entity edge? Cost, recall quality, or something the schema rejected on principle?

2. On the ingest side, is there a per-ingest ledger row separate from the entities table, so a later reader can answer "which run wrote this fact from which version of which source"? Or does that information live implicitly in created_at plus source?

The yopedia adoption is the cleanest public case so far where someone keeps markdown and tries to bolt the verification layer beside it. Worth a comparison against braindb's choice.

• kushal

[dimknaf]

Thanks for the comparison — and for pointing me at yopedia, useful read.

BrainDB keeps three layers, and each one is fetched whole when it's the layer you need:

• Facts and keywords — the small atomic pieces. This is where local knowledge lives. Each fact has its own certainty, importance, embedding, and a link back to where it came from. This is also the layer the agent searches and ranks against. To navigate the memory, you go through facts and keywords.
• Wikis — the overall understanding for a subject. Built by the maintainer/writer agents on top of the facts. When you want the synthesized view, you read the wiki whole; it cites each claim by the fact's ID so you can drill into the local piece.
• Sources — the original document, kept whole. When you want the raw context (verification, background reading), you fetch the source whole.

Span anchors would be a fourth, narrower layer — pointing at a specific paragraph inside the source. The reason BrainDB doesn't need that today: the fact already IS the citation unit. Whatever the anchor would point at, the fact has already been extracted at ingest time, with its own ID, its own provenance edge, and its own searchable embedding. So the wiki cites the fact, the fact links to the source, the source is whole. The chain is navigable without offsets.

That's the real difference vs yopedia:

• yopedia keeps markdown as the source of truth and uses anchors to verify claims against it.
• BrainDB extracts the claims into typed entities at ingest and treats the graph as the source of truth. The markdown was just input.

Different bets, both defensible. Yopedia's is closer to "git + lint over markdown." BrainDB's is closer to "knowledge graph engine with a wiki view on top."

On your two specific questions:

(1) In-source span on derived_from — not done, but I'd split it in two:

• Heavy anchors (offset + quote hash for tamper detection) — feels like it duplicates the fact itself. If a span is meaningful enough to verify against, it's meaningful enough to be its own fact entity, which is what already happens.
• Light pointers (source ID + paragraph index, just for navigation) — that's a different story. It would let the wiki UI jump straight to the right part of a long source. I'll carefully consider lightweight pointers in a future version — there's real value there for traceability.

Short version: anchors-as-verification probably no, light pointers-as-navigation worth considering down the road.

(2) Per-ingest ledger — partly there. Each ingested file becomes a datasource entity, facts edge to it via derived_from, and the activity_log records the operation. So "which run wrote this fact from which version of which source" is reconstructable across joins, but not from a single ledger row. A dedicated ingest_event table (run_id, file_path, file_hash, started_at, finished_at, fact_count) would collapse that into one query and make clean rollback of bad ingests possible. Not on the roadmap yet, but a clean addition the day someone needs it.

Happy to keep this open as a tracking issue.