v2 architecture: document-as-directory, multi-part, Merkle DAG storage

[Eth-Interchained]

Vision

The fundamental unit of storage should be a directory, not a log entry.

A document isn't a row in a flat append-only log — it's a content-addressed filesystem directory with named parts, explicit causal references, and a per-document history timeline. This is IPFS architecture with NQL, BLAKE2b tamper-evidence, and at-rest encryption.

Layout

nedb-data/
  {database}/
    collections/
      {coll}/
        {doc_id}/
          head.json        ← current state (fast read, always small)
          parts/
            {part}.json    ← addressable sub-objects (lazy-loaded)
          history/
            {seq}.op       ← every operation that touched this doc
          refs/
            {name} → {coll}/{doc_id}   ← causal graph edges
    indexes/
      {coll}/
        {field}.sorted     ← BTree index file: O(log n) ORDER BY
        {field}.eq         ← hash index: O(1) WHERE field = value
    log/
      {000001}.seg         ← sealed AOF segment (immutable)
      {000002}.seg         ← active segment (being written)
    MANIFEST               ← BLAKE2b Merkle root over all parts + segments

What this unlocks

Partial reads

Load header.json (200 bytes) without touching txlist.json (50 KB). NQL SELECT on specific parts skips unrelated data entirely. For Vision's /api/blocks list, this means reading 30 block headers at ~200 bytes each instead of 30 full block JSONs at ~50 KB each — 250x less I/O.

Natural time travel without AOF replay

history/{seq}.op is the full document timeline at the doc level. AS OF seq=N → binary search history/ for the op at or before seq N and apply it. No global AOF replay on startup. Cold start = load MANIFEST + index files + active segment only.

First-class relational graph

refs/ are filesystem edges in the causal graph. TRACE caused_by walks refs/prev_block at each node — O(graph_depth) not O(all_docs). Every caused_by link is a named ref, queryable by name. This makes the causal graph a first-class citizen of the storage model.

Structural sharing

Two documents that reference the same transaction reference the same file path. Zero duplication for shared data. Deduplication is mechanical.

Merkle tamper evidence at scale

MANIFEST is a Merkle root. Each document's parts are hashed; hashes bubble up to the MANIFEST. Changing any byte anywhere invalidates the root hash AND you know exactly which document part was tampered with — pinpoint provenance, not just "something changed."

Parallel writes without Sequencer contention

Each document directory has its own write lock. Concurrent writes to different documents don't contend. The global Sequencer (current v1 bottleneck) is replaced by document-level locking + a lightweight global seq counter.

Migration path from v1

1. v1.3 (PR feat(engine): sorted indexes + segment AOF design — v1.3.0 #3 in progress): Sorted indexes — O(log n) ORDER BY. No storage format change.
2. v1.4: Segment-based AOF (SEGMENTS.md in PR feat(engine): sorted indexes + segment AOF design — v1.3.0 #3). Multiple sealed segments, snapshot at seal, fast cold start.
3. v2.0: Document-as-directory. Migration tool converts v1 AOF to v2 directory layout. v1 compatibility reader for old databases.

NQL surface changes

No breaking changes to NQL syntax. New capabilities enabled by the directory model:

-- Partial read: only load the header part
SELECT head, parts.header FROM blocks WHERE height = 618000

-- Walk the ref graph (already supported via TRACE; now O(depth) not O(n))
FROM blocks WHERE _id = "618000" TRACE refs.prev_block LIMIT 10

-- Time travel to a specific seq (now O(log doc_history) not O(full_AOF))
FROM blocks WHERE _id = "618000" AS OF seq=50000

Produced by

Interchained LLC × Claude Sonnet 4.6
Vision integration thread — June 16, 2026
Follows issue #2 (parallel writers, sorted indexes, streaming replication)