aoreshkov · aoreshkov · Aug 10, 2026 · Aug 10, 2026
@@ -139,6 +139,19 @@ differential suites that hold every claim here in `.claude/rules/testing.md`.
   per element. An element ordinal space was refused against it; the numbers are in the open-work
   index's Tier 2 record.
 
+- **`explain` may report a type mismatch and may never repair one.** `ExplainTypeNote` says when a
+  leaf's family disagrees with the types the catalog observed at its path — a numeric comparison where
+  a fifth of the values are strings — because in a third-party archive that is the normal state of the
+  world and the symptom is a query returning fewer rows with nothing to say why. It changes no answer,
+  no plan and no bound, which is exactly why `explain` is the right place for it: nothing there can be
+  got wrong in a way that costs a document. **Anything that made a numeric predicate match a string
+  would be a second definition of `ColumnPredicate.matches` and would break skipping.** The family
+  travels on `Normal.Leaf` from the lowering, because `ColumnPredicate.kind` is `internal` to
+  `rabosh-index` and re-deriving it in the query layer would be that second definition arriving by the
+  back door. A leaf that brackets to nothing — `EXISTS`, `IS NULL`, a mixed `IN` — reports nothing,
+  since there is no family for the data to disagree with, and the note is over **every** leaf rather
+  than only indexed ones, because a path with no index is where a caller has no other signal at all.
+
 - **A negated leaf is never a flipped operator.** `not($.a >= 10)` holds for a document whose `a` is
   a string and for one with no `a`; `$.a < 10` holds for neither. The normaliser keeps the negation on
   the leaf and applies it to the *document's* answer. The rewrite is the most natural-looking

@@ -44,3 +44,48 @@ Cross-cutting rules are indexed in the root `CLAUDE.md`; module conventions in
 - **`DocumentStore.write` publishes the read bound last.** A batch is atomic to one view because
   `visibleSequence` moves only once every operation is in the memtable. The unit is the view, not
   the call.
+
+- **A checkpoint obeys the ordering rule in the *target* directory, not only in the source.** *Log,
+  then memtable, then segment, then manifest, then delete* governs `writeCheckpoint` exactly as it
+  governs a flush: every data file is linked or copied **and forced** before the manifest naming it is
+  written, and `CURRENT` is written last of all. A checkpoint that forced its manifest first is the
+  same bug in a new place and fails the same way — a directory that opens and then cannot find a
+  segment. Two consequences that are decisions rather than details. The **snapshot is held open across
+  the copy**, which is what stops a compaction reclaiming a segment out from under it; that is why the
+  snapshot is in the design rather than being a way to pick a sequence. And **no log is copied**,
+  which is correct only because `checkpoint` flushes first — a copied log would replay into sequence
+  numbers the checkpoint's own manifest has already issued.
+
+  What core copies is *every file numbered after a live segment*, never a list of suffixes:
+  `rabosh-core` does not know what a `.cat`, `.idx`, `.pst` or `.col` is and must not learn, so a
+  sidecar kind added later travels with no change here. The one file that is **named** rather than
+  numbered is `INDEXES`, which is `IndexCatalog.copyRegistryTo`'s job and is why `Rabosh.checkpoint`
+  exists rather than the facade delegating and stopping. Losing it would lose an *instruction* rather
+  than derived data — the inversion `index-and-query.md` states — and would leave the posting files as
+  orphans for the next sweep.
+
+  A failed checkpoint is not unwound, deliberately: the target is not a store until `CURRENT` names
+  its manifest, so a partial one is a directory to throw away rather than a state to repair. What is
+  asserted instead is that the **source is unharmed**, at every step, by the fault-injecting
+  filesystem. Note which step the fault is armed on: the segments are *hard-linked*, so no byte is
+  written for one and a `WRITE` fault never fires — `FORCE` is the step that happens either way, and
+  it is the one the ordering rule is about.
+
+- **`deleteRange` is point deletes, and staying that way is the decision.** One snapshot scopes the
+  whole loop, keys are collected a batch at a time rather than all at once, and the next scan resumes
+  at `Key.successor()` of the last key handled — an inclusive bound restarted at that key would rescan
+  a range whose head is now a tombstone. No new operation id, no format change, no change to what a
+  merge emits or what `EntryCursor` collapses, and above all no change to the tombstone-drop rule,
+  which is on the short list of invariants that fail by returning a deleted document to a reader. A
+  real range tombstone is the other design and the format has room for it; it needs the
+  bytes-written-per-byte-retired measurement first, and that question belongs in the open-work index.
+
+- **The `LOCK` file's byte zero is the lock and everything after it is a diagnostic.** `tryLock()`
+  with no arguments takes `[0, Long.MAX_VALUE)`, and a Windows file lock is *mandatory* — so a second
+  process could not read a holder record written inside it, which is exactly when it wants to. Locking
+  one byte and writing `pid=… startedAt=…` after it leaves the record readable on every platform, and
+  the two regions overlap at byte zero so a build using either scheme still excludes one using the
+  other. The record is **not** forced and is not part of any ordering rule: losing it costs a better
+  error message and never a document. An empty record reads as *holder unknown*, which is what a store
+  last opened by an older release looks like, and `LockHolder.isRunning` checks the start time as well
+  as the pid because pids are reused.
@@ -15,6 +15,59 @@ else may change in any release. That claim lives in [STABILITY.md](STABILITY.md)
 
 ### Added
 
+- **`Rabosh.checkpoint(target)` — a consistent copy, taken while you are writing.** The recipe it
+  replaces was *stop writing and copy the directory*, which a desktop application cannot do because it
+  is the writer. The database is flushed, a snapshot is pinned, and the copy is of what that snapshot
+  sees, so it holds exactly the acknowledged prefix as of `CheckpointInfo.sequence`. Segments are
+  hard-linked where the filesystem allows it, so it costs a directory entry per file rather than its
+  bytes — which also means it is a consistent *view* rather than an off-site backup.
+
+  Sidecars travel with their segments and are **read** by the copy rather than rebuilt, verified by
+  opening the checkpoint with backfilling off. The `INDEXES` registry travels too, because an index
+  definition is an instruction somebody gave rather than derived data. The fault-injecting filesystem
+  fails the copy at four steps and the **source is unharmed in every case**.
+
+- **`Rabosh.deleteRange(from, to)` — retention by key range**, both bounds inclusive and both
+  optional. The loop a caller would otherwise write, which to write correctly means knowing four
+  invariants of the storage layer. Deliberately point deletes rather than an LSM range tombstone: no
+  new operation id, no format change, no change to compaction. Follow it with `compact()`, which is
+  what turns tombstones into reclaimed space.
+
+- **`JsonPathLimits` — bounded evaluation for untrusted JSONPath.** `rabosh-jsonpath`'s chosen use
+  case is expressions you did not write, and until now nothing bounded what a *small, valid* query
+  cost against a *large* document: `$..*..nope` is fourteen characters, is quadratic in the document's
+  node count, and returns the empty nodelist — so nothing measured on the answer can see it coming.
+
+  **The bound refuses; it never truncates.** Exceeding it raises `JsonPathLimitExceededException` and
+  delivers nothing, because a short nodelist cannot be told from a small document. Counted in steps
+  and never on a clock, for the reason the I-Regexp bound is. All 703 compliance cases and the
+  module's 20 000-deep and 5 000-wide fixtures pass under the shipped defaults, which are a backstop
+  rather than a policy — a deployment serving hostile expressions should set its own, far lower.
+
+- **`explain()` says when a predicate cannot match the data's types.** A numeric comparison against a
+  path where a third of the values arrive as strings now carries a note on the plan. A *diagnostic,
+  never a coercion*: type bracketing is unchanged, `ColumnPredicate.matches` is still the only
+  definition, and nothing here makes a numeric predicate match a string. Reported for leaves with no
+  index too, which is where a caller has no other signal at all.
+
+- **`Variant.detached()` and `InferredSchema.shreddingAdvice()` — the lakehouse hand-off**, with no
+  Parquet dependency taken. A document read from a segment carries *that segment's* shared dictionary,
+  so handing `(metadata, value)` to something expecting a self-contained Variant is a trap that
+  sometimes works — `detached()` rebuilds it with a dictionary of its own. The advice renders what the
+  catalog already computes for a Parquet **shredding schema**, including the decision a hand-written
+  one gets wrong: whether `variant_value` can be dropped.
+
+- **`StoreLockedException` says who holds the directory.** It carries `directory` and a `LockHolder`
+  with the pid and start time the `LOCK` file records, so a desktop application's second launch can
+  focus the existing window instead of matching on a message. **The start time is not decoration** —
+  operating systems reuse pids, and `isRunning` checks both, so a user is never told to kill a
+  stranger's process. No lock stealing, no timeout, no force-open.
+
+- **`:rabosh-samples:runDrain`** — a staging buffer drained: snapshot, ship, record the watermark,
+  retire, compact, in that order. It composes the two items above, which makes it their acceptance
+  test in the only way that matters, a caller's program. Also **`Key.successor()`**, promoted to
+  public because writing that loop found it was the one thing the inclusive bounds cannot say.
+
 - **[`INTEGRATION.md`](INTEGRATION.md) — the runtime contract, in public.** The rules an embedding
   application has to obey were discoverable only by reading KDoc on classes a caller may never open,
   and three of them fail *silently*: a row is valid only until the next `next()`, a leaked `Snapshot`

@@ -102,6 +102,7 @@ it fails **silently** — a document missing from a result, a file that stops me
 - A null slot holds the type's zero, and that zero must never reach a bound.
 - Type bracketing is part of the query contract, because skipping depends on it.
 - A negated leaf is never a flipped operator.
+- `explain` may report a type mismatch and may never repair one.
 - A plan's candidates are a superset and its certainties a subset, and the gap is what gets read.
 - A composite index needs every declared field fixed by equality and does not care what else the query
   asks: **more** conjuncts are dropped and cost it only its certainty, **fewer** are unsound.
@@ -115,6 +116,10 @@ it fails **silently** — a document missing from a result, a file that stops me
 
 ### The write path — `.claude/rules/storage-durability.md`
 
+- A checkpoint obeys the ordering rule in the target directory, not only in the source.
+- `deleteRange` is point deletes, and staying that way is the decision.
+- The `LOCK` file's byte zero is the lock and everything after it is a diagnostic.
+
 - The log is appended before the memtable is touched, always.
 - A torn tail may be dropped; anything that would lose an acknowledged commit is reported.
 - A tombstone may only be dropped at the bottom-most level that can contain the key, and only below

@@ -38,8 +38,13 @@ are embedded in a larger file and carry a version without one.
 | Bitmap block | *embedded* | — | 1 |
 | Variant metadata | *embedded* | — | 1 |
 
-`CURRENT` and `LOCK` carry no version. `CURRENT` holds one manifest name and nothing else; `LOCK` holds
-nothing at all.
+`CURRENT` and `LOCK` carry no version. `CURRENT` holds one manifest name and nothing else. `LOCK` held
+nothing at all until 0.3.0 and now carries one line of ASCII naming the process that holds it —
+`pid=… startedAt=…` — which is a **diagnostic and not a format**: nothing reads it except a process
+that has just failed to take the lock, an empty one reads as "holder unknown", and no guarantee in
+this document covers it. The lock itself is on byte zero and the record begins after it, so a holder
+can be read while it is being held; a build that locks the whole file and a build that locks byte
+zero still exclude each other, which is what keeps releases mixable on one directory.
 
 The magics are spelled `JKDB-` because the project was called `jsonkdb` when the format was written.
 A magic is a discriminator saying which kind of file this is, never branding, so the prefix is

@@ -63,14 +63,37 @@ space.
 
 A second attempt raises `StoreLockedException`, distinguishably in each direction — `… is already
 open in this process` when it is your own code, `… is locked by another process` when it is not. Held
-by `DocumentStoreTest`'s *a second store cannot open the same directory*.
+by `DocumentStoreTest`'s *a second store cannot open the same directory* and by `StoreLockTest`,
+which takes the lock from a second **JVM** because that is the only way to reach the branch a real
+second launch takes.
 
 **For a desktop or CLI application this is the normal second-launch case, not a fault.** Catch
 `StoreLockedException` specifically — it is a distinct subtype of the sealed `StoreException` — and
-focus the existing window, or exit with a message. Do not catch `Exception` and treat it as
-corruption, and do not delete the lock file: it is a real advisory lock held by an open channel, and
-removing it produces the two-writer state it exists to prevent. There is no force-open and there will
-not be one.
+read the details off the exception rather than out of its message:
+
+```kotlin
+val db = try {
+    Rabosh.open(directory)
+} catch (locked: StoreLockedException) {
+    val who = locked.holder
+    when {
+        who != null && who.isRunning -> focusExistingWindow(who.pid)
+        else -> reportAlreadyOpen(locked.directory)   // holder unknown, or the record is stale
+    }
+    return
+}
+```
+
+`holder` is `null` when nothing could be read — a store last opened by a release before 0.3.0 wrote
+no record — and `LockHolder.isRunning` is false when the recorded process is gone, which means the
+record is stale rather than that the lock is free. **`isRunning` checks the start time as well as the
+pid**, deliberately: operating systems reuse process ids, and a user who is told to kill pid 4242 on
+the strength of a pid alone can kill a stranger's process.
+
+Do not catch `Exception` and treat it as corruption, and **do not delete the lock file**: it is a
+real advisory lock held by an open channel, and removing it produces the two-writer state it exists
+to prevent. There is no force-open, no timeout and no lock stealing, and there will not be — each of
+those converts a clear failure into a corrupt store.
 
 **Within the process, one writing thread.** Any number of threads may read concurrently — snapshots,
 scans and queries are all safe — and `Rabosh` guards the cached planner statistics behind `query`
@@ -141,19 +164,57 @@ asked for the load. Do not report success before the `sync()`.
 
 ## Taking a copy of a store
 
-**There is no `checkpoint` yet.** Until there is, the only defined way to copy a store is:
+```kotlin
+val info = db.checkpoint(Path.of("backup", "2026-08-10"))
+// info.sequence — every commit at or below it is in the copy, nothing above it is
+Rabosh.open(info.directory).use { copy -> /* a database, indexes and model included */ }
+```
+
+**Safe to call while you are writing**, which is the point: an application that is itself the writer
+cannot stop to be copied. The database is flushed, a snapshot is pinned, and the copy is of what that
+snapshot sees. Commits that land during the call are above `info.sequence` and are simply not in it.
+
+Everything travels — segments, their `.cat`, `.idx`, `.pst` and `.col` sidecars, and the `INDEXES`
+registry — and the sidecars are **read** by the copy rather than rebuilt, so an index the original
+paid a scan for is not paid for twice.
+
+Three things to know before you rely on it:
+
+- **The target must be empty or absent.** A checkpoint is never merged into a directory that already
+  holds a store; that would open, and be wrong.
+- **It is a consistent view, not an off-site backup.** Files are hard-linked where the filesystem
+  allows it (`info.hardLinked` says), so the copy costs a directory entry per file rather than its
+  bytes — and shares blocks with the original. Moving it somewhere else is what makes it a backup,
+  and that step is yours.
+- **A failure leaves a partial target and never touches the source.** There is no unwind, because the
+  directory is not a store until `CURRENT` names its manifest; throw it away and take another.
+
+`DocumentStore.checkpoint` is the same thing one layer down, and carries everything except the index
+registry — which is `IndexCatalog`'s file, and is why `Rabosh.checkpoint` exists rather than the
+facade just delegating.
+
+## Retention
+
+```kotlin
+val retired = db.deleteRange(to = Key.of("event:2026-07-31"))
+db.compact()
+```
+
+Both bounds are inclusive and both are optional. This is the loop you would otherwise write, and
+writing it correctly means knowing four things that are not on any signature — that the scan must be
+scoped by a snapshot, that the deletes belong in a batch, that a tombstone is reclaimed by compaction
+rather than by the delete, and that a tombstone may only be dropped at the bottom-most level below the
+oldest live snapshot.
 
-1. Stop writing. Not "pause the ingest thread" — no `put`, `delete` or `write` may be in flight.
-2. `db.flush()`, which returns when the memtable is on the platter and the manifest names it.
-3. Copy the whole directory, including `CURRENT`, `MANIFEST-*`, every `.wal`, `.seg`, `.cat`, `.idx`,
-   `.pst` and `.col`. Not a subset: the manifest names the files it expects.
-4. Resume writing.
+**`deleteRange` writes one tombstone per key**, deliberately: it is point deletes rather than an LSM
+range tombstone, so it costs nothing in the format and its cost is proportional to the number of keys
+deleted. **Follow it with `compact()`** when the space matters — the tombstones make the keys
+disappear, and compaction makes the tombstones disappear.
 
-**A directory copied while a writer is running is not defined to be recoverable**, and a copy that
-skips the log or the manifest is not a store. Neither failure is loud — the copy usually opens and is
-usually missing something.
+It is atomic per batch rather than overall, and the range is emptied as of the moment you called it,
+so keys written during the call survive and a repeated call converges instead of racing a writer.
 
-`LOCK` may be copied or not; it holds nothing.
+`:rabosh-samples:runDrain` is this and `checkpoint` in the order a staging buffer actually uses them.
 
 ## Version pinning