From 1f02b75ef39c3b085bbdb12cf1db189aeba44e7f Mon Sep 17 00:00:00 2001 From: Atanas Oreshkov Date: Mon, 10 Aug 2026 13:52:19 +0300 Subject: [PATCH] Say which parts of the Kotlin API may move, and retract the flag nobody needed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes every embedding of this engine needs, and none of them touches the engine. Say which declarations are allowed to move. "Major version zero: any signature may change in any release" was honest and unactionable — a consumer could not tell whether `Key.of` was as volatile as `IndexCatalog.readColumn`, so the only rational responses were to wrap all of the API or none of it. There are now two tiers, in STABILITY.md: a small stable core that moves only under a deprecation cycle, and everything else, marked `@RaboshExperimental`. It is deliberately a substitute for 1.0 rather than a step towards one. What is marked is the way *in*, not every member. Holding a `ColumnReader` means passing `Rabosh.indexCatalog` or `IndexCatalog.readColumn`, so both of those are marked and the reader's own methods carry nothing. Marking every member instead forces every stable signature naming an experimental type to be marked too, and that cascade ends with the stable core inside the experimental tier. `SegmentObserver` is that cascade caught at one step — `RaboshOptions`' constructor names it — so it is stable, deliberately. The marker lives in `rabosh-variant` because it has to: everything marked is below `rabosh-api` in the chain, and a marker declared there could not be applied in `rabosh-index` without the upward edge this project does not have. The ABI dumps are not the gate, and that is worth knowing rather than assuming. The JVM dump format writes signature lines and never annotations, so a declaration changing tier is invisible to `checkKotlinAbi` — confirmed by the markers changing the committed dumps by exactly one entry, the annotation class itself. What catches it is `rabosh-samples`: `:rabosh-api` and nothing else, `allWarningsAsErrors`, part of `build`, and the one module the opt-in is withheld from. It is a real consumer compiling against the stable core. Write the runtime contract down. Four rules an embedding application must obey were discoverable only by reading KDoc on classes a caller may never open, and three of them fail silently: a row is valid only until the next `next()`, a leaked `Snapshot` pins disk indefinitely, a second writing thread gets contention rather than an error, and a directory copied under a live writer is not defined to be recoverable. INTEGRATION.md is one page and every claim in it names the type, option or test that enforces it. Declare a module name in every published jar. Without `Automatic-Module-Name` the seven jars resolve on the module path under names derived from their filenames, which is unstable by construction and is where a jlink/jpackage build stops. Derived from the project name rather than listed, for the reason `PublishedModules` gives. Held by `:rabosh-samples:runThreeStepsOnModulePath`, which asks the JVM for `app.oreshkov.rabosh.api` by name — delete the attribute and it fails at boot-layer initialisation rather than resolving something else. And retract the native-access requirement, which was never real. `FileChannel.map(mode, offset, size, Arena)` is not a restricted method: it carries no `@Restricted` and declares no `IllegalCallerException` in JDK 25, and neither do `Arena.ofShared`, `Arena.allocate` or `MemorySegment.ofArray`. The restricted set is `MemorySegment::reinterpret`, the `Linker` and `SymbolLookup` entry points and the `load`/`loadLibrary` family, and nothing here calls one. So no module needs `--enable-native-access`, and the comment in `rabosh-samples` asserting otherwise is corrected in place. Settled by running the engine under `--illegal-native-access=deny` with no grant and watching it pass, and then — because a check nobody has watched fail proves nothing — confirming the same flag does kill a two-line program that calls `MemorySegment.reinterpret`. The module-path sample is where the claim now lives, and the module path is the only place it can: `ALL-UNNAMED`, which the two classpath samples pass, would cover a restricted call and hide the answer. The existing flags are kept as harmless future-proofing; only the reasoning is withdrawn. Co-Authored-By: Claude Opus 5 --- .claude/rules/build-and-release.md | 69 ++++++- .github/workflows/ci.yml | 8 + CHANGELOG.md | 51 ++++- CLAUDE.md | 12 +- COMPATIBILITY.md | 8 +- INTEGRATION.md | 191 ++++++++++++++++++ README.md | 25 ++- STABILITY.md | 134 ++++++++++++ .../kotlin/rabosh.kotlin-library.gradle.kts | 50 +++++ .../kotlin/app/oreshkov/rabosh/api/Rabosh.kt | 14 ++ rabosh-bench/build.gradle.kts | 12 ++ .../oreshkov/rabosh/catalog/HyperLogLog.kt | 2 + .../oreshkov/rabosh/catalog/InferredSchema.kt | 11 +- .../oreshkov/rabosh/catalog/SchemaCatalog.kt | 11 +- .../oreshkov/rabosh/catalog/SegmentSketch.kt | 2 + .../oreshkov/rabosh/catalog/ValueBounds.kt | 2 + .../oreshkov/rabosh/catalog/ValueSignature.kt | 2 + .../app/oreshkov/rabosh/core/DocumentStore.kt | 7 + .../app/oreshkov/rabosh/index/Bitmap.kt | 3 + .../app/oreshkov/rabosh/index/BitmapCursor.kt | 3 + .../app/oreshkov/rabosh/index/BitmapView.kt | 2 + .../oreshkov/rabosh/index/ColumnPredicate.kt | 2 + .../app/oreshkov/rabosh/index/ColumnQuery.kt | 3 + .../app/oreshkov/rabosh/index/ColumnReader.kt | 3 + .../oreshkov/rabosh/index/CompositeTerm.kt | 2 + .../oreshkov/rabosh/index/ElementExtractor.kt | 2 + .../app/oreshkov/rabosh/index/IndexCatalog.kt | 11 +- .../app/oreshkov/rabosh/index/IndexQuery.kt | 2 + .../app/oreshkov/rabosh/index/IndexReader.kt | 3 + .../oreshkov/rabosh/index/ReadableBitmap.kt | 3 + .../oreshkov/rabosh/index/RoaringPortable.kt | 2 + .../oreshkov/rabosh/index/TermExtractor.kt | 2 + .../app/oreshkov/rabosh/query/QueryEngine.kt | 3 +- rabosh-samples/build.gradle.kts | 77 ++++++- rabosh-testkit/build.gradle.kts | 14 ++ rabosh-variant/api/rabosh-variant.api | 3 + .../app/oreshkov/rabosh/RaboshExperimental.kt | 61 ++++++ 37 files changed, 778 insertions(+), 34 deletions(-) create mode 100644 INTEGRATION.md create mode 100644 STABILITY.md create mode 100644 rabosh-variant/src/main/kotlin/app/oreshkov/rabosh/RaboshExperimental.kt diff --git a/.claude/rules/build-and-release.md b/.claude/rules/build-and-release.md index 8557b78..e40a390 100644 --- a/.claude/rules/build-and-release.md +++ b/.claude/rules/build-and-release.md @@ -84,12 +84,63 @@ claim: `release.yml` derives the release version from the git tag and nowhere el development one by construction. Do not "fix" it to a release number — that would put the version in two places and make the tag advisory. -## The format claim - -**The format claim lives in `COMPATIBILITY.md` and nowhere else.** It used to live in the README's -status blockquote, coupled to the version in `gradle.properties`, and the coupling was what kept it: -each artefact cited another and none cited the format. The two guarantees are now separate and stated -at the strength of their own evidence — the **on-disk format is declared and stable**, held by the -golden stores; the **Kotlin API is major-version zero** and free to move, held by nothing, which is -exactly why it is not claimed. A change to either belongs in `COMPATIBILITY.md` first; the README -links it rather than restating it, so the two cannot drift. +## The format claim, the API claim and the runtime contract + +**Each lives in exactly one file, and the README links all three rather than restating any.** +`COMPATIBILITY.md` holds the on-disk format, `STABILITY.md` the Kotlin API, `INTEGRATION.md` the +runtime contract. The format claim used to live in the README's status blockquote, coupled to the +version in `gradle.properties`, and the coupling was what kept it: each artefact cited another and +none cited the format. + +The two guarantees are stated at the strength of their own evidence. The **on-disk format is declared +and stable**, held by the golden stores. The **Kotlin API is tiered** — a stable core that moves only +under a deprecation cycle, and an explicit `@RaboshExperimental` for the rest. It used to be +"major-version zero, held by nothing", which was honest and unactionable: a consumer could not tell +whether `Key.of` was as volatile as `IndexCatalog.readColumn`, so the rational response was to wrap +all of the API or none of it. Phase 23 replaced it with the smaller, truer claim, and **that is a +substitute for 1.0 rather than a step towards one** — say so wherever it is described. + +Three things about the marker that a change must not quietly undo. + +**It lives in `rabosh-variant`, package `app.oreshkov.rabosh`, and it has to.** Everything marked is +below `rabosh-api` in the chain, so a marker declared there could not be applied in `rabosh-index` +without the upward edge this project does not have. The package is the project's rather than the +codec's for the same reason. + +**What is marked is the way *in*, not every member.** `Rabosh.store`/`catalog`/`indexCatalog`, +`DocumentStore.open`, the `SchemaCatalog`/`IndexCatalog`/`QueryEngine` constructors, +`IndexCatalog.read`/`readColumn`, `SchemaCatalog.sketchOf`, `InferredField.sketch`, and the bitmap, +column and sketch *types*. Marking every member instead forces every stable signature naming an +experimental type to be marked too, and that cascade ends with the stable core inside the experimental +tier. `SegmentObserver` is that cascade caught at one step: `RaboshOptions`' constructor names it, so +marking the interface would have put `RaboshOptions(...)` behind an opt-in. It is stable, deliberately. + +**The gate is `rabosh-samples` not opting in, and the ABI dumps are not the gate.** The JVM dump +format writes signature lines and never annotations — verified in the dumper, and confirmed by the +markers changing the committed dumps by exactly one entry, the annotation class itself — so a +declaration changing tier is invisible to `checkKotlinAbi`. What catches it is the samples module: +`:rabosh-api` and nothing else, `allWarningsAsErrors`, part of `build`, and the one module the +opt-in is deliberately withheld from. Do not "tidy" that asymmetry by giving every module the same +compiler options; `rabosh.kotlin-library`, `rabosh-testkit` and `rabosh-bench` opt in, samples do not. + +## Native access: the flag nobody needs + +**No module requires `--enable-native-access`, and the reason is not that the engine avoids the FFM +API.** It maps every segment through `FileChannel.map(mode, offset, size, Arena)` — which is simply +**not a restricted method**: it carries no `@Restricted` and declares no `IllegalCallerException` in +JDK 25, and neither do `Arena.ofShared`, `Arena.allocate` or `MemorySegment.ofArray`. The restricted +set is `MemorySegment::reinterpret`, the `Linker` and `SymbolLookup` entry points and the +`load`/`loadLibrary` family, and nothing here calls one. + +This was believed otherwise for several phases and written into a build comment as fact, so it is +worth stating how it was settled: not by reading the JEP, but by running the engine under +`--illegal-native-access=deny` with no grant and watching it pass, and then confirming that the same +flag *does* fail a two-line program that calls `MemorySegment.reinterpret`. A check that has not been +seen fail proves nothing, and that applies to a check on the JVM's behaviour as much as to one in the +suite. + +`:rabosh-samples:runThreeStepsOnModulePath` is where the claim now lives, and the module path is the +only place it can: `ALL-UNNAMED`, which the other two samples pass, would cover a restricted call from +the classpath and hide the answer. The `--enable-native-access=ALL-UNNAMED` on `Test` tasks and on the +two classpath samples is retained as harmless future-proofing; the *reasoning* attached to it is not +load-bearing and should not be repeated as though it were. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1778c44..11ff0eb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,14 @@ jobs: if: matrix.os == 'ubuntu-latest' run: ./gradlew publishToMavenLocal --stacktrace + # The published jars carry `Automatic-Module-Name`, and the only thing that checks it is a run + # with the library on the module path — asked for by module name, so a missing attribute fails + # at startup rather than silently resolving the jar under a filename-derived name. It is also + # the one place the *module-name* spelling of `--enable-native-access` is exercised, which is + # what INTEGRATION.md tells a jlink/jpackage consumer to use. + - name: Run a sample on the module path + run: ./gradlew :rabosh-samples:runThreeStepsOnModulePath --stacktrace + # `publishToMavenLocal` builds each module's own Dokka HTML for the `javadoc` classifier, but # not the aggregated site — that is the root's `rabosh.api-docs`, and only `docs.yml` on `main` # would otherwise run it. A pull request that broke the aggregation would go green here and diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d578eb..61515b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,14 +5,55 @@ All notable changes to this project are recorded here. The format follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html) — with one qualification that matters more here than the version number does. -**Two guarantees, at different strengths.** The Kotlin API is major-version zero: any signature may -change in any release, and `0.x` gives you no compatibility promise at all. The **on-disk format** is -declared and stable — a store written by an earlier release opens on every later one — and that -promise does not wait for `1.0`. Anything affecting it is stated in -[COMPATIBILITY.md](COMPATIBILITY.md) first and only summarised here. +**Two guarantees, at different strengths, and neither waits for `1.0`.** The **on-disk format** is +declared and stable — a store written by an earlier release opens on every later one — and anything +affecting it is stated in [COMPATIBILITY.md](COMPATIBILITY.md) first and only summarised here. The +**Kotlin API** is tiered: a small stable core moves only under a deprecation cycle, and everything +else may change in any release. That claim lives in [STABILITY.md](STABILITY.md), on the same terms. ## [Unreleased] +### Added + +- **[`INTEGRATION.md`](INTEGRATION.md) — the runtime contract, in public.** The rules an embedding + application has to obey were discoverable only by reading KDoc on classes a caller may never open, + and three of them fail *silently*: a row is valid only until the next `next()`, a leaked `Snapshot` + pins disk indefinitely, and a second writing thread gets contention rather than an error. One page, + every claim naming the type, option or test that enforces it. + +- **[`STABILITY.md`](STABILITY.md) and `@RaboshExperimental` — which parts of the Kotlin API are + allowed to move.** "Major version zero, any signature may change" was honest and unactionable: a + consumer could not tell whether `Key.of` was as volatile as `IndexCatalog.readColumn`, so the only + rational responses were to wrap all of the API or none of it. There are now two tiers — a small + stable core that moves only under a deprecation cycle, and everything else, marked with an opt-in + requirement. It is deliberately **not** a promise of 1.0. + + What is marked is the way *in* rather than every member: `Rabosh.store`/`catalog`/`indexCatalog`, + `DocumentStore.open`, the `SchemaCatalog`/`IndexCatalog`/`QueryEngine` constructors, + `IndexCatalog.read`/`readColumn`, and the bitmap, column and sketch types themselves. Holding one + of those objects means you already opted in, so its own methods carry nothing. + + `rabosh-samples` is what holds the claim, and it holds it by *not* opting in: it depends on + `:rabosh-api` alone, compiles with `allWarningsAsErrors`, and is part of `build`, so it is a real + consumer compiling against the stable core. The ABI dumps cannot do this job — the JVM dump format + writes signatures and never annotations, so a tier change is invisible to `checkKotlinAbi`. + +- **`Automatic-Module-Name` in every published jar**, derived from the module name: + `app.oreshkov.rabosh.{variant,core,catalog,index,query,api,jsonpath}`. On the module path the jars + previously resolved under names derived from their filenames, which is unstable by construction and + is where a `jlink`/`jpackage` build stopped. Held by a new + `:rabosh-samples:runThreeStepsOnModulePath`, which asks the JVM for `app.oreshkov.rabosh.api` **by + name** — so a missing attribute fails at startup rather than silently resolving something else. + +### Changed + +- **No `--enable-native-access` flag is required, by any module**, and `INTEGRATION.md` now says so. + The engine maps segments through `FileChannel.map(mode, offset, size, Arena)`, which is *not* a + restricted method — it carries no `@Restricted` and declares no `IllegalCallerException` — and + nothing here calls one that is. The new module-path task runs the full cycle under + `--illegal-native-access=deny` with no grant of any kind, so the claim is checked rather than + asserted, and a future release that acquires a restricted call fails it. + ### Compatibility - **The composite index's on-disk shape is now pinned by committed bytes**, in a fifth golden store diff --git a/CLAUDE.md b/CLAUDE.md index ba32a4c..67253db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -59,9 +59,15 @@ them at the latest stable release; do not adopt pre-releases (e.g. Kotlin `-Beta `./gradlew -p build-logic check` as its own step. - **`gradle.properties` stays `0.1.0-SNAPSHOT`**: `release.yml` derives the release version from the git tag and nowhere else. Do not "fix" it to a release number. -- **The format claim lives in `COMPATIBILITY.md` and nowhere else**, and the README links it rather - than restating it. The on-disk format is declared and stable; the Kotlin API is major-version zero - and is deliberately not claimed. +- **The format claim lives in `COMPATIBILITY.md` and the API claim in `STABILITY.md`, each in one + place**, and the README links both rather than restating either. The on-disk format is declared and + stable; the Kotlin API is tiered, with a stable core and an explicit `@RaboshExperimental`. +- **The runtime contract lives in `INTEGRATION.md`** — JDK floor, the native-access question, one + writer, the `AutoCloseable`s, copy-before-`next()`. A sentence about bytes on disk belongs in + `COMPATIBILITY.md` and is linked, never moved. +- **No module needs `--enable-native-access`, and that is checked**: + `:rabosh-samples:runThreeStepsOnModulePath` runs under `--illegal-native-access=deny` with no grant. + `FileChannel::map` is not a restricted method; do not add the flag back on the assumption that it is. ## Module layout diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index d832f7d..4048185 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -9,11 +9,17 @@ rather than in code; this is that declaration, and it is deliberately separate f | | Guarantee | |---|---| | **On-disk format** | Stable. Governed by this document. | -| **Kotlin API** | Not stable. Major version zero: any signature may change in any release. | +| **Kotlin API** | Tiered. A small stable core moves only under a deprecation cycle; everything else may change in any release. Governed by [STABILITY.md](STABILITY.md). | A store is data somebody owns; an API is a call somebody can rewrite. Freezing the first and not the second says exactly what the evidence supports, and no more. +The API row used to read *not stable, major version zero, any signature may change in any release*. +That was honest and unactionable — it gave a consumer no way to tell whether `Key.of` was as volatile +as `IndexCatalog.readColumn` — so it has been replaced by the smaller, truer claim in its own +document. It is **not** a promise of 1.0 and is not a step towards one; the two files stay separate +for the same reason they always were, and each links the other rather than restating it. + ## What is covered Ten independently versioned encodings. Eight carry an eight-byte magic, legible in a hex dump; two diff --git a/INTEGRATION.md b/INTEGRATION.md new file mode 100644 index 0000000..d4734d7 --- /dev/null +++ b/INTEGRATION.md @@ -0,0 +1,191 @@ +# Integrating rabosh + +The rules an embedding application has to obey. This is a **contract, not a tutorial** — the +[samples](README.md#samples) are the tutorial and the [README](README.md) is the argument. Everything +here is a rule that, if broken, either fails silently or does not fail until production. + +Three of the four in "Lifetimes" are the silent ones. If you read nothing else, read those. + +Related: [COMPATIBILITY.md](COMPATIBILITY.md) for the on-disk format, [STABILITY.md](STABILITY.md) +for which Kotlin declarations are allowed to move. + +## The runtime + +**JDK 25 or later.** Not a floor picked for tidiness: the engine maps every segment through +`FileChannel.map(mode, offset, size, Arena)` and reads it as a `MemorySegment`, so segments are +unmapped deterministically when their arena closes rather than whenever a garbage collector gets +round to a `ByteBuffer`. That API is final from JDK 22; 25 is the LTS. + +**No `--enable-native-access` flag is required, by any module.** This is worth stating explicitly +because it is easy to assume otherwise from the fact that the engine uses the Foreign Function & +Memory API, and because adding the flag "to be safe" propagates into launcher scripts, Dockerfiles +and IDE run configurations that then outlive the reason for them. + +The flag governs **restricted** methods — `MemorySegment::reinterpret`, `Linker::downcallHandle`, +`SymbolLookup::libraryLookup`, `System::loadLibrary` and their neighbours. rabosh calls none of them. +`FileChannel::map` is not among them: in JDK 25 it carries no `@Restricted` annotation and declares no +`IllegalCallerException`, and neither does `Arena.ofShared`, `Arena.allocate` or +`MemorySegment.ofArray`. + +This is checked rather than asserted. `./gradlew :rabosh-samples:runThreeStepsOnModulePath` runs a +full write/model/index/query cycle under `--illegal-native-access=deny` with **no** grant of any kind, +on the module path, where the engine's code sits in a named module that `ALL-UNNAMED` would not cover +even if it were passed. If a future release acquires a restricted call, that task fails. + +> If you are running an older JDK than 25 you may see a warning; that is JDK 22-24 behaviour and not +> a supported configuration. If you ever do need the grant — because a future release takes a +> restricted call — the spelling on the module path is `--enable-native-access=app.oreshkov.rabosh.core`, +> not `ALL-UNNAMED`. + +**On the module path**, each published jar declares its own name via `Automatic-Module-Name`, so +`jlink` and `jpackage` builds resolve them stably rather than under a name derived from the filename: + +| Artefact | Module | +|---|---| +| `rabosh-api` | `app.oreshkov.rabosh.api` | +| `rabosh-query` | `app.oreshkov.rabosh.query` | +| `rabosh-index` | `app.oreshkov.rabosh.index` | +| `rabosh-catalog` | `app.oreshkov.rabosh.catalog` | +| `rabosh-core` | `app.oreshkov.rabosh.core` | +| `rabosh-variant` | `app.oreshkov.rabosh.variant` | +| `rabosh-jsonpath` | `app.oreshkov.rabosh.jsonpath` | + +These are automatic modules — there is no `module-info.java` — so they read every other module on the +path and export every package. `kotlin-stdlib` ships a real module descriptor, so name it explicitly +if nothing else already requires it. + +## One process, one writer + +**One process may have the directory open, and within it one `Rabosh` (or one `DocumentStore`).** The +lock file is what makes that a guarantee rather than a convention: two writers over one LSM directory +do not produce a merge conflict, they produce two interleaved logs and an unrecoverable sequence +space. + +A second attempt raises `StoreLockedException`, distinguishably in each direction — `… is already +open in this process` when it is your own code, `… is locked by another process` when it is not. Held +by `DocumentStoreTest`'s *a second store cannot open the same directory*. + +**For a desktop or CLI application this is the normal second-launch case, not a fault.** Catch +`StoreLockedException` specifically — it is a distinct subtype of the sealed `StoreException` — and +focus the existing window, or exit with a message. Do not catch `Exception` and treat it as +corruption, and do not delete the lock file: it is a real advisory lock held by an open channel, and +removing it produces the two-writer state it exists to prevent. There is no force-open and there will +not be one. + +**Within the process, one writing thread.** Any number of threads may read concurrently — snapshots, +scans and queries are all safe — and `Rabosh` guards the cached planner statistics behind `query` +itself. What is not safe is two threads calling `put`, `delete` or `write` at once; that is +contention rather than an exception, so it fails as corruption of your own ordering rather than +loudly. Use a single writer thread, or your own lock around the writes. + +## Lifetimes + +Four `AutoCloseable`s, and leaking any of them costs something specific. Three of the rules here fail +silently. + +| Type | Leaking it costs | +|---|---| +| `Rabosh` | the directory lock, every mapping, and any background index build | +| `Snapshot` | disk: it holds back the versions compaction would otherwise drop, indefinitely | +| `DocumentCursor` | the segments it is reading, which cannot be reclaimed | +| `QueryCursor` | the same, plus its own snapshot if it took one | + +Behind the opt-in marker, `IndexReader` and `ColumnReader` are the same story: each pins every sidecar +it may consult. + +**On Windows a mapped file cannot be deleted at all**, so a leak there is not a slow drift in memory — +it is a compaction that can never reclaim its inputs. This is why `RaboshLifecycleTest` and +`ResourceLeakTest` assert by *deleting the directory* rather than by measuring anything. + +### A row is a view, not a copy — copy before `next()` + +This is the rule most likely to be discovered in production. + +```kotlin +// WRONG: every element ends up reading whatever the cursor last landed on. +val found = mutableListOf() +db.query(query.project(Projection.DOCUMENT)).use { rows -> + while (rows.next()) found += rows.row.document() +} + +// RIGHT: take a copy at the point you decide to keep it. +val found = mutableListOf() +db.query(query.project(Projection.DOCUMENT)).use { rows -> + while (rows.next()) found += rows.row.toJsonString() // or Variant.toByteArray() +} +``` + +Every `Variant` in a row reads straight out of a mapped segment. `QueryCursor.row` and +`DocumentCursor.key` / `.document` are valid **until the next `next()`**, and the underlying bytes are +valid only while the snapshot behind the read is open. That is the trade that makes reads cheap; the +copy is available wherever it is actually wanted, via `Row.toJsonString()` or `Variant.toByteArray()`. + +### `Query.where` projects keys only + +`Row.document()` throws `IllegalStateException` under `Projection.KEY` — the default — and also for a +row that was filled from shredded columns, because in neither case was a document opened. That is not +a defect: `documentsRead == 0` is reachable *because* of it, and it is the largest single win an index +buys. Ask for `Projection.DOCUMENT` when you are going to read one. + +## Durability + +The default is `Durability.SYNC`: every commit is `fsync`ed before the call returns, so an +acknowledged write survives power loss and not merely process death. A `WriteBatch` is one commit — +one append and one force however many documents it carries — which is what makes durable bulk writing +fast. + +`Durability.BUFFERED` writes to the operating system without forcing. Commits survive `kill -9`, +because a killed process does not discard the page cache, and are lost only if the machine stops. The +pattern it exists for is bulk load: write, call `sync()`, and only then report success to whoever +asked for the load. Do not report success before the `sync()`. + +## Taking a copy of a store + +**There is no `checkpoint` yet.** Until there is, the only defined way to copy a store is: + +1. Stop writing. Not "pause the ingest thread" — no `put`, `delete` or `write` may be in flight. +2. `db.flush()`, which returns when the memtable is on the platter and the manifest names it. +3. Copy the whole directory, including `CURRENT`, `MANIFEST-*`, every `.wal`, `.seg`, `.cat`, `.idx`, + `.pst` and `.col`. Not a subset: the manifest names the files it expects. +4. Resume writing. + +**A directory copied while a writer is running is not defined to be recoverable**, and a copy that +skips the log or the manifest is not a store. Neither failure is loud — the copy usually opens and is +usually missing something. + +`LOCK` may be copied or not; it holds nothing. + +## Version pinning + +The on-disk format is declared and stable, and a store written by an earlier release opens on every +later one — see [COMPATIBILITY.md](COMPATIBILITY.md). The **Kotlin API** is a different claim with a +different strength: [STABILITY.md](STABILITY.md) names a small stable core that moves only under a +deprecation cycle, and everything else may change in any release and is marked `@RaboshExperimental` +where the compiler can say so. + +Practically: pin an exact version, stay inside the stable core, and treat an opt-in error as the +library telling you that you have left it. Note that `rabosh-api` brings the five modules of the +storage chain with it, but **`rabosh-jsonpath` is not one of them** — ask for it by name. + +One behaviour worth knowing before you open the same directory with two different builds: +`DamagedIndexPolicy.REBUILD` and `DamagedSketchPolicy.REBUILD` do not distinguish a damaged sidecar +from one written by a *newer* build, so alternating builds rewrites sidecars downward. Documents are +never affected. [COMPATIBILITY.md](COMPATIBILITY.md#one-behaviour-worth-stating-plainly) has the +detail and the `REPORT` alternative. + +## What rabosh is not + +Stated here so that it is stated somewhere a reader looks before building on an assumption. + +- **Not a server.** It opens no sockets. It is a library in your process. +- **No replication, no clustering, no multi-process access.** One directory, one owner. +- **No encryption at rest.** Use filesystem-level or volume-level encryption; the engine writes plain + bytes and does not pretend otherwise. +- **No authentication or authorisation.** There is no principal to authenticate; access control is + the file permissions on the directory. +- **No `ORDER BY` on a value, no aggregation, no joins, no string expression language.** Results come + back in key order. These are declined rather than pending. + +Both of the engine's inputs — the JSON you write and the files on disk — are treated as hostile; +[SECURITY.md](SECURITY.md) sets out what that means and what counts as a vulnerability in a library +with no network surface. diff --git a/README.md b/README.md index c0fd504..59d290d 100644 --- a/README.md +++ b/README.md @@ -26,9 +26,11 @@ normal storage maintenance, and lets you add indexes to data that is already on > the wiring below has to be remembered. Plus interop with the portable Roaring bitmap format, so a set > of document positions can be handed to Lucene, Spark or pyroaring and read back, and index builds > that run in the background — cancellable, resumable, and usable while they run. Nothing here is -> production-ready, and the Kotlin API is major-version zero: any signature may change in any release. -> The **on-disk format is the exception** — it is declared and stable, and a store written by an -> earlier release opens on every later one. See [COMPATIBILITY.md](COMPATIBILITY.md). +> production-ready. The **on-disk format is declared and stable** — a store written by an earlier +> release opens on every later one ([COMPATIBILITY.md](COMPATIBILITY.md)) — and the **Kotlin API is +> tiered**: a small stable core moves only under a deprecation cycle, everything else may change in +> any release and says so with an opt-in marker ([STABILITY.md](STABILITY.md)). The runtime contract +> an embedding application has to obey is [INTEGRATION.md](INTEGRATION.md). ## Why @@ -87,7 +89,16 @@ pass per layer. Each layer stays public and unwrapped underneath, reached through `db.store`, `db.catalog` and `db.indexCatalog`. The sections below are about those layers, and every one of them can be wired up -by hand; the facade is what stops you having to. +by hand; the facade is what stops you having to. Those three accessors are marked +`@RaboshExperimental` — not a warning-off, just the honest statement that the facade's own surface is +what carries a stability promise and the layers beneath it do not. See +[STABILITY.md](STABILITY.md). + +**Before you build on it, read [INTEGRATION.md](INTEGRATION.md).** It is the runtime contract in one +page — the JDK floor, why no `--enable-native-access` flag is needed, one process and one writing +thread, what leaking each `AutoCloseable` costs, and the copy-before-`next()` rule that decides +whether a row you kept still means what you think. Three of those fail silently, which is the whole +reason the file exists rather than living in KDoc on classes you may never open. That snippet is also a runnable program. `./gradlew :rabosh-samples:runThreeSteps` writes a few thousand events of a shape nobody declared, prints the model derived from them and the indexes it @@ -643,9 +654,9 @@ from a shared runner are not a regression gate. The aggregated API documentation for the seven published modules is at **[aoreshkov.github.io/rabosh](https://aoreshkov.github.io/rabosh/)**, generated by Dokka from `main` -on every push. It tracks the current API rather than the last release, deliberately: the Kotlin API -is major-version zero and free to move, so a versioned copy would document something you are being -told not to rely on. Each release also ships its own module documentation as the `javadoc` +on every push. It tracks the current API rather than the last release, deliberately: outside the +stable core the Kotlin API is free to move, so a versioned copy would document something you are +being told not to rely on. Each release also ships its own module documentation as the `javadoc` classifier artefact, for anyone who needs to pin one. Published to Maven Central under the group `app.oreshkov`; all code lives under the diff --git a/STABILITY.md b/STABILITY.md new file mode 100644 index 0000000..cb33dba --- /dev/null +++ b/STABILITY.md @@ -0,0 +1,134 @@ +# Stability + +This document is rabosh's declared public API for its **Kotlin surface**. Its sibling +[COMPATIBILITY.md](COMPATIBILITY.md) does the same job for the on-disk format, and the two are +deliberately separate because they move at different speeds and rest on different evidence. + +**This is not a promise of 1.0, and it is not a step towards one.** It is a smaller and truer claim: +*these* declarations move under a deprecation cycle, and the rest may move in any release. "Major +version zero, any signature may change" was honest and unactionable — a consumer could not tell +whether `Key.of` was as volatile as `IndexCatalog.readColumn`, so the only rational responses were to +wrap all of the API or none of it. Two tiers cost nothing and say what the evidence supports. + +| | Guarantee | +|---|---| +| **Stable core** | Removed or changed incompatibly only after a release deprecating it. Listed below. | +| **Everything else** | May change or be removed in any release. Marked `@RaboshExperimental` where the compiler can enforce it. | + +## The stable core + +Small on purpose. It is the surface the README's examples call, the surface the two samples call, and +it has not moved in two releases. + +**`rabosh-api`** — `Rabosh` (`open`, `close`, `put`, `get`, `delete`, `write`, `scan`, `snapshot`, +`query`, `keys`, `explain`, `createIndex`, `createIndexInBackground`, `buildIndexesInBackground`, +`dropIndex`, `indexes`, `schema`, `indexCandidates`, `attach`, `flush`, `sync`, `rotate`, `compact`, +`stats`, `directory`, `options`) and `RaboshOptions`. + +**`rabosh-core`** — `Key`, `WriteBatch`, `Durability`, `Snapshot`, `DocumentCursor`, `StoreOptions`, +`StoreStats`, `LogRecoveryMode`, `SegmentObserver`, `SegmentObservation`, `SegmentSummary`, and the +whole `StoreException` hierarchy. + +**`rabosh-query`** — `Query`, `Predicate` and its cases, the predicate DSL (`path`, `and`, `or`, +`not`, `eq`, `anyOf`, `exists`, `isNull`, `elemMatch`, the comparison operators), `PathRef`, +`Comparison`, `QueryValue`, `Projection`, `Row`, `QueryCursor`, `QueryStats`, `Explain`, +`ExplainSource`, `IndexUse`. + +**`rabosh-index`** — `IndexDefinition`, `IndexHandle`, `IndexBuild`, `IndexBuildProgress`, +`IndexBuildState`, `IndexCoverage`, `IndexOptions`, `DamagedIndexPolicy`, `CompositeSegmentObserver`, +and the whole `IndexException` hierarchy. + +**`rabosh-catalog`** — `CatalogPath`, `CatalogStep` and the node walk, `InferredSchema`, +`InferredField` (except its `sketch`), `CatalogCoverage`, `IndexCandidate`, `IndexCandidateOptions`, +`IndexKind`, `ValueBounds`, `NumericRange`, `TextRange`, `CatalogOptions`, `DamagedSketchPolicy`, and +the whole `CatalogException` hierarchy. + +**`rabosh-variant`** — `Variant` and its readers, `VariantNode`, `VariantPath`, `VariantPathStep`, +`VariantKind`, `VariantBasicType`, `VariantPrimitiveType`, `VariantBuilder`, `VariantMetadata`, +`DuplicateFieldPolicy`, `toJsonString` / `toJsonSummaryString`, and the whole `VariantException` +hierarchy. + +**`rabosh-jsonpath`** — `JsonPathQuery`. + +### Two entries that are in the list for a reason worth knowing + +**`SegmentObserver` is stable.** It is the seam every layer above `rabosh-core` is built on, and +`RaboshOptions` takes one, so it is part of the supported way to compose the engine rather than an +internal that leaked. It could not have been marked experimental even if that had been wanted: +`RaboshOptions`' own constructor names it, and opt-in propagates through signatures, so marking the +interface would have made constructing `RaboshOptions` require opt-in — the stable core's own options +object, inside the experimental tier. + +**`InferredField` is stable except for one property.** Everything on it is a named reading against +the document count; `sketch` is the serialised estimator those readings are derived from, and its +registers, hash and sparse limit belong to the sidecar format. + +## `@RaboshExperimental` + +Everything not listed above may change or be removed in any release, with no deprecation cycle. Where +that can be stated to the compiler, it is: + +```kotlin +@OptIn(RaboshExperimental::class) +fun dumpPostings(db: Rabosh) { … } +``` + +Marked today: `Bitmap`, `BitmapView`, `BitmapCursor`, `ReadableBitmap`, `RoaringPortable`, +`ColumnReader`, `ColumnQuery`, `ColumnScan`, `ColumnMatch`, `ColumnPredicate`, `IndexReader`, +`IndexQuery`, `KeyCursor`, `CompositeTerm`, `TermExtractor`, `ElementExtractor`, `HyperLogLog`, +`SegmentSketch`, `ValueSignature`, `ValueBoundsBuilder`; the `Rabosh.store`, `Rabosh.catalog` and +`Rabosh.indexCatalog` accessors; `DocumentStore.open`; the `SchemaCatalog`, `IndexCatalog` and +`QueryEngine` constructors; `IndexCatalog.read` and `IndexCatalog.readColumn`; +`SchemaCatalog.sketchOf`; and `InferredField.sketch`. + +### What is marked is the way *in*, not every member + +A `ColumnReader` can only be reached through `Rabosh.indexCatalog` or `IndexCatalog.readColumn`, and +both of those are marked — so once you hold one, its methods carry no further annotation. The same +goes for a handful of types that carry no marker of their own and are reachable only through one: +`PathSketch`, `IndexTerm` and the rest of the sidecar vocabulary. They are outside the stable core by +this list, which is the claim; the annotation is the enforcement, applied at the entrances. + +Marking every member instead would take some hundred and fifty annotations and, worse, would force +every stable signature naming an experimental *type* to be marked too — a cascade that ends with the +stable core inside the experimental tier. The `SegmentObserver` note above is that cascade caught at +one step. + +## The deprecation cycle + +A stable-core declaration is never removed in the release that stops recommending it. It first ships +with `@Deprecated(DeprecationLevel.WARNING)` carrying a `ReplaceWith` wherever a mechanical +replacement exists; a later release moves it to `DeprecationLevel.HIDDEN`, which keeps the symbol in +the bytecode so already-compiled callers keep linking; only after that may it go. A declaration in the +experimental tier gets none of this, which is the whole difference between the tiers. + +Moving a declaration *between* tiers is a change like any other: into the stable core is additive and +may happen in any release; out of it goes through the cycle above. + +## How this is held to + +Two mechanisms, and it is worth being precise about which does what, because the obvious one does +less than it looks. + +**`checkKotlinAbi` holds the signatures, not the tiers.** The committed dumps at `/api/*.api` +fail the build on any binary-incompatible change to any published declaration, stable or not. What +they do **not** carry is the markers: the JVM dump format writes signature lines only and never +annotations, and the synthetic method Kotlin emits for an annotated property is filtered out as +synthetic. A declaration changing tier is invisible to it. + +**`rabosh-samples` is what holds the tiers.** It depends on `:rabosh-api` and nothing else, it +compiles with `allWarningsAsErrors`, it is part of `./gradlew build`, and — unlike every other module +in the repository — it deliberately does **not** opt in to `@RaboshExperimental`. It is therefore a +real consumer compiling against the stable core with no opt-in. A stable declaration that silently +acquires the marker fails there, and so does a sample that reaches past the facade. That asymmetry is +load-bearing and should not be tidied away by giving every module the same build configuration. + +Verified by breaking it, which is this repository's standing rule for a check nobody has watched +fail: adding `db.store.flush()` to a sample fails `./gradlew build` with the opt-in error naming the +marker, and commenting out the opt-in in `rabosh.kotlin-library` fails the published modules. + +## Reporting + +A stable-core declaration that changed without a deprecation cycle is a bug. Please +[open an issue](https://github.com/aoreshkov/rabosh/issues) naming the declaration and the two +releases. diff --git a/build-logic/src/main/kotlin/rabosh.kotlin-library.gradle.kts b/build-logic/src/main/kotlin/rabosh.kotlin-library.gradle.kts index 9164bc0..f8f1f3b 100644 --- a/build-logic/src/main/kotlin/rabosh.kotlin-library.gradle.kts +++ b/build-logic/src/main/kotlin/rabosh.kotlin-library.gradle.kts @@ -16,6 +16,27 @@ plugins { kotlin { explicitApi() + /* + * The engine opts in to its own experimental tier, once, here. + * + * `@RaboshExperimental` is a statement to *consumers* about which declarations may move; inside + * the library every layer reaches through the marked entrances by construction — `Rabosh.open` + * calls `DocumentStore.open`, the planner takes an `IndexCatalog` — so annotating each of those + * call sites would be several hundred `@OptIn`s carrying no information. This is what + * kotlinx.coroutines and the standard library do with their own markers. + * + * **`rabosh-samples` deliberately does not apply this plugin and does not get this line**, and + * that is what makes the tier claim checkable rather than asserted. It depends on `:rabosh-api` + * and nothing else, it compiles with `allWarningsAsErrors`, and it is part of `build` — so it is + * a real consumer compiling against the stable core with no opt-in. A stable declaration that + * silently changes tier fails there, and so does a sample that reaches past the facade. The ABI + * dumps cannot do this job: the JVM dump format writes signatures only and never annotations, so + * a tier change is invisible to `checkKotlinAbi`. + */ + compilerOptions { + optIn.add("app.oreshkov.rabosh.RaboshExperimental") + } + // Kotlin's built-in ABI validation (2.4+), used in place of the standalone // binary-compatibility-validator plugin, whose ASM cannot read Java 25 bytecode. // @@ -34,6 +55,35 @@ java { withSourcesJar() } +/* + * The module name this jar answers to on the module path. + * + * There is no `module-info.java` anywhere here and this is not a step towards one. Without the + * attribute an automatic module is named after the *file*, which is derived from an artefact id and + * is therefore unstable by construction — a jar renamed, shaded or republished under another + * coordinate silently becomes a different module, and every `requires` naming it stops resolving. A + * `jlink`/`jpackage` build is the normal shape for an embedded store, so this is the difference + * between "packageable" and "not", not a nicety. + * + * Derived from the project name rather than listed, for the reason `PublishedModules` gives: a + * hand-maintained list would disagree with `settings.gradle.kts`, and the symptom would be one jar + * with the wrong name. Each module's own root package is what comes out — `rabosh-core` -> + * `app.oreshkov.rabosh.core` — which is the JPMS convention and, usefully, already unique per jar, + * so no two of the seven can claim a package and make the set unresolvable. + * + * `runThreeStepsOnModulePath` in `rabosh-samples` is what stops this being a claim: it asks the JVM + * for `app.oreshkov.rabosh.api` by name, which fails outright if this attribute is missing. + * + * An attribute is reversible and a descriptor is not — that asymmetry is the whole decision. + */ +val automaticModuleName = "app.oreshkov.rabosh.${project.name.removePrefix("rabosh-")}" + +tasks.named("jar") { + manifest { + attributes("Automatic-Module-Name" to automaticModuleName) + } +} + /** * Dokka's HTML, packaged under the `javadoc` classifier. * diff --git a/rabosh-api/src/main/kotlin/app/oreshkov/rabosh/api/Rabosh.kt b/rabosh-api/src/main/kotlin/app/oreshkov/rabosh/api/Rabosh.kt index 682134d..1336bde 100644 --- a/rabosh-api/src/main/kotlin/app/oreshkov/rabosh/api/Rabosh.kt +++ b/rabosh-api/src/main/kotlin/app/oreshkov/rabosh/api/Rabosh.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.api +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.IndexCandidate import app.oreshkov.rabosh.catalog.IndexCandidateOptions import app.oreshkov.rabosh.catalog.InferredSchema @@ -78,7 +79,12 @@ public class Rabosh private constructor( * The escape hatch, and it is a supported one rather than an admission: everything the facade * does not cover — `scanSegments`, `liveSegmentNumbers`, `backfill` with an observer of your own * — is here, at full width, unchanged. Do not close it; [close] does, in order. + * + * Outside the stable core, and this property is the entrance that says so: the store's surface + * is the engine's internals at full width, and pinning it would pin the engine. See + * `STABILITY.md`. */ + @RaboshExperimental public val store: DocumentStore, /** * The schema catalog, or `null` when [RaboshOptions.schema] is `false`. @@ -86,7 +92,11 @@ public class Rabosh private constructor( * Attached and maintained by this object. `null` rather than an empty catalog, because a model * that was never collected and a model of nothing are different answers and the type should say * which one this is. + * + * Outside the stable core. [schema] and [indexCandidates] are the stable way to the model; this + * is the way to the sketches behind it, which are a format rather than an answer. */ + @RaboshExperimental public val catalog: SchemaCatalog?, /** * The index catalog, or `null` when [RaboshOptions.indexes] is `false`. @@ -94,7 +104,11 @@ public class Rabosh private constructor( * Attached and maintained by this object, and **closed by [close]** — which is the wiring mistake * this class exists to make impossible. Reach through it for `read`, `readColumn` and the rest of * the index surface; do not close it yourself. + * + * Outside the stable core. [createIndex], [dropIndex], [indexes] and [query] are the stable + * index surface; everything below them is sidecar bytes and reader lifetimes. */ + @RaboshExperimental public val indexCatalog: IndexCatalog?, private val observer: SegmentObserver?, ) : AutoCloseable { diff --git a/rabosh-bench/build.gradle.kts b/rabosh-bench/build.gradle.kts index 6164922..ead2ee8 100644 --- a/rabosh-bench/build.gradle.kts +++ b/rabosh-bench/build.gradle.kts @@ -13,6 +13,18 @@ plugins { description = "JMH benchmark suites: ingest throughput, point-get latency, scan throughput, amplification." +/* + * A benchmark measures the engine, not the facade: `BitmapBenchmark` times container transitions and + * `AmplificationMain` opens a `DocumentStore` directly, both of which are outside the stable core on + * purpose. Same line as `rabosh.kotlin-library` gives the published modules, and deliberately not + * given to `rabosh-samples`, which is the one module that has to compile as a consumer does. + */ +kotlin { + compilerOptions { + optIn.add("app.oreshkov.rabosh.RaboshExperimental") + } +} + dependencies { implementation(project(":rabosh-api")) implementation(libs.kotlinx.benchmark.runtime) diff --git a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/HyperLogLog.kt b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/HyperLogLog.kt index fc89cd3..93a76d2 100644 --- a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/HyperLogLog.kt +++ b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/HyperLogLog.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.catalog +import app.oreshkov.rabosh.RaboshExperimental import java.util.Arrays /** @@ -38,6 +39,7 @@ import java.util.Arrays * path, and a copy per value would dwarf the sketch. Merging into a *new* sketch is [mergedWith]; * [merge] mutates. Not thread-safe — one observation belongs to one segment writer. */ +@RaboshExperimental public class HyperLogLog private constructor( private var hashes: LongArray?, private var hashCount: Int, diff --git a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/InferredSchema.kt b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/InferredSchema.kt index 3c8e2ee..77a6cf4 100644 --- a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/InferredSchema.kt +++ b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/InferredSchema.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.catalog +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.variant.VariantKind /** @@ -36,7 +37,15 @@ public class CatalogCoverage internal constructor( public class InferredField internal constructor( /** The path, with array indices collapsed. See [CatalogPath]. */ public val path: CatalogPath, - /** The raw statistics this reading is derived from. */ + /** + * The raw statistics this reading is derived from. + * + * Outside the stable core, and the only member of this class that is: the named readings below + * are a contract, and a `PathSketch` is a serialised estimator whose registers, hash and sparse + * limit belong to `SketchFormat`. Everything a caller needs about a path is already a property + * here; reaching for the sketch means reaching for the format. + */ + @RaboshExperimental public val sketch: PathSketch, private val documentCount: Long, ) { diff --git a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SchemaCatalog.kt b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SchemaCatalog.kt index edeff3e..4f382b0 100644 --- a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SchemaCatalog.kt +++ b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SchemaCatalog.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.catalog +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.core.DocumentStore import app.oreshkov.rabosh.core.Key import app.oreshkov.rabosh.core.SegmentObservation @@ -46,7 +47,7 @@ import kotlin.concurrent.withLock * [beginSegment], [SegmentObservation.complete] and [retain] while a reader may be inside * [inferSchema]; the accumulation of one segment happens on one thread and is not shared. */ -public class SchemaCatalog( +public class SchemaCatalog @RaboshExperimental constructor( /** The store directory sidecars live in. The same directory the store was opened on. */ public val directory: Path, /** Tuning. See [CatalogOptions]. */ @@ -175,7 +176,13 @@ public class SchemaCatalog( options: IndexCandidateOptions = IndexCandidateOptions.DEFAULT, ): List = rankIndexCandidates(inferSchema(), options) - /** The sketch of one segment, or `null` if it is not covered. For tests and for diagnostics. */ + /** + * The sketch of one segment, or `null` if it is not covered. For tests and for diagnostics. + * + * Outside the stable core: a `SegmentSketch` is the `.cat` sidecar's contents, so its shape is + * the format's. [inferSchema] is the stable reading of the same data. + */ + @RaboshExperimental public fun sketchOf(segmentNumber: Long): SegmentSketch? = lock.withLock { sketches[segmentNumber] } override fun toString(): String = diff --git a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SegmentSketch.kt b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SegmentSketch.kt index a62f499..943860c 100644 --- a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SegmentSketch.kt +++ b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/SegmentSketch.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.catalog +import app.oreshkov.rabosh.RaboshExperimental import java.util.TreeMap /** @@ -15,6 +16,7 @@ import java.util.TreeMap * [droppedObservations]. Machine-generated field names — an object keyed by user id, a log line * carrying a request id in the key — would otherwise make the path space a copy of the data. */ +@RaboshExperimental public class SegmentSketch internal constructor( /** Documents observed. Tombstones are not documents and are not counted. */ public val documentCount: Long, diff --git a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueBounds.kt b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueBounds.kt index acc7d41..bf3a7b5 100644 --- a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueBounds.kt +++ b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueBounds.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.catalog +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.variant.Variant import app.oreshkov.rabosh.variant.VariantKind import java.math.BigDecimal @@ -217,6 +218,7 @@ public class ValueBounds internal constructor( * @param textBoundBytes how many bytes of a string the bound may keep before truncating. Truncation * widens, so a smaller limit costs precision and never correctness. */ +@RaboshExperimental public class ValueBoundsBuilder(private val textBoundBytes: Int) { private var numeric: NumericRange? = null private var text: TextRange? = null diff --git a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueSignature.kt b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueSignature.kt index c0a02e5..f8b1617 100644 --- a/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueSignature.kt +++ b/rabosh-catalog/src/main/kotlin/app/oreshkov/rabosh/catalog/ValueSignature.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.catalog +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.variant.Variant import app.oreshkov.rabosh.variant.VariantKind import java.math.BigDecimal @@ -29,6 +30,7 @@ import java.math.BigDecimal * **The tags are permanent.** Every HyperLogLog register ever written is a function of them, and so * is every term in every posting file. Add, never renumber. */ +@RaboshExperimental public object ValueSignature { /** `true` or `false`. One payload byte, `1` or `0`. */ public const val BOOLEAN: Int = 0 diff --git a/rabosh-core/src/main/kotlin/app/oreshkov/rabosh/core/DocumentStore.kt b/rabosh-core/src/main/kotlin/app/oreshkov/rabosh/core/DocumentStore.kt index cb2836e..02eadf6 100644 --- a/rabosh-core/src/main/kotlin/app/oreshkov/rabosh/core/DocumentStore.kt +++ b/rabosh-core/src/main/kotlin/app/oreshkov/rabosh/core/DocumentStore.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.core +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.variant.Variant import app.oreshkov.rabosh.variant.VariantMetadata import java.io.IOException @@ -715,7 +716,13 @@ public class DocumentStore private constructor( * @throws UnsupportedFormatException if the files are from a newer format version. * @throws NoSuchFileException if the directory is absent and * [StoreOptions.createIfMissing] is `false`. + * + * **Outside the stable core**, and this is the only entrance to it, which is why the marker + * is here rather than on every member of the class. `Rabosh.open` is the stable way to open + * a database; assembling the store, the schema catalog and the index catalog by hand is + * supported and is not a signature anything is promised about. See `STABILITY.md`. */ + @RaboshExperimental public fun open(directory: Path, options: StoreOptions = StoreOptions.DEFAULT): DocumentStore { prepareDirectory(directory, options) val lock = DirectoryLock.acquire(directory) diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/Bitmap.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/Bitmap.kt index 5935411..8ae1e74 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/Bitmap.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/Bitmap.kt @@ -1,5 +1,7 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental + /** * A mutable set of document ordinals, held as one block per 65 536 ordinals. * @@ -27,6 +29,7 @@ package app.oreshkov.rabosh.index * `hashCode` walks the ordinals, so it is `O(cardinality)`. Sound rather than fast, which is the right * way round for a structure nobody puts in a hash map by design. */ +@RaboshExperimental public class Bitmap private constructor( private var keys: IntArray, private var blocks: Array, diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapCursor.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapCursor.kt index 9cfc8c7..6f34b21 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapCursor.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapCursor.kt @@ -1,5 +1,7 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental + /** * A walk over a bitmap's ordinals, ascending. * @@ -15,6 +17,7 @@ package app.oreshkov.rabosh.index * start costs the sum of their cardinalities; walking the sparser one and jumping the denser costs the * sparser, which is the difference between reading a sidecar and reading past it. */ +@RaboshExperimental public class BitmapCursor internal constructor(private val source: ContainerSource) { private var blockIndex = -1 diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapView.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapView.kt index 37b792c..29416fb 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapView.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/BitmapView.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import java.lang.foreign.MemorySegment /** @@ -32,6 +33,7 @@ import java.lang.foreign.MemorySegment * phase 7's sidecar does for a bitmap what `SegmentBytes.verifyBlock` does for a segment's data block. * [verify] is the deep pass, for tests and for anything that wants to audit a file it did not write. */ +@RaboshExperimental public class BitmapView private constructor( private val bytes: IndexBytes, private val blockCount: Int, diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnPredicate.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnPredicate.kt index 4a71985..2a0b477 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnPredicate.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnPredicate.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.variant.Variant import app.oreshkov.rabosh.variant.VariantKind import java.math.BigDecimal @@ -18,6 +19,7 @@ import java.math.BigDecimal * `.claude/rules/index-and-query.md` requires the recheck to run the same logic that built the * index, so [matches] is what both the column scan and the fallback document scan evaluate. */ +@RaboshExperimental public class ColumnPredicate private constructor( internal val kind: Kind, internal val numericMin: BigDecimal?, diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnQuery.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnQuery.kt index 1179ae9..862be09 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnQuery.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnQuery.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.core.DocumentStore import app.oreshkov.rabosh.core.Key @@ -11,6 +12,7 @@ import app.oreshkov.rabosh.core.Key * suite asserts it is zero **in the same test** as the differential equality against a full scan. On * its own it would pass trivially for a query that returned nothing. */ +@RaboshExperimental public class ColumnScan internal constructor( /** The matching keys, deduplicated and sorted. */ public val keys: List, @@ -59,6 +61,7 @@ public class ColumnScan internal constructor( * on a compacted, write-once, fully covered store, which is the same shape `SchemaInferenceTest` * requires before it asserts the catalog's counts are exact. */ +@RaboshExperimental public object ColumnQuery { /** * Keys whose visible document satisfies [predicate] at the reader's path. diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnReader.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnReader.kt index 32e4437..04af0bd 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnReader.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ColumnReader.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.core.Key import app.oreshkov.rabosh.core.Snapshot @@ -19,6 +20,7 @@ import app.oreshkov.rabosh.variant.Variant * Close it. On Windows a mapped file cannot be deleted, so a reader left open blocks reclamation of * everything it touched. */ +@RaboshExperimental public class ColumnReader internal constructor( private val handle: IndexHandle, /** The snapshot this reader answers at. */ @@ -310,6 +312,7 @@ public class ColumnReader internal constructor( * answers: [matches] is a claim about the version the column recorded, which a newer segment or a * memtable may have replaced. */ +@RaboshExperimental public class ColumnMatch internal constructor( /** Ordinals whose stored value satisfies the predicate. Exact, and decided without a document. */ public val matches: ReadableBitmap, diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/CompositeTerm.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/CompositeTerm.kt index 22eec6a..01dc853 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/CompositeTerm.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/CompositeTerm.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.catalog.CatalogStep import app.oreshkov.rabosh.variant.Variant @@ -59,6 +60,7 @@ import app.oreshkov.rabosh.variant.Variant * key, and a variable-width prefix would give one tuple two spellings the moment a length crossed * 128 — which is the canonicality rule `IndexBytes.varint` enforces, arriving from the other side. */ +@RaboshExperimental public object CompositeTerm { /** Bounded so the two-byte field index in a term cannot overflow, with room to spare. */ diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ElementExtractor.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ElementExtractor.kt index c3668cb..070f3d6 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ElementExtractor.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ElementExtractor.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.catalog.CatalogStep import app.oreshkov.rabosh.variant.Variant @@ -24,6 +25,7 @@ import app.oreshkov.rabosh.variant.VariantBasicType * calls this class, and there is exactly one definition of "which containers does `$.items[*]` stand * for in this document". */ +@RaboshExperimental public class ElementExtractor( /** The container paths, in the order elements are reported against. */ private val paths: List, diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexCatalog.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexCatalog.kt index 0f8f0ba..082a580 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexCatalog.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexCatalog.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.catalog.IndexKind import app.oreshkov.rabosh.core.DocumentStore @@ -61,7 +62,7 @@ import kotlin.concurrent.withLock * [createIndexInBackground] — which drives exactly the same three callbacks and is subject to exactly * the same rules. */ -public class IndexCatalog( +public class IndexCatalog @RaboshExperimental constructor( /** The store directory sidecars live in. The same directory the store was opened on. */ public val directory: Path, /** Tuning. See [IndexOptions]. */ @@ -485,7 +486,12 @@ public class IndexCatalog( * * Close it. On Windows a mapped file cannot be deleted, so a reader left open blocks reclamation * of every segment it touched — which `IndexLifecycleTest` asserts in both directions. + * + * Outside the stable core: this hands back an ordinal-space reader over sidecar bytes, and both + * the ordinals and the bytes are the format's rather than a contract's. `Rabosh.query` is the + * stable way to ask an index a question. See `STABILITY.md`. */ + @RaboshExperimental public fun read(store: DocumentStore, handle: IndexHandle, snapshot: Snapshot): IndexReader { // A composite index's sidecar *is* a posting file — same dictionary, same postings, same // presence bitmap — so it is read by this reader and not by a third one. What differs is only @@ -501,7 +507,10 @@ public class IndexCatalog( * Opens a reader over the shredded column [handle] at [snapshot], pinning every sidecar it needs. * * Close it, for the reason [read]'s result must be closed. + * + * Outside the stable core, for the reason [read] is. */ + @RaboshExperimental public fun readColumn(store: DocumentStore, handle: IndexHandle, snapshot: Snapshot): ColumnReader { require(handle.kind == IndexKind.SHREDDED_COLUMN) { "index #${handle.id} is a ${handle.kind}; open it with read" diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexQuery.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexQuery.kt index 0268a46..47b46eb 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexQuery.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexQuery.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.core.DocumentStore import app.oreshkov.rabosh.core.Key @@ -32,6 +33,7 @@ import app.oreshkov.rabosh.variant.Variant * that decided what to index. A second, differently-shaped evaluation would be a second definition * of what a path means, and the two would eventually disagree about an array or a nested null. */ +@RaboshExperimental public object IndexQuery { /** Keys whose visible document carries [term] at the reader's path. */ public fun keysEqualTo(store: DocumentStore, reader: IndexReader, term: IndexTerm): List = diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexReader.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexReader.kt index ec10129..7e60f84 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexReader.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/IndexReader.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.core.Key import app.oreshkov.rabosh.core.Snapshot @@ -22,6 +23,7 @@ import app.oreshkov.rabosh.core.Snapshot * The pins are released by [close], which must be called — on Windows a mapped file cannot be deleted * at all, so a reader left open blocks reclamation of everything it touched. */ +@RaboshExperimental public class IndexReader internal constructor( private val handle: IndexHandle, /** The snapshot this reader answers at. */ @@ -244,6 +246,7 @@ internal class SegmentHits(val segment: SegmentIndex, val ordinals: ReadableBitm * Valid only while the [IndexReader] that produced it is open, because the ordinals are read straight * off that reader's mappings. */ +@RaboshExperimental public class KeyCursor internal constructor(private val hits: List) { private var segmentIndex = -1 private var cursor: BitmapCursor? = null diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ReadableBitmap.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ReadableBitmap.kt index 5e72754..8c8bbe2 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ReadableBitmap.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/ReadableBitmap.kt @@ -1,5 +1,7 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental + /** * A set of document ordinals that can be read, whether it is being built on the heap or read straight * out of a mapped file. @@ -16,6 +18,7 @@ package app.oreshkov.rabosh.index * * Sizes and ranks are `Int`. See [BitmapFormat.MAX_ORDINAL] for the one ordinal that costs. */ +@RaboshExperimental public sealed interface ReadableBitmap { /** How many ordinals are present. */ diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/RoaringPortable.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/RoaringPortable.kt index d7e7699..3d6a5c0 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/RoaringPortable.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/RoaringPortable.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import java.lang.foreign.MemorySegment /** @@ -31,6 +32,7 @@ import java.lang.foreign.MemorySegment * [decode] reads that perfectly well while [encode] will never write it. Re-exporting such a stream * therefore shrinks it, and does not return the bytes that came in. */ +@RaboshExperimental public object RoaringPortable { /** diff --git a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/TermExtractor.kt b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/TermExtractor.kt index 2765fd3..92683c8 100644 --- a/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/TermExtractor.kt +++ b/rabosh-index/src/main/kotlin/app/oreshkov/rabosh/index/TermExtractor.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.index +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.CatalogPath import app.oreshkov.rabosh.catalog.CatalogStep import app.oreshkov.rabosh.variant.Variant @@ -43,6 +44,7 @@ import app.oreshkov.rabosh.variant.VariantBasicType * caller of it. `rabosh-query` builds exactly one of these over every path a predicate mentions, so * a whole predicate costs one narrowing walk per document rather than one walk per leaf. */ +@RaboshExperimental public class TermExtractor( /** The indexed paths, in the order terms are reported against. */ private val paths: List, diff --git a/rabosh-query/src/main/kotlin/app/oreshkov/rabosh/query/QueryEngine.kt b/rabosh-query/src/main/kotlin/app/oreshkov/rabosh/query/QueryEngine.kt index 400b35e..49778cd 100644 --- a/rabosh-query/src/main/kotlin/app/oreshkov/rabosh/query/QueryEngine.kt +++ b/rabosh-query/src/main/kotlin/app/oreshkov/rabosh/query/QueryEngine.kt @@ -1,5 +1,6 @@ package app.oreshkov.rabosh.query +import app.oreshkov.rabosh.RaboshExperimental import app.oreshkov.rabosh.catalog.InferredSchema import app.oreshkov.rabosh.core.DocumentStore import app.oreshkov.rabosh.core.Key @@ -27,7 +28,7 @@ import app.oreshkov.rabosh.index.IndexCatalog * The engine holds nothing and owns nothing. It is safe to make one per query or one per store, and * to use one from several threads at once — the state of a query lives in its [QueryCursor]. */ -public class QueryEngine( +public class QueryEngine @RaboshExperimental constructor( private val store: DocumentStore, private val indexes: IndexCatalog, /** diff --git a/rabosh-samples/build.gradle.kts b/rabosh-samples/build.gradle.kts index 5f285ff..d75a9c4 100644 --- a/rabosh-samples/build.gradle.kts +++ b/rabosh-samples/build.gradle.kts @@ -29,10 +29,16 @@ dependencies { * and `distTar` to `assemble`, so `./gradlew build` would start producing distribution archives of a * demo nobody installs. Plain `JavaExec` costs two lines more per sample and neither. * - * `--enable-native-access=ALL-UNNAMED` is not optional and not copied from habit: - * `rabosh.kotlin-base` adds it to `Test` tasks only, and the engine maps every segment through - * `FileChannel.map(mode, offset, size, Arena)`, which is a restricted method on JDK 25. Without it - * the first line of a sample's output is a JVM warning about the library it is demonstrating. + * `--enable-native-access=ALL-UNNAMED` on the two tasks below is **future-proofing, not a + * requirement**, and the comment that used to stand here said otherwise. It claimed that + * `FileChannel.map(mode, offset, size, Arena)` is a restricted method and that without the flag a + * sample's first line of output is a JVM warning. Neither is true: that overload carries no + * `@Restricted` and declares no `IllegalCallerException` in JDK 25, and the sample runs silently + * under `--illegal-native-access=deny` with no grant at all. The engine calls no restricted method. + * + * The flag is kept because it costs nothing and is right the day one arrives. What is *not* kept is + * the reasoning — `runThreeStepsOnModulePath` below is where the claim now lives, checked rather + * than asserted. */ tasks.register("runThreeSteps") { group = "sample" @@ -49,3 +55,66 @@ tasks.register("runIndexLater") { classpath = sourceSets["main"].runtimeClasspath jvmArgs("--enable-native-access=ALL-UNNAMED") } + +/* + * The same sample again, with the library on the **module path** — which is the only thing that + * checks `Automatic-Module-Name`. + * + * A packaging claim nothing packages is the same defect as documentation nothing executes, and this + * is the cheapest place to hold the claim: a desktop app built with `jlink`/`jpackage` is the normal + * shape for an embedded store, and that is a module-path build. Two of the flags below are the + * assertion and neither can pass vacuously. + * + * `--add-modules app.oreshkov.rabosh.api` names the module rather than the file. Without the + * manifest attribute the jar resolves as an automatic module called `rabosh.api`, derived from its + * filename, and the JVM fails at startup with `module not found`. Delete the attribute from + * `rabosh.kotlin-library` and this task stops working — which is the only reason to have it. + * + * `--illegal-native-access=deny` with **no** `--enable-native-access` beside it is the second + * assertion, and it is the one this task is uniquely able to make: *the engine calls no restricted + * method, so it needs no native-access grant.* That is checkable here and nowhere else. The other + * two samples pass `--enable-native-access=ALL-UNNAMED`, which covers the classpath and would hide + * the answer; on the module path the engine's code is in a **named** module, which `ALL-UNNAMED` + * does not reach — so if `rabosh-core` ever acquired a `MemorySegment.reinterpret`, a `Linker` + * downcall or a `System.loadLibrary`, this task would fail with `IllegalCallerException` while every + * other sample carried on passing. Granting native access here would make the check vacuous, which + * is why the flag is deliberately absent rather than merely unset. + * + * `FileChannel.map(mode, offset, size, Arena)` is *not* restricted — it carries no `@Restricted` and + * declares no `IllegalCallerException` in JDK 25 — so mapping a segment costs no grant. That is the + * fact `INTEGRATION.md` states, and this is what holds it. + * + * The sample itself stays on the classpath, in the unnamed module. That is the realistic shape for a + * consumer that has not modularised, and it keeps this an `Automatic-Module-Name` check rather than + * the beginnings of a `module-info.java` — the attribute is reversible and a descriptor is not. + * + * `kotlin.stdlib` is named alongside it because it has to be. Resolving one automatic module + * resolves every *other* automatic module on the path — which is what brings the remaining six + * rabosh jars in without listing them — but `kotlin-stdlib` ships a real `module-info`, so it is an + * explicit module and nothing pulls it in implicitly. The sample's own classes are in the unnamed + * module and call into it directly, so without this the first thing that happens is a + * `NoClassDefFoundError` on `kotlin.jvm.internal.Intrinsics`. + */ +val modulePath: FileCollection = configurations["runtimeClasspath"] + +tasks.register("runThreeStepsOnModulePath") { + group = "sample" + description = "The three steps again, with the library resolved by module name on the module path." + mainClass = "app.oreshkov.rabosh.samples.ThreeStepsMain" + + // Only the sample's own classes. Everything it depends on is reached by module name below. + classpath = files(sourceSets["main"].output) + dependsOn(modulePath) + + jvmArgs( + "--add-modules", "app.oreshkov.rabosh.api,kotlin.stdlib", + "--illegal-native-access=deny", + ) + // Through a provider rather than `jvmArgs(...)` directly: resolving the configuration to build a + // string would do it while the build is being configured, which the configuration cache stores + // and would then serve stale. The local is not a stylistic choice — a lambda that read + // `modulePath` directly would capture the *script object*, which the configuration cache + // refuses to serialise. + val jars = modulePath + jvmArgumentProviders.add(CommandLineArgumentProvider { listOf("--module-path", jars.asPath) }) +} diff --git a/rabosh-testkit/build.gradle.kts b/rabosh-testkit/build.gradle.kts index 6a4b5a0..0bd57b6 100644 --- a/rabosh-testkit/build.gradle.kts +++ b/rabosh-testkit/build.gradle.kts @@ -8,6 +8,20 @@ description = "Test infrastructure: seeded property harness, generators, out-of-process kill harness, " + "fault-injecting filesystem, reference models." +/* + * The reference models are *of* the experimental tier — a `BitSet` oracle for `Bitmap`, a `TreeMap` + * oracle for the LSM — so this module works below the stable core by definition. The published + * modules get the same line from `rabosh.kotlin-library`. + * + * `rabosh-samples` deliberately has neither, and that asymmetry is the whole gate: a sample is a + * consumer, a testkit is not. + */ +kotlin { + compilerOptions { + optIn.add("app.oreshkov.rabosh.RaboshExperimental") + } +} + // Consumed as testImplementation by the other modules, so its dependencies are // api-scoped: a test that uses the harness needs the harness's types on its own path. dependencies { diff --git a/rabosh-variant/api/rabosh-variant.api b/rabosh-variant/api/rabosh-variant.api index 7c746a6..2be2753 100644 --- a/rabosh-variant/api/rabosh-variant.api +++ b/rabosh-variant/api/rabosh-variant.api @@ -1,3 +1,6 @@ +public abstract interface annotation class app/oreshkov/rabosh/RaboshExperimental : java/lang/annotation/Annotation { +} + public final class app/oreshkov/rabosh/variant/DuplicateFieldPolicy : java/lang/Enum { public static final field LAST_WINS Lapp/oreshkov/rabosh/variant/DuplicateFieldPolicy; public static final field REJECT Lapp/oreshkov/rabosh/variant/DuplicateFieldPolicy; diff --git a/rabosh-variant/src/main/kotlin/app/oreshkov/rabosh/RaboshExperimental.kt b/rabosh-variant/src/main/kotlin/app/oreshkov/rabosh/RaboshExperimental.kt new file mode 100644 index 0000000..227bcc4 --- /dev/null +++ b/rabosh-variant/src/main/kotlin/app/oreshkov/rabosh/RaboshExperimental.kt @@ -0,0 +1,61 @@ +package app.oreshkov.rabosh + +/** + * Marks a declaration as outside the stable core: it may change or disappear in any release, with no + * deprecation cycle. + * + * `STABILITY.md` holds the tiers and the promise attached to each. The short version is that the + * stable core is implicit — the surface `Rabosh`, `Query`, `Key`, `Variant` and the samples actually + * use — and everything reached *past* it is marked with this. That is not a demotion of the marked + * declarations; it is the accurate statement, and it is what makes the unmarked list mean something. + * "Major version zero, anything may change" is honest and unactionable: a consumer cannot tell + * whether `Key.of` is as volatile as `IndexCatalog.readColumn`, so the rational response is to wrap + * all of the API or none of it. + * + * **What gets marked is the way *in*, not every member.** A consumer holding a `ColumnReader` had to + * pass through `Rabosh.indexCatalog` or `IndexCatalog.readColumn` to get one, and both of those are + * marked — so the reader's own methods carry nothing. Marking every member instead would + * cost around a hundred and fifty annotations and, worse, would force every stable signature naming + * an experimental *type* to be marked as well, which is a cascade that ends with the stable core + * inside the experimental tier. Gate the entrances. + * + * The counterpart rule, and the reason a few classes here are deliberately **not** marked: a type + * named by a stable signature cannot be marked without dragging that signature in with it. + * `SegmentObserver` is the case worth knowing — `RaboshOptions` takes one, so marking the interface + * would make constructing `RaboshOptions` require opt-in. It is a supported seam and it is stable. + * + * It lives in `rabosh-variant` because that is the only module every other one can see; a marker in + * `rabosh-api` could not be applied in `rabosh-index`, which is the dependency edge this project + * does not have. The package is `app.oreshkov.rabosh` rather than `…rabosh.variant` because it + * belongs to the project rather than to the Variant codec. + * + * `ERROR` rather than a warning: reaching past the stable core is a decision, and a warning in a + * build that does not fail on warnings is a line nobody reads. + * + * ```kotlin + * @OptIn(RaboshExperimental::class) + * fun dumpPostings(db: Rabosh) { … } // one function + * + * // or, for a module that lives down there: + * kotlin { compilerOptions { optIn.add("app.oreshkov.rabosh.RaboshExperimental") } } + * ``` + */ +@RequiresOptIn( + level = RequiresOptIn.Level.ERROR, + message = "This is outside rabosh's stable core: it may change or be removed in any release, " + + "with no deprecation cycle. Opt in with @OptIn(RaboshExperimental::class) — or module-wide " + + "via the compiler's opt-in option — and see STABILITY.md for what each tier promises.", +) +// BINARY is what an opt-in marker is required to have: the compiler reads it from the class file of +// a dependency, and nothing reads it at run time. +@Retention(AnnotationRetention.BINARY) +@Target( + AnnotationTarget.CLASS, + AnnotationTarget.FUNCTION, + AnnotationTarget.PROPERTY, + AnnotationTarget.CONSTRUCTOR, + AnnotationTarget.PROPERTY_GETTER, + AnnotationTarget.PROPERTY_SETTER, + AnnotationTarget.TYPEALIAS, +) +public annotation class RaboshExperimental