Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/workflows/manual-generate-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,18 @@ jobs:
-PinMemoryDb=false \
--no-daemon --stacktrace

# ─── Compact seforim.db: what ships (COMPACT_DB.md) ────────────────────
# The line texts deflated with the packaging's dictionaries, the dev-only
# indexes dropped, 4 KiB pages: ~2.4x smaller. The bundle and the patch
# fan below both use it (the previous releases are compact too).
- name: Compact Seforim Database
run: |
set -euo pipefail
./gradlew :packaging:compactDb -PcompactDb=$PWD/build/seforim.compact.db --no-daemon --stacktrace
rm -f build/seforim.db-wal build/seforim.db-shm
mv build/seforim.compact.db build/seforim.db
ls -lh build/seforim.db

- name: Package Seforim Bundle (split tar.zst)
run: ./gradlew packageSeforimBundle --no-daemon --stacktrace

Expand Down
76 changes: 76 additions & 0 deletions COMPACT_DB.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# The compact seforim.db

The database that ships is not the generator's: `:packaging:compactDb` rewrites it, 2.5x smaller and faster to read,
for the phones (small storage, little RAM for the page cache, slow flash) as for the desktop.

```bash
./gradlew :packaging:compactDb # build/seforim.db → build/seforim.compact.db (about 70 s)
./gradlew :packaging:compactDb -PseforimDb=… -PcompactDb=…
```

## What changes

- **Line texts compressed** (`LineTextCodec`, dao, common code): one byte per Hebrew letter, nikud or taam (UTF-8
takes two), then LZ77 against 60 KiB of the corpus' own text, LZ4's block format. 3.1x on lines of 750 bytes on
average, as much as deflate with a dictionary, but decoding is copies only: no entropy decoding, nothing to set up
per line, no native call, the same Kotlin on the JVM, Android and iOS. Deflate cost 5 µs a line on ART (JNI, the
dictionary's install, the inflate, the UTF-8), more than plain text's reading saved; this reads faster than plain
UTF-8 (2 to 3 times fewer bytes from SQLite, chars straight from a table). Each line still reads on its own,
through the same queries: `content` is `BLOB AS String`, and SQLDelight's adapter decodes it.
- A compressed value is `0xFF`, the dictionary's id, the length in Hebrew bytes, then the LZ sequences; anything
else is plain UTF-8 (the generator writes plain text, so do older databases, and both read the same way).
- The dictionary is in the database (`line_codec_dict`) and in the packaging's resources (`line_codec_dict/1.bin`):
every release compresses with the same one, so deltas carry texts that read with the client's.
`-PtrainDictionaries=<dir>` builds a new one (a release with it can't be reached by a delta: the logical hash
covers the dictionaries).
- `heRef` stays plain TEXT: short (36 bytes), compressing it would save 180 MB for a second decode per line.
- **Indexes**: `idx_line_book_index` covers `charCount` (a book's scrollbar no longer reads all its texts),
`idx_line_toc` is partial (a tenth of the lines name their TOC entry), and the indexes no query used go
(`idx_line_heref`, 313 MB; `idx_link_source_line` and `idx_link_type`, prefixes of others; `idx_toc_text`, the
UNIQUE constraint's twin).
- **Pages**: written once, in order and full (the generator's had 13% free space), 16 KiB: on flash a 16 KiB read
costs about a 4 KiB one's latency and holds 50 compressed lines, a page of the reader (4 KiB pages read 3 times
more often below). No `ANALYZE`: the queries are tuned on plans without statistics.
- **SQLite's memory per platform** (`SqliteTuning`): the desktop keeps its 256 MB cache and 512 MB map; Android
takes a 16 MB cache and no map (mapped pages count in the memory Android kills apps by); iOS 32 MB and 256 MB.
- `PatchDbSchema.CURRENT_VERSION` is 2: a client with a version 1 database takes the full bundle, and an applier of
version 1 refuses version 2 patches. **An app version that predates `LineTextCodec` can't read a compact
database**: the release that ships it needs a minimum app version.

## Results (the 2026-10 corpus: 7,019,636 lines, 4,428,487 links)

| | generator's | compact | |
|---|---:|---:|---:|
| file | 8,022 MB | 3,253 MB | **2.47x** |
| `line` | 6,184 MiB | 2,088 MiB | 3.0x |
| line indexes | 575 MiB | 223 MiB | 2.6x |
| links and their indexes | 545 MiB | 448 MiB | 1.2x |

Every line reads back byte for byte (`CompactDbVerification`, and an independent decoder on a sample). A delta
between two compact releases (20 lines changed) carries those 20 lines, 150 KB, and its apply reproduces the new
release's hash (`producePatchAndVerify`).

Reads through the repository (`DbReadWorkload`), each phase cold (the file dropped from the page cache) then warm.
Cold / warm, in ms:

| | Android 10 (emulator) | small phone (simulated) | desktop (JVM, NVMe) |
|---|---:|---:|---:|
| open 150 books (char counts + 50 lines) | 407–480 / 221–262 → 213–223 / 151–171 | 5,025 / 4,678 → 473 / 58 | 431 / 79 → 177 / 67 |
| jump to 300 pages of random books | 260–268 / 142–144 → 199–204 / 124–131 | 814 / 102 → 476 / 130 | 168 / 59 → 144 / 57 |
| scroll 20 books, 30 pages each | 296–301 / 250–255 → 271–283 / 218–229 | 824 / 140 → 340 / 78 | 122 / 73 → 116 / 87 |
| 3,000 lines at random | 580–626 / 240–252 → 512–517 / 249–258 | 1,559 / 78 → 1,463 / 123 | 380 / 50 → 349 / 54 |
| commentaries of 300 lines | 124–178 / 58–60 → 85–87 / 54 | 200 / 9 → 152 / 9 | 39 / 5 → 45 / 8 |

- **Android 10**: SQLDelight's Android driver on ART, in an x86_64 emulator (2 GB) reading the host's NVMe, two
rounds each: faster cold everywhere, warm as fast or faster.
- **Small phone**: the JVM with Android's SQLite memory over a simulated storage (`LD_PRELOAD`: a 128 MB page
cache, each miss 0.2 ms plus 150 MB/s, an entry-level eMMC): the old database's working set no longer fits.
- **Desktop**: faster cold but for the commentaries (mmap reads ahead), warm within 15 ms.
- Slower: warm random lines on the small phone (78 → 123 ms, the old pages fit its cache there), the desktop's
commentaries (+6 ms cold) and warm scrolling (+14 ms for 1,500 pages).

```bash
BENCH_DB=/path/seforim.db ./gradlew :dao:jvmTest --tests '*DbReadBenchmark*' --rerun -i # BENCH_PROFILE=android
# Android: see AndroidDbReadBenchmark (adb push, then am instrument per phase)
SOURCE_DB=build/seforim.db COMPACT_DB=build/seforim.compact.db ./gradlew :packaging:jvmTest --tests '*CompactDbVerification*' --rerun
```
10 changes: 10 additions & 0 deletions dao/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ kotlin {
namespace = "io.github.kdroidfilter.seforimlibrary.dao"
compileSdk = 35
minSdk = 21
// The reads on a real database, on a device or an emulator (AndroidDbReadBenchmark)
withDeviceTestBuilder { sourceSetTreeName = "test" }.configure {
instrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
}
jvm()
iosArm64()
Expand Down Expand Up @@ -56,6 +60,12 @@ kotlin {
implementation(libs.sqlDelight.driver.native)
}

getByName("androidDeviceTest").dependencies {
implementation(kotlin("test"))
implementation(libs.androidx.test.runner)
implementation(libs.androidx.test.junit)
}

}

}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package io.github.kdroidfilter.seforimlibrary.dao.repository

import android.util.Log
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.platform.app.InstrumentationRegistry
import app.cash.sqldelight.driver.android.AndroidSqliteDriver
import io.github.kdroidfilter.seforimlibrary.db.SeforimDb
import kotlinx.coroutines.runBlocking
import org.junit.Test
import org.junit.runner.RunWith
import java.io.File

/**
* [DbReadWorkload] on Android, through SQLDelight's Android driver, on a database pushed to the app's external files
* directory. Runs only with the instrumentation argument benchDb (the file's name there); `phase` runs one phase, the
* host dropping the page cache before (an app can't: posix_fadvise isn't public on Android):
*
* adb push seforim.db /sdcard/Android/data/io.github.kdroidfilter.seforimlibrary.dao.test/files/
* adb root && adb shell 'echo 3 > /proc/sys/vm/drop_caches'
* adb shell am instrument -w -e benchDb seforim.db -e phase 0 \
* io.github.kdroidfilter.seforimlibrary.dao.test/androidx.test.runner.AndroidJUnitRunner
*/
@RunWith(AndroidJUnit4::class)
class AndroidDbReadBenchmark {
@Test
fun benchmark() {
val name = InstrumentationRegistry.getArguments().getString("benchDb") ?: return
val context = InstrumentationRegistry.getInstrumentation().targetContext
val file = File(context.getExternalFilesDir(null), name)
check(file.exists()) { "No database at $file" }
val driver = AndroidSqliteDriver(SeforimDb.Schema, context, file.absolutePath)
val repository = SeforimRepository(file.absolutePath, driver)
report("BENCH ${file.name} ${file.length() / 1_048_576} MB on Android ${android.os.Build.VERSION.RELEASE}")
val phases = InstrumentationRegistry.getArguments().getString("phase")?.let { setOf(it.toInt()) }
runBlocking { DbReadWorkload(driver, repository, {}, ::readBytes).run(phases, ::report) }
driver.close()
}

private fun report(line: String) {
Log.i("DbReadBenchmark", line)
println(line)
}

private fun readBytes(): Long =
runCatching {
File("/proc/self/io").readLines().first { it.startsWith("read_bytes:") }.substringAfter(':').trim().toLong()
}.getOrDefault(-1)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package io.github.kdroidfilter.seforimlibrary.dao.repository

// Android counts an app's mapped pages in the memory it kills apps by (small devices first): a small cache, no map
internal actual const val SQLITE_CACHE_KIB: Long = 16000
internal actual const val SQLITE_MMAP_BYTES: Long = 0
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ import io.github.kdroidfilter.seforimlibrary.core.models.Source
import io.github.kdroidfilter.seforimlibrary.core.models.TocEntry
import io.github.kdroidfilter.seforimlibrary.core.models.Topic
import io.github.kdroidfilter.seforimlibrary.dao.extensions.toModel
import io.github.kdroidfilter.seforimlibrary.dao.text.LineTextCodec
import io.github.kdroidfilter.seforimlibrary.db.Line as LineRow
import io.github.kdroidfilter.seforimlibrary.db.SeforimDb
import io.github.kdroidfilter.seforimlibrary.env.getEnvironmentVariable
import kotlinx.coroutines.Dispatchers
Expand All @@ -39,7 +41,16 @@ import kotlinx.serialization.json.Json
* @constructor Creates a repository with the specified database path and driver
*/
class SeforimRepository(databasePath: String, private val driver: SqlDriver) : LineSelectionRepository {
private val database = SeforimDb(driver)
// Its dictionaries load in init, once the schema exists
private val lineTextCodec: LineTextCodec =
LineTextCodec {
database.lineQueriesQueries.selectCodecDictionaries().executeAsList().associate { it.id.toInt() to it.data_ }
}
private val database: SeforimDb =
SeforimDb(
driver,
lineAdapter = LineRow.Adapter(contentAdapter = lineTextCodec.columnAdapter),
)
private val json = Json { ignoreUnknownKeys = true }
private val logger = Logger.withTag("SeforimRepository")

Expand Down Expand Up @@ -69,11 +80,13 @@ class SeforimRepository(databasePath: String, private val driver: SqlDriver) : L
logger.d{"Initializing SeforimRepository"}
// Create the database schema (fresh builds only; no runtime migrations needed)
SeforimDb.Schema.create(driver)
// SQLite optimizations (balanced mode for desktop: 256MB cache + 512MB mmap)
lineTextCodec.loadDictionaries()
// SQLite's memory sized to the platform: the desktop's 256 MB cache and 512 MB map would get a small phone's
// app killed (mapped pages count in its memory)
pragma("journal_mode=WAL")
pragma("synchronous=NORMAL")
pragma("cache_size=-256000") // 256MB cache (negative = KiB)
pragma("mmap_size=536870912") // 512MB memory-mapped I/O
pragma("cache_size=-$SQLITE_CACHE_KIB") // negative = KiB
pragma("mmap_size=$SQLITE_MMAP_BYTES")
pragma("temp_store=MEMORY")

// Check if the database is empty
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
package io.github.kdroidfilter.seforimlibrary.dao.repository

/** SQLite's page cache, in KiB. */
internal expect val SQLITE_CACHE_KIB: Long

/** How much of the database SQLite maps into memory, in bytes (0: none). */
internal expect val SQLITE_MMAP_BYTES: Long
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package io.github.kdroidfilter.seforimlibrary.dao.text

// One byte per character of the corpus' alphabet, where UTF-8 takes two for each Hebrew letter, nikud and taam:
// 0x00-0x7F ASCII, 0x80-0xEF U+0590-U+05FF, 0xF0-0xFD the commonest other characters, 0xFE then a UTF-16 unit
private const val HEBREW_FIRST = 0x0590
private const val HEBREW_LAST = 0x05FF
private const val HEBREW_BYTE = 0x80
private const val COMMON_BYTE = 0xF0
private const val ESCAPE = 0xFE

// Dashes, quotes, the bidi marks, no-break space
private val COMMON_CHARS =
charArrayOf(
'\u2013', '\u2014', '\u201D', '\u201C', '\u2019', '\u2018', '\u200E',
'\u00A0', '\u202C', '\u202B', '\u200F', '\u202A', '\u2026', '\u00B7',
)

private val BYTE_CHARS =
CharArray(ESCAPE) { b ->
when {
b < HEBREW_BYTE -> b.toChar()
b < COMMON_BYTE -> (HEBREW_FIRST + b - HEBREW_BYTE).toChar()
else -> COMMON_CHARS[b - COMMON_BYTE]
}
}

/** [text] in Hebrew bytes. */
internal fun encodeHebrewBytes(text: String): ByteArray {
val out = ByteArray(text.length * 3)
var n = 0
for (c in text) {
val code = c.code
when {
code < HEBREW_BYTE -> out[n++] = code.toByte()
code in HEBREW_FIRST..HEBREW_LAST -> out[n++] = (HEBREW_BYTE + code - HEBREW_FIRST).toByte()
else -> {
val common = COMMON_CHARS.indexOf(c)
if (common >= 0) {
out[n++] = (COMMON_BYTE + common).toByte()
} else {
out[n++] = ESCAPE.toByte()
out[n++] = (code ushr 8).toByte()
out[n++] = code.toByte()
}
}
}
}
return out.copyOf(n)
}

/** The text of [length] Hebrew bytes of [bytes]. */
internal fun decodeHebrewBytes(bytes: ByteArray, length: Int): String {
val chars = CharArray(length)
var n = 0
var i = 0
while (i < length) {
val b = bytes[i++].toInt() and 0xFF
if (b == ESCAPE) {
chars[n++] = (((bytes[i].toInt() and 0xFF) shl 8) or (bytes[i + 1].toInt() and 0xFF)).toChar()
i += 2
} else {
chars[n++] = BYTE_CHARS[b]
}
}
return chars.concatToString(0, n)
}
Loading
Loading