...
" + "references": [{"osis": "Gen.1.1", "book": 1, "chapter": 1, "verse": 1}] } ] } ``` -Book and chapter introductions use chapter or verse `0`; they are not discarded. +Introductions are published, not discarded. A book introduction is chapter `0`, +so Clarke's introduction to Daniel is `clarke/27/0.json`. A chapter introduction +is verse `0`, and appears as the first entry of its own chapter document. + +`books.json` reports which books and chapters a commentary covers, and +`metadata.json` reports its licence, counts, and the byte size of the +whole-commentary document so a client can decide before requesting it. ## Dictionary API ```text GET https://dictionaries.getbible.net/v1/dictionaries.json +GET https://dictionaries.getbible.net/v1/{dictionary}.json GET https://dictionaries.getbible.net/v1/{dictionary}/metadata.json -GET https://dictionaries.getbible.net/v1/{dictionary}/keys.json +GET https://dictionaries.getbible.net/v1/{dictionary}/index.json GET https://dictionaries.getbible.net/v1/{dictionary}/{entry}.json -GET https://dictionaries.getbible.net/v1/{dictionary}/indexes/{sha256-prefix}.json ``` +Searching a dictionary takes two requests. `index.json` lists every word once, +sorted by an accent-insensitive lowercase `search` term, so a client can fetch it +once and then search, prefix-match, or binary-search entirely in memory: + +```json +{"id": "k-KADESH", "key": "KADESH", "search": "kadesh"} +``` + +The record's `id` is the path of the word itself — `{entry}.json` — so a hit in +the index resolves to exactly one document with no further lookup. + Strong's paths match Bible API v3 tokens directly: ```text @@ -133,17 +165,48 @@ H0430 -> https://dictionaries.getbible.net/v1/strongshebrew/H0430.json Greek keys use `G` plus the unpadded number; Hebrew keys use `H0` plus the unpadded number. Other dictionary keys receive deterministic, path-safe IDs. -`keys.json` maps source keys and aliases, while 256 SHA-256-prefix shards provide -smaller lookup indexes for constrained clients. + +Each word document carries the dictionary's own link graph, so a client can +navigate in either direction without rebuilding an index: + +```json +{ + "schema": "getbible-dictionary-entry-v1", + "dictionary": "easton", + "id": "k-KADESH", + "key": "KADESH", + "occurrence": 1, + "aliases": ["KADESH"], + "text": "Holy; a place in the wilderness of Zin.", + "see_also": [{"id": "k-MERIBAH", "key": "MERIBAH"}], + "backlinks": [{"id": "k-ZIN", "key": "ZIN"}], + "references": [{"osis": "Num.20.1", "book": 4, "chapter": 20, "verse": 1}] +} +``` + +`see_also` lists the words this entry points at and `backlinks` the words that +point back. Only targets that resolve to a real key in the same dictionary are +published. Scripture references stay in `references`, in the same shape the +commentary API uses. Some SWORD dictionaries legitimately contain more than one definition for the same public key. The first definition keeps the canonical direct path, and later definitions receive deterministic `--2`, `--3`, and subsequent suffixes. For example, Easton's repeated `KADESH` records are available as `k-KADESH.json` and -`k-KADESH--2.json`. Every definition appears in `keys.json` with an `occurrence` +`k-KADESH--2.json`. Every definition appears in `index.json` with an `occurrence` value. Dictionary metadata reports both the total `entry_count` and the distinct `unique_key_count`. +`{dictionary}.json` is the complete dictionary in index order, for offline +clients that would otherwise request every word individually. + +## Integrity and schemas + +Each API root publishes `hashes.json`, a SHA-256 digest of every other generated +document, which is also the manifest of the paths a build owns. The JSON Schemas +for every document type are served beside the data under `v1/schema/`, so each +schema `$id` resolves to the document that defines it. + ## Build flow ```mermaid @@ -152,15 +215,37 @@ flowchart TD B --> C["NDJSON v1 subprocess stream"] C --> D["Independent stream + artifact validator"] D --> E["Python API adapter + JSON Schema"] - E --> F["Atomic static v1 trees + hash sidecars"] - F --> G["v1_commentaries, when publication secrets exist"] - F --> H["v1_dictionaries, when publication secrets exist"] + E --> F["Atomic static v1 trees + SHA-256 manifest"] + F --> G["commentaries, when publication secrets exist"] + F --> H["dictionaries, when publication secrets exist"] ``` The static output is the system of record. Nginx and a CDN can serve direct lookups without an application process, database connection pool, or request throttling bottleneck. +## Deployment + +A production origin is a pull, a verify, a compress, and a sync: + +```bash +scripts/deploy_static_api.sh \ + --repo git@github.com:getbible/commentaries.git \ + --root /var/www/getbible/commentaries \ + --require-signature + +scripts/verify_live_api.sh https://commentaries.getbible.net +``` + +The whole tree is checked against `hashes.json` before it reaches the live root, +so a failed build leaves the previous one serving. + +`docs/nginx/` holds the origin configuration for both hosts; install it with +`scripts/install_nginx_config.sh`, which adapts it to the host's nginx version, +brotli availability, and IPv6 support rather than leaving those as footguns. +`docs/deployment.md` describes the server layout, the caching model, the CDN and +security posture, rollback, and monitoring. + ## Local development ```bash @@ -232,8 +317,8 @@ Publication secret set: | `GETBIBLE_SSH_KEY` | SSH private key with write access to both outputs | | `GETBIBLE_SSH_PUB` | Matching public key | -The default output remotes are `getbible/v1_commentaries` and -`getbible/v1_dictionaries`. Optional `GETBIBLE_COMMENTARIES_REPO` and +The default output remotes are `getbible/commentaries` and +`getbible/dictionaries`. Optional `GETBIBLE_COMMENTARIES_REPO` and `GETBIBLE_DICTIONARIES_REPO` secrets may select staging remotes. ## Redistribution policy diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..bba9a4c --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,330 @@ +# Deploying the study APIs + +Two origins, each serving one generated repository as static JSON. There is no +application process, no database, and no request-time logic — a request is a +file read, which is why the whole design leans on the filesystem and the CDN +rather than on code. + +| Origin | Serves | Live root | +| --- | --- | --- | +| `commentaries.getbible.net` | `getbible/commentaries` | `/var/www/getbible/commentaries` | +| `dictionaries.getbible.net` | `getbible/dictionaries` | `/var/www/getbible/dictionaries` | + +Everything below applies identically to both. Where a command names one, run the +same command with the other's repository and root. + +## What the builder guarantees + +The deploy depends on four properties. If you change the builder, keep them, or +the deployment stops being safe: + +1. **Output is byte-stable.** A module that has not changed rebuilds to an + identical file. Content documents carry no timestamp — only `build.json` and + the catalog do — so an unchanged corpus produces an unchanged tree. +2. **`v1/hashes.json` describes the whole tree.** It holds a SHA-256 for every + other document, so it is both the integrity manifest and the list of paths + the builder owns. +3. **The builder owns only version directories.** Anything else in the + publication repository — README, licence, workflows — is the repository's + own and is never served. +4. **Every document is plain text JSON.** No HTML is published anywhere, which + is why the origin can send `Content-Security-Policy: default-src 'none'` and + why no consumer has to sanitize a response. + +## Server layout + +``` +/var/lib/getbible/commentaries persistent Git checkout (working copy) +/var/www/getbible/commentaries live root nginx serves — version dirs only +/etc/nginx/conf.d/getbible-api-http.conf zones and log format (http only) +/etc/nginx/snippets/getbible-api-server.conf tuning, TLS, compression +/etc/nginx/snippets/getbible-api-headers.conf CORS and security headers +/etc/nginx/snippets/getbible-api-v1.conf the v1 locations +/etc/nginx/sites-available/commentaries.getbible.net.conf +``` + +The live root is **not** a Git checkout. The checkout stays in `/var/lib`, and +only the version directories are copied across. That keeps repository metadata +off the origin entirely rather than relying on a rule to hide it. + +## First-time setup + +Requires nginx (1.25.1+ preferred; older is adapted automatically), git, +rsync, python3, gzip, and ideally brotli. + +```bash +apt-get install -y nginx git rsync python3 certbot brotli +# Brotli for nginx is a separate module; without it only .gz is served, which +# still works and is roughly 15-20% larger on JSON. +apt-get install -y libnginx-mod-http-brotli # where packaged + +install -d -m 755 /var/lib/getbible +``` + +Issue the certificates first — nginx refuses to start when a referenced +certificate is missing. The stock nginx site already serves `/var/www/html` on +port 80, so this works before anything below is installed, and the installed +configuration keeps serving the same webroot afterwards so renewals need no +further changes: + +```bash +certbot certonly --webroot -w /var/www/html \ + -d commentaries.getbible.net -d dictionaries.getbible.net +``` + +Then install the origin configuration: + +```bash +scripts/install_nginx_config.sh --reload +``` + +Do not copy `docs/nginx/` into place by hand. Three things legitimately differ +between hosts, and guessing wrong on any of them stops nginx from starting, so +the installer detects them instead: + +- **`http2 on` is nginx 1.25.1+.** On older nginx — Ubuntu 24.04 LTS still ships + 1.24 — it is folded back into the `listen` line. +- **`brotli_static` needs `ngx_brotli`**, which many distributions do not + package. It is enabled only when the module is actually present. +- **IPv6 listeners fail outright** on a host without IPv6. + +The installer stages the adapted files, refuses to proceed if a certificate is +missing (naming the exact certbot command), and runs `nginx -t` before +reloading. `--dry-run` shows what it would install and leaves the adapted files +for inspection. + +Reload nginx automatically after each renewal: + +```bash +cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK' +#!/bin/sh +systemctl reload nginx +HOOK +chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh +``` + +## Deploying + +```bash +scripts/deploy_static_api.sh \ + --repo git@github.com:getbible/commentaries.git \ + --root /var/www/getbible/commentaries \ + --require-signature \ + --verify-url https://commentaries.getbible.net/v1/commentaries.json +``` + +The script does five things, and the order is the point — nothing reaches the +live root until it has been proven good: + +1. **Pull.** `git reset --hard` into the persistent checkout. Git rewrites only + files whose content changed, so every unchanged document keeps its mtime. +2. **Verify.** Every digest in `hashes.json` is recomputed, and any `.json` in + the tree that the manifest does not list is an error. A build that fails here + never reaches the live root; the previous deploy keeps serving. +3. **Compress.** `.gz` and `.br` are written beside each document, but only + where missing or stale, and each variant is stamped with its document's + mtime. A monthly rebuild that changes two modules recompresses two modules. +4. **Sync.** `rsync --delete --delay-updates`, one version directory at a time. + Scoping the delete to a single version means no failure of this script can + remove anything outside a version directory. +5. **Reload.** `nginx -t` then reload. New workers start with an empty + open-file cache, which is what makes the new documents visible at once. + +`--require-signature` refuses any commit without a valid GPG signature. The +build workflow signs its commits when the publication secrets are present, so on +a production origin this should always be on: it means the origin serves only +what the build key signed. + +It verifies against the deploying user's own GPG keyring, so import and trust +the build key once on each origin before enabling it — otherwise every deploy +fails closed with an unsigned-commit error: + +```bash +sudo -u deploy gpg --import getbible-build-key.asc +sudo -u deploy gpg --lsign-keyA comment on creation.
", - "verse": { - "osis": "Gen.1.1", - "testament": 1, - "book": 1, - "chapter": 1, - "verse": 1, - }, - } - ], - ) +def entry(osis, book, chapter, verse, text, *, raw=None, html=""): + return { + "record_type": "entry", + "key": osis, + "raw": raw if raw is not None else text, + "plain": text, + "html": html, + "verse": { + "osis": osis, + "testament": 1 if book <= 39 else 2, + "book": book if book <= 39 else book - 39, + "chapter": chapter, + "verse": verse, + }, + } + + +def write(tmp_path, project_root, module, entries): writer = CommentaryWriter( tmp_path, BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", + project_root / "schemas", ) - summary = writer.write(commentary_module, export) - chapter_path = tmp_path / "testcom/1/1.json" - chapter = json.loads(chapter_path.read_text(encoding="utf-8")) - assert summary["chapter_url_template"] == "{book}/{chapter}.json" - assert (chapter["book"], chapter["chapter"]) == (1, 1) - assert chapter["entries"][0]["verse"] == 1 - assert chapter["entries"][0]["anchor"]["osis"] == "Gen.1.1" - assert chapter["entries"][0]["references"][0]["book"] == 43 + return writer.write(module, NativeExport(metadata={"record_type": "module"}, entries=entries)) -def test_commentary_consumes_source_entries_once(tmp_path, project_root, commentary_module) -> None: - entries = OnePassEntries( +def test_commentary_matches_v3_book_chapter_verse_contract( + tmp_path, project_root, commentary_module +) -> None: + record, metadata = write( + tmp_path, + project_root, + commentary_module, [ - { - "key": "Genesis 1:1", - "raw": "First chapter", - "plain": "First chapter", - "html": "", - "verse": { - "osis": "Gen.1.1", - "testament": 1, - "book": 1, - "chapter": 1, - "verse": 1, - }, - }, - { - "key": "Genesis 2:1", - "raw": "Second chapter", - "plain": "Second chapter", - "html": "", - "verse": { - "osis": "Gen.2.1", - "testament": 1, - "book": 1, - "chapter": 2, - "verse": 1, - }, - }, - ] + entry( + "John.1.1", + 43, + 1, + 1, + "A comment on creation.", + raw='A comment on creation.
", + ) + ], ) - export = NativeExport(metadata={}, entries=entries) + chapter = json.loads((tmp_path / "testcom/43/1.json").read_text(encoding="utf-8")) + assert metadata["chapter_url_template"] == "{book}/{chapter}.json" + assert (chapter["book"], chapter["chapter"]) == (43, 1) + assert chapter["entries"][0]["verse"] == 1 + assert chapter["entries"][0]["anchor"]["osis"] == "John.1.1" + assert chapter["entries"][0]["references"][0]["book"] == 43 + assert record["entry_count"] == 1 - summary = CommentaryWriter( - tmp_path, - BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", - ).write(commentary_module, export) - assert entries.iterated - assert summary["entry_count"] == 2 - assert (tmp_path / "testcom/1/1.json").is_file() - assert (tmp_path / "testcom/1/2.json").is_file() +def test_commentary_publishes_text_without_markup( + tmp_path, project_root, commentary_module +) -> None: + write( + tmp_path, + project_root, + commentary_module, + [entry("Gen.1.1", 1, 1, 1, "Plain words.", html="Plain words.
")], + ) + chapter = json.loads((tmp_path / "testcom/1/1.json").read_text(encoding="utf-8")) + published = chapter["entries"][0] + assert published["text"] == "Plain words." + assert "html" not in published + assert "html" not in (tmp_path / "testcom/1/1.json").read_text(encoding="utf-8") -def test_commentary_skips_entries_without_a_scripture_chapter( +def test_commentary_publishes_book_and_chapter_introductions( tmp_path, project_root, commentary_module ) -> None: - export = NativeExport( - metadata={}, - entries=[ - { - "key": "Genesis", - "raw": "Book introduction", - "plain": "Book introduction", - "html": "", - "verse": { - "osis": "Gen", - "testament": 1, - "book": 1, - "chapter": 0, - "verse": 0, - }, - }, - { - "key": "Genesis 1", - "raw": "Chapter introduction", - "plain": "Chapter introduction", - "html": "", - "verse": { - "osis": "Gen.1", - "testament": 1, - "book": 1, - "chapter": 1, - "verse": 0, - }, - }, + write( + tmp_path, + project_root, + commentary_module, + [ + entry("Dan.0.0", 27, 0, 0, "About the book of Daniel."), + entry("Dan.1.0", 27, 1, 0, "About chapter one."), + entry("Dan.1.1", 27, 1, 1, "On the first verse."), ], ) + introduction = json.loads((tmp_path / "testcom/27/0.json").read_text(encoding="utf-8")) + assert introduction["chapter"] == 0 + assert introduction["entries"][0]["name"] == "Daniel" + assert introduction["entries"][0]["text"] == "About the book of Daniel." - summary = CommentaryWriter( - tmp_path, - BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", - ).write(commentary_module, export) + chapter = json.loads((tmp_path / "testcom/27/1.json").read_text(encoding="utf-8")) + assert [item["verse"] for item in chapter["entries"]] == [0, 1] + assert chapter["entries"][0]["name"] == "Daniel 1" - chapter = json.loads((tmp_path / "testcom/1/1.json").read_text(encoding="utf-8")) - assert summary["entry_count"] == 1 - assert not (tmp_path / "testcom/1/0.json").exists() - assert [entry["name"] for entry in chapter["entries"]] == ["Genesis 1"] + books = json.loads((tmp_path / "testcom/books.json").read_text(encoding="utf-8")) + assert books["books"][0]["chapters"] == [0, 1] -def test_commentary_canonicalizes_interleaved_source_chapters( +def test_book_and_commentary_documents_embed_their_parts_verbatim( tmp_path, project_root, commentary_module ) -> None: - def source(osis: str, testament: int, book: int, chapter: int, verse: int, text: str): - return { - "key": osis, - "raw": text, - "plain": text, - "html": "", - "verse": { - "osis": osis, - "testament": testament, - "book": book, - "chapter": chapter, - "verse": verse, - }, - } + write( + tmp_path, + project_root, + commentary_module, + [ + entry("Gen.1.1", 1, 1, 1, "First."), + entry("Gen.2.1", 1, 2, 1, "Second."), + entry("John.1.1", 43, 1, 1, "Third."), + ], + ) + chapters = [ + json.loads((tmp_path / f"testcom/1/{number}.json").read_text(encoding="utf-8")) + for number in (1, 2) + ] + book = json.loads((tmp_path / "testcom/1.json").read_text(encoding="utf-8")) + assert book["schema"] == "getbible-commentary-book-v1" + assert book["chapters"] == chapters + + complete = json.loads((tmp_path / "testcom.json").read_text(encoding="utf-8")) + books = [ + json.loads((tmp_path / f"testcom/{number}.json").read_text(encoding="utf-8")) + for number in (1, 43) + ] + assert complete["schema"] == "getbible-commentary-v1" + assert complete["books"] == books + + +def test_commentary_reports_counts_and_bulk_size(tmp_path, project_root, commentary_module) -> None: + record, metadata = write( + tmp_path, + project_root, + commentary_module, + [ + entry("Gen.1.1", 1, 1, 1, "First."), + entry("Gen.2.1", 1, 2, 1, "Second."), + entry("John.1.1", 43, 1, 1, "Third."), + ], + ) + assert (record["book_count"], record["chapter_count"], record["entry_count"]) == (2, 3, 3) + assert record["bytes"] == (tmp_path / "testcom.json").stat().st_size + assert metadata["bytes"] == record["bytes"] + +def test_commentary_consumes_source_entries_once(tmp_path, project_root, commentary_module) -> None: entries = OnePassEntries( [ - source("2Macc.7.1", 1, 48, 7, 1, "Maccabees first"), - source("Job.2.1", 1, 18, 2, 1, "Job second chapter"), - source("Job.1.2", 1, 18, 1, 2, "Job second verse"), - source("2Macc.7.1", 1, 48, 7, 1, "Maccabees first"), - source("Job.1.1", 1, 18, 1, 1, "Job first verse"), + entry("Gen.1.1", 1, 1, 1, "First chapter"), + entry("Gen.2.1", 1, 2, 1, "Second chapter"), ] ) - export = NativeExport(metadata={}, entries=entries) + write(tmp_path, project_root, commentary_module, entries) + assert entries.passes == 1 + assert (tmp_path / "testcom/1/1.json").is_file() + assert (tmp_path / "testcom/1/2.json").is_file() - summary = CommentaryWriter( - tmp_path, - BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", - ).write(commentary_module, export) - books = json.loads((tmp_path / "testcom/books.json").read_text(encoding="utf-8")) - job = json.loads((tmp_path / "testcom/18.json").read_text(encoding="utf-8")) - job_one = json.loads((tmp_path / "testcom/18/1.json").read_text(encoding="utf-8")) - maccabees_seven = json.loads((tmp_path / "testcom/81/7.json").read_text(encoding="utf-8")) - - assert entries.iterated - assert summary["entry_count"] == 4 - assert [item["book"] for item in books] == [18, 81] - assert [item["chapter"] for item in job["chapters"]] == [1, 2] - assert [item["verse"] for item in job_one["entries"]] == [1, 2] - assert list(job_one["entries"][0]) == ["book", "chapter", "verse", "name", "anchor", "text"] - assert len(maccabees_seven["entries"]) == 1 +def test_commentary_skips_entries_without_a_bible_coordinate( + tmp_path, project_root, commentary_module +) -> None: + record, _ = write( + tmp_path, + project_root, + commentary_module, + [ + {"key": "Preface", "raw": "front matter", "plain": "front matter", "html": ""}, + entry("Gen.1.1", 1, 1, 1, "Kept."), + ], + ) + assert record["entry_count"] == 1 diff --git a/tests/test_content.py b/tests/test_content.py index 5e0c19a..a6e81d2 100644 --- a/tests/test_content.py +++ b/tests/test_content.py @@ -1,21 +1,32 @@ -from study_builder.content import clean_html, extract_osis_references, public_content +from study_builder.content import extract_osis_references, public_content, strip_markup -def test_html_sanitizer_removes_scripts_and_unsafe_links() -> None: - cleaned = clean_html( - "Hello world
" - 'bad' +def test_markup_stripper_drops_scripts_and_keeps_readable_text() -> None: + text = strip_markup( + 'Safe text
' + 'link' ) - assert "A word
"}) + assert content == {"text": "A word"} -def test_structural_html_without_text_is_not_public_content() -> None: +def test_public_content_falls_back_to_markup_when_stripped_text_is_empty() -> None: + content = public_content({"plain": "", "html": "Only in the rendered form
"}) + assert content == {"text": "Only in the rendered form"} + + +def test_structural_markup_without_text_is_not_public_content() -> None: assert public_content({"plain": "", "html": 'logos: a word
", } ], + metadata={"feature": "GreekDef"}, ) - summary = DictionaryWriter( - tmp_path, project_root / "schemas/dictionary-entry.schema.json" - ).write(greek_dictionary_module, export) - path = tmp_path / "strongsgreek/G3056.json" - document = json.loads(path.read_text(encoding="utf-8")) - assert summary["strong_prefix"] == "G" + document = json.loads((tmp_path / "strongsgreek/G3056.json").read_text(encoding="utf-8")) + assert record["strong_prefix"] == "G" assert document["id"] == "G3056" assert document["occurrence"] == 1 assert document["aliases"] == ["03056", "G3056"] - assert document["references"] == ["John.1.1"] + assert document["text"] == "logos: a word" + assert "html" not in document + assert document["references"] == [{"osis": "John.1.1", "book": 43, "chapter": 1, "verse": 1}] + assert metadata["index_url"] == "index.json" + + +def test_dictionary_index_is_sorted_and_slim( + tmp_path, project_root, greek_dictionary_module +) -> None: + write( + tmp_path, + project_root, + greek_dictionary_module, + [ + {"key": "Zeta", "raw": "last", "plain": "last", "html": ""}, + {"key": "Alpha", "raw": "first", "plain": "first", "html": ""}, + ], + ) + index = json.loads((tmp_path / "strongsgreek/index.json").read_text(encoding="utf-8")) + assert index["schema"] == "getbible-dictionary-index-v1" + assert [record["key"] for record in index["entries"]] == ["Alpha", "Zeta"] + assert index["entries"][0] == {"id": "k-Alpha", "key": "Alpha", "search": "alpha"} + assert index["entry_url_template"] == "{entry}.json" + + +def test_dictionary_links_resolve_in_both_directions( + tmp_path, project_root, greek_dictionary_module +) -> None: + write( + tmp_path, + project_root, + greek_dictionary_module, + [ + { + "key": "KADESH", + "raw": 'also Meribah', + "plain": "A place in the wilderness.", + "html": "", + }, + { + "key": "MERIBAH", + "raw": "Waters of strife.", + "plain": "Waters of strife.", + "html": "", + }, + ], + ) + kadesh = json.loads((tmp_path / "strongsgreek/k-KADESH.json").read_text(encoding="utf-8")) + meribah = json.loads((tmp_path / "strongsgreek/k-MERIBAH.json").read_text(encoding="utf-8")) + assert kadesh["see_also"] == [{"id": "k-MERIBAH", "key": "MERIBAH"}] + assert "backlinks" not in kadesh + assert meribah["backlinks"] == [{"id": "k-KADESH", "key": "KADESH"}] + assert "see_also" not in meribah def test_dictionary_preserves_repeated_keys_as_distinct_definitions( tmp_path, project_root, greek_dictionary_module ) -> None: - export = NativeExport( - metadata={"feature": "GreekDef"}, - entries=[ + record, _ = write( + tmp_path, + project_root, + greek_dictionary_module, + [ {"key": "03056", "raw": "First", "plain": "First", "html": ""}, {"key": "03056", "raw": "Second", "plain": "Second", "html": ""}, ], + metadata={"feature": "GreekDef"}, ) - summary = DictionaryWriter( - tmp_path, project_root / "schemas/dictionary-entry.schema.json" - ).write(greek_dictionary_module, export) - first = json.loads((tmp_path / "strongsgreek/G3056.json").read_text(encoding="utf-8")) second = json.loads((tmp_path / "strongsgreek/G3056--2.json").read_text(encoding="utf-8")) - keys = json.loads((tmp_path / "strongsgreek/keys.json").read_text(encoding="utf-8")) + index = json.loads((tmp_path / "strongsgreek/index.json").read_text(encoding="utf-8")) assert first["text"] == "First" assert first["occurrence"] == 1 assert second["text"] == "Second" assert second["occurrence"] == 2 - assert [entry["url"] for entry in keys] == ["G3056.json", "G3056--2.json"] - assert summary["entry_count"] == 2 - assert summary["unique_key_count"] == 1 + assert [item["id"] for item in index["entries"]] == ["G3056", "G3056--2"] + assert index["entries"][1]["occurrence"] == 2 + assert record["entry_count"] == 2 + assert record["unique_key_count"] == 1 + + +def test_whole_dictionary_embeds_entries_in_index_order( + tmp_path, project_root, greek_dictionary_module +) -> None: + record, _ = write( + tmp_path, + project_root, + greek_dictionary_module, + [ + {"key": "Zeta", "raw": "last", "plain": "last", "html": ""}, + {"key": "Alpha", "raw": "first", "plain": "first", "html": ""}, + ], + ) + complete = json.loads((tmp_path / "strongsgreek.json").read_text(encoding="utf-8")) + entries = [ + json.loads((tmp_path / f"strongsgreek/k-{name}.json").read_text(encoding="utf-8")) + for name in ("Alpha", "Zeta") + ] + assert complete["schema"] == "getbible-dictionary-v1" + assert complete["entries"] == entries + assert record["bytes"] == (tmp_path / "strongsgreek.json").stat().st_size diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py new file mode 100644 index 0000000..e8dbb45 --- /dev/null +++ b/tests/test_pipeline.py @@ -0,0 +1,67 @@ +import pytest + +from study_builder.pipeline import ( + BASE_URLS, + CATALOG_TEMPLATES, + RESERVED_MODULE_IDS, + BuildPipeline, + PipelineConfig, +) + + +@pytest.fixture +def pipeline(tmp_path, project_root) -> BuildPipeline: + return BuildPipeline( + PipelineConfig( + root=project_root, + work_dir=tmp_path / "work", + dist_dir=tmp_path / "dist", + policy_path=project_root / "conf/module_policy.json", + books_path=project_root / "conf/book_registry.json", + schemas_dir=project_root / "schemas", + engine_manifest_path=project_root / "conf/getbiblesword.json", + engine_schema_path=project_root / "schemas/getbiblesword-ndjson-v1.schema.json", + ) + ) + + +def test_reserved_identifiers_cover_every_root_document() -> None: + # A module directory and its whole-module document share the v1 root with these. + for name in ("commentaries", "dictionaries", "build", "hashes", "schema"): + assert name in RESERVED_MODULE_IDS + + +def test_every_resource_publishes_a_base_url_and_url_templates() -> None: + for kind in ("commentaries", "dictionaries"): + assert BASE_URLS[kind].endswith("/v1/") + assert CATALOG_TEMPLATES[kind] + for template in CATALOG_TEMPLATES[kind].values(): + assert template.startswith("{" + kind.removesuffix("ies") + "y}") + + +def test_schemas_are_published_beside_the_data(pipeline, tmp_path) -> None: + root = tmp_path / "generated" + pipeline._publish_schemas(root, "commentaries") + published = {path.name for path in (root / "schema").glob("*.json")} + assert published == { + "commentary.json", + "commentary-book.json", + "commentary-books.json", + "commentary-chapter.json", + } + + dictionary_root = tmp_path / "generated-dictionaries" + pipeline._publish_schemas(dictionary_root, "dictionaries") + published = {path.name for path in (dictionary_root / "schema").glob("*.json")} + assert published == {"dictionary.json", "dictionary-entry.json", "dictionary-index.json"} + + +def test_published_schema_ids_match_their_served_paths(pipeline, tmp_path) -> None: + from study_builder.util import read_json + + for kind in ("commentaries", "dictionaries"): + root = tmp_path / kind + pipeline._publish_schemas(root, kind) + for path in sorted((root / "schema").glob("*.json")): + expected = f"{BASE_URLS[kind].removesuffix('v1/')}schema/v1/{path.name}" + assert read_json(path)["$id"] == expected diff --git a/tests/test_pipeline_build.py b/tests/test_pipeline_build.py new file mode 100644 index 0000000..a6625c6 --- /dev/null +++ b/tests/test_pipeline_build.py @@ -0,0 +1,156 @@ +"""Exercise the whole publication assembly without CrossWire or the extractor.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from study_builder import pipeline as pipeline_module +from study_builder.models import ModuleDescriptor, NativeExport +from study_builder.pipeline import BuildPipeline, PipelineConfig + + +def descriptor(name: str, driver: str, category: str, **extra: str) -> ModuleDescriptor: + fields = { + "description": (f"{name} Description",), + "lang": ("en",), + "moddrv": (driver,), + "category": (category,), + "distributionlicense": ("Public Domain",), + "version": ("1.0",), + } + fields.update({key: (value,) for key, value in extra.items()}) + return ModuleDescriptor(name=name, fields=fields, conf_path=f"mods.d/{name.lower()}.conf") + + +COMMENTARY = descriptor("Clarke", "zCom", "Commentaries") +DICTIONARY = descriptor("Easton", "RawLD", "Lexicons / Dictionaries") + +EXPORTS: dict[str, NativeExport] = { + "Clarke": NativeExport( + metadata={"classification": "commentary"}, + entries=[ + { + "key": "Dan.0.0", + "raw": "Introduction to Daniel.", + "plain": "Introduction to Daniel.", + "html": "", + "verse": {"osis": "Dan.0.0", "testament": 1, "book": 27, "chapter": 0, "verse": 0}, + }, + { + "key": "Dan.1.1", + "raw": "In the third year.", + "plain": "In the third year.", + "html": "In the third year.
", + "verse": {"osis": "Dan.1.1", "testament": 1, "book": 27, "chapter": 1, "verse": 1}, + }, + ], + ), + "Easton": NativeExport( + metadata={"classification": "dictionary_or_lexicon"}, + entries=[ + { + "key": "KADESH", + "raw": 'see ZIN and Num.20.1', + "plain": "Holy.", + "html": "", + }, + {"key": "ZIN", "raw": "a low palm tree", "plain": "A low palm tree.", "html": ""}, + ], + ), +} + + +class StubInstaller: + def __init__(self, *args, **kwargs) -> None: + pass + + def install(self, module: ModuleDescriptor) -> Path: + return Path("/nonexistent") / module.name + + +class StubExporter: + def __init__(self, *args, **kwargs) -> None: + pass + + def export(self, installation: Path, name: str) -> NativeExport: + return EXPORTS[name] + + +@pytest.fixture +def built(tmp_path, project_root, monkeypatch): + monkeypatch.setattr(pipeline_module, "ModuleInstaller", StubInstaller) + monkeypatch.setattr(pipeline_module, "SwordExporter", StubExporter) + monkeypatch.setattr(BuildPipeline, "_catalog", lambda self: [COMMENTARY, DICTIONARY]) + monkeypatch.setattr( + pipeline_module.GetBibleSwordManager, "ensure", lambda self, path=None: Path("/stub") + ) + config = PipelineConfig( + root=project_root, + work_dir=tmp_path / "work", + dist_dir=tmp_path / "dist", + policy_path=project_root / "conf/module_policy.json", + books_path=project_root / "conf/book_registry.json", + schemas_dir=project_root / "schemas", + engine_manifest_path=project_root / "conf/getbiblesword.json", + engine_schema_path=project_root / "schemas/getbiblesword-ndjson-v1.schema.json", + ) + report = BuildPipeline(config).run() + return report, tmp_path / "dist" + + +def test_build_publishes_both_resources_under_v1(built) -> None: + report, dist = built + assert report.built == {"commentaries": ["Clarke"], "dictionaries": ["Easton"]} + assert not report.failed + assert (dist / "commentaries/v1/clarke.json").is_file() + assert (dist / "commentaries/v1/clarke/27/0.json").is_file() + assert (dist / "dictionaries/v1/easton/index.json").is_file() + assert (dist / "dictionaries/v1/easton/k-KADESH.json").is_file() + + +def test_catalog_url_templates_resolve_from_the_version_root(built) -> None: + _, dist = built + catalog = json.loads((dist / "commentaries/v1/commentaries.json").read_text(encoding="utf-8")) + assert catalog["base_url"] == "https://commentaries.getbible.net/v1/" + assert catalog["module_count"] == 1 + record = catalog["commentaries"][0] + assert record["id"] == "clarke" + assert "about" not in record and "copyright" not in record + + for template, replacements in ( + (catalog["chapter_url_template"], {"commentary": "clarke", "book": "27", "chapter": "1"}), + (catalog["book_url_template"], {"commentary": "clarke", "book": "27"}), + (catalog["commentary_url_template"], {"commentary": "clarke"}), + (catalog["books_url_template"], {"commentary": "clarke"}), + (catalog["metadata_url_template"], {"commentary": "clarke"}), + ): + relative = template + for key, value in replacements.items(): + relative = relative.replace("{" + key + "}", value) + assert (dist / "commentaries/v1" / relative).is_file(), relative + + +def test_hashes_manifest_covers_every_other_document(built) -> None: + _, dist = built + root = dist / "dictionaries/v1" + manifest = json.loads((root / "hashes.json").read_text(encoding="utf-8")) + assert manifest["algorithm"] == "sha256" + published = {path.relative_to(root).as_posix() for path in root.rglob("*.json")} + assert set(manifest["files"]) == published - {"hashes.json"} + assert all(len(digest) == 64 for digest in manifest["files"].values()) + + +def test_schemas_are_served_beside_the_data(built) -> None: + _, dist = built + assert (dist / "commentaries/v1/schema/commentary-chapter.json").is_file() + assert (dist / "dictionaries/v1/schema/dictionary-entry.json").is_file() + assert not (dist / "commentaries/v1/schema/dictionary-entry.json").exists() + + +def test_no_document_publishes_markup(built) -> None: + _, dist = built + for path in dist.rglob("*.json"): + assert '"html"' not in path.read_text(encoding="utf-8"), path diff --git a/tests/test_util.py b/tests/test_util.py index 6ecd2a2..9ea6fc1 100644 --- a/tests/test_util.py +++ b/tests/test_util.py @@ -1,6 +1,6 @@ import json -from study_builder.util import replace_tree, write_hash_sidecars, write_json +from study_builder.util import hash_tree, replace_tree, write_composed_json, write_json def test_json_writes_are_stable_and_hashed(tmp_path) -> None: @@ -9,9 +9,36 @@ def test_json_writes_are_stable_and_hashed(tmp_path) -> None: first = (root / "data.json").read_bytes() write_json(root / "data.json", {"hello": "world"}) assert (root / "data.json").read_bytes() == first - hashes = write_hash_sidecars(root) + hashes = hash_tree(root) assert len(hashes["data.json"]) == 64 - assert len((root / "data.json.sha").read_text().strip()) == 40 + assert not list(root.glob("*.sha")) + + +def test_hash_tree_can_exclude_its_own_manifest(tmp_path) -> None: + write_json(tmp_path / "data.json", {"hello": "world"}) + write_json(tmp_path / "hashes.json", {"files": {}}) + assert set(hash_tree(tmp_path, exclude={"hashes.json"})) == {"data.json"} + + +def test_composed_document_embeds_members_verbatim(tmp_path) -> None: + parts = [] + for index in (1, 2): + path = tmp_path / f"part{index}.json" + write_json(path, {"index": index, "nested": {"values": [index, index + 1]}}) + parts.append(path) + composed = tmp_path / "composed.json" + write_composed_json(composed, {"schema": "test", "name": "all"}, "members", parts) + + document = json.loads(composed.read_text(encoding="utf-8")) + assert document["schema"] == "test" + assert document["members"] == [json.loads(path.read_text(encoding="utf-8")) for path in parts] + assert composed.read_text(encoding="utf-8").endswith("]\n}\n") + + +def test_composed_document_handles_an_empty_member_list(tmp_path) -> None: + composed = tmp_path / "composed.json" + write_composed_json(composed, {"schema": "test"}, "members", []) + assert json.loads(composed.read_text(encoding="utf-8")) == {"schema": "test", "members": []} def test_replace_tree_preserves_only_new_generation(tmp_path) -> None: