...
" + "references": [{"osis": "Gen.1.1", "book": 1, "chapter": 1, "verse": 1}] } ] } ``` -Book and chapter introductions use chapter or verse `0`; they are not discarded. +Introductions are published, not discarded. A book introduction is chapter `0`, +so Clarke's introduction to Daniel is `clarke/27/0.json`. A chapter introduction +is verse `0`, and appears as the first entry of its own chapter document. + +`books.json` reports which books and chapters a commentary covers, and +`metadata.json` reports its licence, counts, and the byte size of the +whole-commentary document so a client can decide before requesting it. ## Dictionary API ```text GET https://dictionaries.getbible.net/v1/dictionaries.json +GET https://dictionaries.getbible.net/v1/{dictionary}.json GET https://dictionaries.getbible.net/v1/{dictionary}/metadata.json -GET https://dictionaries.getbible.net/v1/{dictionary}/keys.json +GET https://dictionaries.getbible.net/v1/{dictionary}/index.json GET https://dictionaries.getbible.net/v1/{dictionary}/{entry}.json -GET https://dictionaries.getbible.net/v1/{dictionary}/indexes/{sha256-prefix}.json ``` +Searching a dictionary takes two requests. `index.json` lists every word once, +sorted by an accent-insensitive lowercase `search` term, so a client can fetch it +once and then search, prefix-match, or binary-search entirely in memory: + +```json +{"id": "k-KADESH", "key": "KADESH", "search": "kadesh"} +``` + +The record's `id` is the path of the word itself — `{entry}.json` — so a hit in +the index resolves to exactly one document with no further lookup. + Strong's paths match Bible API v3 tokens directly: ```text @@ -133,17 +165,48 @@ H0430 -> https://dictionaries.getbible.net/v1/strongshebrew/H0430.json Greek keys use `G` plus the unpadded number; Hebrew keys use `H0` plus the unpadded number. Other dictionary keys receive deterministic, path-safe IDs. -`keys.json` maps source keys and aliases, while 256 SHA-256-prefix shards provide -smaller lookup indexes for constrained clients. + +Each word document carries the dictionary's own link graph, so a client can +navigate in either direction without rebuilding an index: + +```json +{ + "schema": "getbible-dictionary-entry-v1", + "dictionary": "easton", + "id": "k-KADESH", + "key": "KADESH", + "occurrence": 1, + "aliases": ["KADESH"], + "text": "Holy; a place in the wilderness of Zin.", + "see_also": [{"id": "k-MERIBAH", "key": "MERIBAH"}], + "backlinks": [{"id": "k-ZIN", "key": "ZIN"}], + "references": [{"osis": "Num.20.1", "book": 4, "chapter": 20, "verse": 1}] +} +``` + +`see_also` lists the words this entry points at and `backlinks` the words that +point back. Only targets that resolve to a real key in the same dictionary are +published. Scripture references stay in `references`, in the same shape the +commentary API uses. Some SWORD dictionaries legitimately contain more than one definition for the same public key. The first definition keeps the canonical direct path, and later definitions receive deterministic `--2`, `--3`, and subsequent suffixes. For example, Easton's repeated `KADESH` records are available as `k-KADESH.json` and -`k-KADESH--2.json`. Every definition appears in `keys.json` with an `occurrence` +`k-KADESH--2.json`. Every definition appears in `index.json` with an `occurrence` value. Dictionary metadata reports both the total `entry_count` and the distinct `unique_key_count`. +`{dictionary}.json` is the complete dictionary in index order, for offline +clients that would otherwise request every word individually. + +## Integrity and schemas + +Each API root publishes `hashes.json`, a SHA-256 digest of every other generated +document, which is also the manifest of the paths a build owns. The JSON Schemas +for every document type are served beside the data under `v1/schema/`, so each +schema `$id` resolves to the document that defines it. + ## Build flow ```mermaid @@ -152,9 +215,9 @@ flowchart TD B --> C["NDJSON v1 subprocess stream"] C --> D["Independent stream + artifact validator"] D --> E["Python API adapter + JSON Schema"] - E --> F["Atomic static v1 trees + hash sidecars"] - F --> G["v1_commentaries, when publication secrets exist"] - F --> H["v1_dictionaries, when publication secrets exist"] + E --> F["Atomic static v1 trees + SHA-256 manifest"] + F --> G["commentaries, when publication secrets exist"] + F --> H["dictionaries, when publication secrets exist"] ``` The static output is the system of record. Nginx and a CDN can serve direct @@ -232,8 +295,8 @@ Publication secret set: | `GETBIBLE_SSH_KEY` | SSH private key with write access to both outputs | | `GETBIBLE_SSH_PUB` | Matching public key | -The default output remotes are `getbible/v1_commentaries` and -`getbible/v1_dictionaries`. Optional `GETBIBLE_COMMENTARIES_REPO` and +The default output remotes are `getbible/commentaries` and +`getbible/dictionaries`. Optional `GETBIBLE_COMMENTARIES_REPO` and `GETBIBLE_DICTIONARIES_REPO` secrets may select staging remotes. ## Redistribution policy diff --git a/docs/nginx.conf b/docs/nginx.conf index bf1d4e8..a6a1491 100644 --- a/docs/nginx.conf +++ b/docs/nginx.conf @@ -1,30 +1,45 @@ # Add the relevant server block to each API virtual host. Set root to the -# checkout of v1_commentaries or v1_dictionaries respectively. +# checkout of getbible/commentaries or getbible/dictionaries respectively. +# +# The builder owns the v1/ directory of each checkout, so the public URL keeps an +# explicit version segment and a future v2/ can be served beside it unchanged. +# +# Generate the precompressed variants at deploy time rather than committing them; +# binary blobs delta poorly in Git and would grow both repositories every month: +# +# find v1 -name '*.json' -exec brotli -kf {} \; -exec gzip -kf9 {} \; server { - listen 443 ssl http2; + listen 443 ssl; + http2 on; server_name commentaries.getbible.net; - root /var/www/getbible/v1_commentaries; + root /var/www/getbible/commentaries; etag on; gzip on; - gzip_types application/json text/plain; + gzip_types application/json; + gzip_static on; + # brotli_static on; # requires ngx_brotli + # Chapter, book, and whole-commentary documents change only when the source + # module changes, so they are cached hard between monthly builds. location /v1/ { try_files $uri =404; add_header Access-Control-Allow-Origin "*" always; add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always; - add_header Cache-Control "public, max-age=300, stale-while-revalidate=86400" always; + add_header Cache-Control "public, max-age=86400, stale-while-revalidate=604800" always; } - location ~* \.sha$ { + # A regular expression location wins over the prefix above, so the three + # discovery documents stay short-lived while everything else does not. + location ~* ^/v1/(commentaries|build|hashes)\.json$ { try_files $uri =404; - default_type text/plain; add_header Access-Control-Allow-Origin "*" always; - add_header Cache-Control "public, max-age=31536000, immutable" always; + add_header Access-Control-Allow-Methods "GET, HEAD, OPTIONS" always; + add_header Cache-Control "public, max-age=300, stale-while-revalidate=86400" always; } } # Duplicate the block for dictionaries.getbible.net and set: -# root /var/www/getbible/v1_dictionaries; - +# root /var/www/getbible/dictionaries; +# and match (dictionaries|build|hashes)\.json for the discovery documents. diff --git a/docs/target-repositories.md b/docs/target-repositories.md index 4eae3d6..18da63b 100644 --- a/docs/target-repositories.md +++ b/docs/target-repositories.md @@ -1,13 +1,14 @@ # Target repository setup -Create two empty repositories with `main` as their default branch: +Create two repositories with `main` as their default branch: -- `getbible/v1_commentaries` -- `getbible/v1_dictionaries` +- `getbible/commentaries` +- `getbible/dictionaries` Seed each repository with a README and commit it before the first builder run. The builder owns only the `v1/` directory; repository documentation and server -configuration outside that directory are preserved. +configuration outside that directory are preserved, and a future `v2/` can be +published beside it without disturbing v1 consumers. Add these Actions secrets to `v1_study_builder`: diff --git a/pyproject.toml b/pyproject.toml index 6bfbbbe..40fb15b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,7 +11,6 @@ requires-python = ">=3.12" license = {text = "GPL-2.0-only"} authors = [{name = "GetBible", email = "github@vdm.io"}] dependencies = [ - "bleach>=6.2,<7", "jsonschema>=4.23,<5", ] diff --git a/requirements.txt b/requirements.txt index 30b3c20..41a1657 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,2 @@ -bleach>=6.2,<7 jsonschema>=4.23,<5 diff --git a/schemas/commentary-book.schema.json b/schemas/commentary-book.schema.json new file mode 100644 index 0000000..08dbe8d --- /dev/null +++ b/schemas/commentary-book.schema.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://commentaries.getbible.net/schema/v1/commentary-book.json", + "title": "GetBible Commentary Book v1", + "description": "Every chapter of one book of one commentary. Each member of chapters is byte-for-byte the document served at {commentary}/{book}/{chapter}.json.", + "type": "object", + "required": ["schema", "commentary", "language", "book", "name", "chapters"], + "properties": { + "schema": {"const": "getbible-commentary-book-v1"}, + "commentary": {"type": "string", "minLength": 1}, + "language": {"type": "string", "minLength": 2}, + "book": {"type": "integer", "minimum": 1, "maximum": 83}, + "name": {"type": "string", "minLength": 1}, + "chapters": { + "type": "array", + "items": {"$ref": "https://commentaries.getbible.net/schema/v1/commentary-chapter.json"} + } + }, + "additionalProperties": false +} diff --git a/schemas/commentary-books.schema.json b/schemas/commentary-books.schema.json new file mode 100644 index 0000000..482a74e --- /dev/null +++ b/schemas/commentary-books.schema.json @@ -0,0 +1,41 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://commentaries.getbible.net/schema/v1/commentary-books.json", + "title": "GetBible Commentary Books Index v1", + "description": "Which books and chapters one commentary covers. Chapter 0 is a book introduction.", + "type": "object", + "required": [ + "schema", + "commentary", + "language", + "name", + "book_url_template", + "chapter_url_template", + "book_count", + "books" + ], + "properties": { + "schema": {"const": "getbible-commentary-books-v1"}, + "commentary": {"type": "string", "minLength": 1}, + "language": {"type": "string", "minLength": 2}, + "name": {"type": "string", "minLength": 1}, + "book_url_template": {"const": "{book}.json"}, + "chapter_url_template": {"const": "{book}/{chapter}.json"}, + "book_count": {"type": "integer", "minimum": 0}, + "books": { + "type": "array", + "items": { + "type": "object", + "required": ["book", "name", "chapters", "entry_count"], + "properties": { + "book": {"type": "integer", "minimum": 1, "maximum": 83}, + "name": {"type": "string", "minLength": 1}, + "chapters": {"type": "array", "items": {"type": "integer", "minimum": 0}}, + "entry_count": {"type": "integer", "minimum": 0} + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false +} diff --git a/schemas/commentary-chapter.schema.json b/schemas/commentary-chapter.schema.json index eda1b6e..b63391b 100644 --- a/schemas/commentary-chapter.schema.json +++ b/schemas/commentary-chapter.schema.json @@ -2,6 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://commentaries.getbible.net/schema/v1/commentary-chapter.json", "title": "GetBible Commentary Chapter v1", + "description": "One chapter of one commentary. Chapter 0 carries the book introduction and verse 0 carries a chapter introduction.", "type": "object", "required": ["schema", "commentary", "language", "book", "name", "chapter", "entries"], "properties": { @@ -10,35 +11,51 @@ "language": {"type": "string", "minLength": 2}, "book": {"type": "integer", "minimum": 1, "maximum": 83}, "name": {"type": "string", "minLength": 1}, - "chapter": {"type": "integer", "minimum": 1}, + "chapter": {"type": "integer", "minimum": 0}, "entries": { + "type": "array", + "items": {"$ref": "#/$defs/entry"} + } + }, + "additionalProperties": false, + "$defs": { + "entry": { + "type": "object", + "required": ["book", "chapter", "verse", "name", "anchor", "text"], + "properties": { + "book": {"type": "integer", "minimum": 1, "maximum": 83}, + "chapter": {"type": "integer", "minimum": 0}, + "verse": {"type": "integer", "minimum": 0}, + "name": {"type": "string"}, + "anchor": { + "type": "object", + "required": ["book", "chapter", "verse"], + "properties": { + "book": {"type": "integer", "minimum": 1, "maximum": 83}, + "chapter": {"type": "integer", "minimum": 0}, + "verse": {"type": "integer", "minimum": 0}, + "osis": {"type": "string"} + }, + "additionalProperties": false + }, + "text": {"type": "string"}, + "references": {"$ref": "#/$defs/references"} + }, + "additionalProperties": false + }, + "references": { "type": "array", "items": { "type": "object", - "required": ["book", "chapter", "verse", "name", "anchor", "text"], + "required": ["osis", "book", "chapter"], "properties": { + "osis": {"type": "string", "minLength": 1}, "book": {"type": "integer", "minimum": 1, "maximum": 83}, - "chapter": {"type": "integer", "minimum": 1}, - "verse": {"type": "integer", "minimum": 0}, - "name": {"type": "string"}, - "anchor": { - "type": "object", - "required": ["book", "chapter", "verse"], - "properties": { - "book": {"type": "integer"}, - "chapter": {"type": "integer", "minimum": 1}, - "verse": {"type": "integer"}, - "osis": {"type": "string"} - }, - "additionalProperties": false - }, - "text": {"type": "string"}, - "html": {"type": "string"}, - "references": {"type": "array", "items": {"type": "object"}} + "chapter": {"type": "integer", "minimum": 0}, + "verse": {"type": "integer", "minimum": 0} }, "additionalProperties": false } } - }, - "additionalProperties": false + } } diff --git a/schemas/commentary.schema.json b/schemas/commentary.schema.json new file mode 100644 index 0000000..25f44cf --- /dev/null +++ b/schemas/commentary.schema.json @@ -0,0 +1,19 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://commentaries.getbible.net/schema/v1/commentary.json", + "title": "GetBible Commentary v1", + "description": "One complete commentary. Each member of books is byte-for-byte the document served at {commentary}/{book}.json. This is a bulk document; metadata.json publishes its size in bytes.", + "type": "object", + "required": ["schema", "commentary", "language", "name", "books"], + "properties": { + "schema": {"const": "getbible-commentary-v1"}, + "commentary": {"type": "string", "minLength": 1}, + "language": {"type": "string", "minLength": 2}, + "name": {"type": "string", "minLength": 1}, + "books": { + "type": "array", + "items": {"$ref": "https://commentaries.getbible.net/schema/v1/commentary-book.json"} + } + }, + "additionalProperties": false +} diff --git a/schemas/dictionary-entry.schema.json b/schemas/dictionary-entry.schema.json index aec1a25..50d1b15 100644 --- a/schemas/dictionary-entry.schema.json +++ b/schemas/dictionary-entry.schema.json @@ -2,6 +2,7 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://dictionaries.getbible.net/schema/v1/dictionary-entry.json", "title": "GetBible Dictionary Entry v1", + "description": "One word of one dictionary. see_also lists the words this entry points at; backlinks lists the words that point back.", "type": "object", "required": ["schema", "dictionary", "language", "id", "key", "occurrence", "aliases", "text"], "properties": { @@ -13,8 +14,36 @@ "occurrence": {"type": "integer", "minimum": 1}, "aliases": {"type": "array", "items": {"type": "string"}, "minItems": 1}, "text": {"type": "string"}, - "html": {"type": "string"}, - "references": {"type": "array", "items": {"type": "string"}} + "see_also": {"$ref": "#/$defs/links"}, + "backlinks": {"$ref": "#/$defs/links"}, + "references": { + "type": "array", + "items": { + "type": "object", + "required": ["osis", "book", "chapter"], + "properties": { + "osis": {"type": "string", "minLength": 1}, + "book": {"type": "integer", "minimum": 1, "maximum": 83}, + "chapter": {"type": "integer", "minimum": 0}, + "verse": {"type": "integer", "minimum": 0} + }, + "additionalProperties": false + } + } }, - "additionalProperties": false + "additionalProperties": false, + "$defs": { + "links": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "key"], + "properties": { + "id": {"type": "string", "minLength": 1}, + "key": {"type": "string", "minLength": 1} + }, + "additionalProperties": false + } + } + } } diff --git a/schemas/dictionary-index.schema.json b/schemas/dictionary-index.schema.json new file mode 100644 index 0000000..6dbab95 --- /dev/null +++ b/schemas/dictionary-index.schema.json @@ -0,0 +1,42 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://dictionaries.getbible.net/schema/v1/dictionary-index.json", + "title": "GetBible Dictionary Index v1", + "description": "Every word in one dictionary, sorted by the accent-insensitive lowercase search term. One fetch is enough to search a dictionary in any direction; the word itself is then at entry_url_template with the record's id.", + "type": "object", + "required": [ + "schema", + "dictionary", + "language", + "name", + "entry_url_template", + "entry_count", + "unique_key_count", + "entries" + ], + "properties": { + "schema": {"const": "getbible-dictionary-index-v1"}, + "dictionary": {"type": "string", "minLength": 1}, + "language": {"type": "string", "minLength": 2}, + "name": {"type": "string", "minLength": 1}, + "entry_url_template": {"const": "{entry}.json"}, + "entry_count": {"type": "integer", "minimum": 0}, + "unique_key_count": {"type": "integer", "minimum": 0}, + "entries": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "key", "search"], + "properties": { + "id": {"type": "string", "minLength": 1}, + "key": {"type": "string", "minLength": 1}, + "search": {"type": "string", "minLength": 1}, + "aliases": {"type": "array", "items": {"type": "string"}, "minItems": 1}, + "occurrence": {"type": "integer", "minimum": 2} + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false +} diff --git a/schemas/dictionary.schema.json b/schemas/dictionary.schema.json new file mode 100644 index 0000000..82dbd1e --- /dev/null +++ b/schemas/dictionary.schema.json @@ -0,0 +1,19 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://dictionaries.getbible.net/schema/v1/dictionary.json", + "title": "GetBible Dictionary v1", + "description": "One complete dictionary in index order. Each member of entries is byte-for-byte the document served at {dictionary}/{entry}.json. This is a bulk document for offline clients; metadata.json publishes its size in bytes.", + "type": "object", + "required": ["schema", "dictionary", "language", "name", "entries"], + "properties": { + "schema": {"const": "getbible-dictionary-v1"}, + "dictionary": {"type": "string", "minLength": 1}, + "language": {"type": "string", "minLength": 2}, + "name": {"type": "string", "minLength": 1}, + "entries": { + "type": "array", + "items": {"$ref": "https://dictionaries.getbible.net/schema/v1/dictionary-entry.json"} + } + }, + "additionalProperties": false +} diff --git a/scripts/validate_build.py b/scripts/validate_build.py index edacca0..adb7758 100644 --- a/scripts/validate_build.py +++ b/scripts/validate_build.py @@ -10,42 +10,104 @@ from study_builder.util import read_json, slug -def validate_commentary(root: Path) -> dict[str, Any]: +def _reject_markup(document: Any, where: str) -> None: + if isinstance(document, dict): + if "html" in document: + raise RuntimeError(f"{where} still publishes an html member") + for key, value in document.items(): + _reject_markup(value, f"{where}.{key}") + elif isinstance(document, list): + for index, value in enumerate(document): + _reject_markup(value, f"{where}[{index}]") + + +def _assert_composed(composed: list[Any], parts: list[Path], where: str) -> None: + """A composed document must contain its parts exactly as they are served alone.""" + if len(composed) != len(parts): + raise RuntimeError(f"{where} holds {len(composed)} members for {len(parts)} documents") + for member, path in zip(composed, parts, strict=True): + if member != read_json(path): + raise RuntimeError(f"{where} does not match the document served at {path}") + + +def validate_commentary(root: Path, complete_path: Path) -> dict[str, Any]: metadata = read_json(root / "metadata.json") books = read_json(root / "books.json") if metadata.get("schema") != "getbible-commentary-metadata-v1": raise RuntimeError("Unexpected commentary metadata schema") - if int(metadata.get("entry_count", 0)) <= 0 or not books: + if books.get("schema") != "getbible-commentary-books-v1": + raise RuntimeError("Unexpected commentary books index schema") + if int(metadata.get("entry_count", 0)) <= 0 or not books.get("books"): raise RuntimeError("Commentary produced no addressable entries") - first_book = books[0] - book_index = read_json(root / str(first_book["url"])) - if not book_index.get("chapters"): - raise RuntimeError("Commentary book index produced no chapters") - chapter = read_json(root / str(book_index["chapters"][0]["url"])) - if chapter.get("schema") != "getbible-commentary-chapter-v1": - raise RuntimeError("Unexpected commentary chapter schema") - if not chapter.get("entries"): + + first_book = books["books"][0] + book_path = root / f"{first_book['book']}.json" + book = read_json(book_path) + if book.get("schema") != "getbible-commentary-book-v1" or not book.get("chapters"): + raise RuntimeError("Commentary book document produced no chapters") + chapter_paths = [ + root / str(first_book["book"]) / f"{number}.json" for number in first_book["chapters"] + ] + _assert_composed(book["chapters"], chapter_paths, f"{book_path}.chapters") + + chapter = read_json(chapter_paths[0]) + if chapter.get("schema") != "getbible-commentary-chapter-v1" or not chapter.get("entries"): raise RuntimeError("Commentary chapter produced no entries") first = chapter["entries"][0] if not all(name in first for name in ("book", "chapter", "verse", "anchor", "text")): raise RuntimeError("Commentary entry is not linked to a Bible API coordinate") - return {"entries": metadata["entry_count"], "books": metadata["book_count"]} + _reject_markup(chapter, "chapter") + complete = read_json(complete_path) + if complete.get("schema") != "getbible-commentary-v1": + raise RuntimeError("Unexpected whole-commentary schema") + book_paths = [root / f"{record['book']}.json" for record in books["books"]] + _assert_composed(complete["books"], book_paths, f"{complete_path}.books") -def validate_dictionary(root: Path) -> dict[str, Any]: + return { + "books": metadata["book_count"], + "chapters": metadata["chapter_count"], + "entries": metadata["entry_count"], + "bytes": metadata["bytes"], + "introductions": sum(1 for record in books["books"] if 0 in record["chapters"]), + } + + +def validate_dictionary(root: Path, complete_path: Path) -> dict[str, Any]: metadata = read_json(root / "metadata.json") - keys = read_json(root / "keys.json") + index = read_json(root / "index.json") if metadata.get("schema") != "getbible-dictionary-metadata-v1": raise RuntimeError("Unexpected dictionary metadata schema") - if int(metadata.get("entry_count", 0)) <= 0 or not keys: + if index.get("schema") != "getbible-dictionary-index-v1": + raise RuntimeError("Unexpected dictionary index schema") + if int(metadata.get("entry_count", 0)) <= 0 or not index.get("entries"): raise RuntimeError("Dictionary produced no addressable entries") - first = keys[0] - document = read_json(root / str(first["url"])) + + terms = [record["search"] for record in index["entries"]] + if terms != sorted(terms): + raise RuntimeError("Dictionary index is not sorted by its search term") + + entry_paths = [root / f"{record['id']}.json" for record in index["entries"]] + document = read_json(entry_paths[0]) if document.get("schema") != "getbible-dictionary-entry-v1": raise RuntimeError("Unexpected dictionary entry schema") if not all(name in document for name in ("dictionary", "id", "key", "aliases", "text")): raise RuntimeError("Dictionary entry is missing its lookup contract") - return {"entries": metadata["entry_count"], "strong_prefix": metadata["strong_prefix"]} + _reject_markup(document, "entry") + + complete = read_json(complete_path) + if complete.get("schema") != "getbible-dictionary-v1": + raise RuntimeError("Unexpected whole-dictionary schema") + _assert_composed(complete["entries"], entry_paths, f"{complete_path}.entries") + + linked = sum(1 for entry in complete["entries"] if entry.get("see_also")) + return { + "entries": metadata["entry_count"], + "unique_keys": metadata["unique_key_count"], + "strong_prefix": metadata["strong_prefix"], + "bytes": metadata["bytes"], + "entries_with_links": linked, + } def main() -> int: @@ -54,9 +116,14 @@ def main() -> int: parser.add_argument("--module", required=True) parser.add_argument("--dist-dir", type=Path, default=Path("dist")) args = parser.parse_args() - root = args.dist_dir / args.resource / "v1" / slug(args.module) + module_id = slug(args.module) + version_root = args.dist_dir / args.resource / "v1" + root = version_root / module_id + complete_path = version_root / f"{module_id}.json" result = ( - validate_commentary(root) if args.resource == "commentaries" else validate_dictionary(root) + validate_commentary(root, complete_path) + if args.resource == "commentaries" + else validate_dictionary(root, complete_path) ) print(json.dumps({"resource": args.resource, "module": args.module, **result}, indent=2)) return 0 diff --git a/src/study_builder/cli.py b/src/study_builder/cli.py index 843a9cc..6f91e1f 100644 --- a/src/study_builder/cli.py +++ b/src/study_builder/cli.py @@ -55,14 +55,14 @@ def parser() -> argparse.ArgumentParser: "--commentaries-repo", default=os.environ.get( "STUDY_BUILDER_COMMENTARIES_REPO", - "git@github.com:getbible/v1_commentaries.git", + "git@github.com:getbible/commentaries.git", ), ) build.add_argument( "--dictionaries-repo", default=os.environ.get( "STUDY_BUILDER_DICTIONARIES_REPO", - "git@github.com:getbible/v1_dictionaries.git", + "git@github.com:getbible/dictionaries.git", ), ) build.add_argument("--commentaries-branch", default="main") diff --git a/src/study_builder/commentaries.py b/src/study_builder/commentaries.py index 8617c88..517e29a 100644 --- a/src/study_builder/commentaries.py +++ b/src/study_builder/commentaries.py @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0-only from __future__ import annotations from collections import defaultdict @@ -11,80 +12,42 @@ from study_builder.chapter_spool import CommentaryChapterSpool from study_builder.content import extract_osis_references, public_content from study_builder.models import ModuleDescriptor, NativeExport -from study_builder.util import read_json, slug, write_json +from study_builder.util import read_json, slug, write_composed_json, write_json + +# Reserved inside a commentary directory; an entry may never claim these names. +RESERVED_DOCUMENTS = {"metadata.json", "books.json"} class CommentaryWriter: - def __init__(self, root: Path, books: BookRegistry, schema_path: Path) -> None: + """Write the chapter, book, and whole-commentary documents for one module. + + Chapter documents are the addressable unit. Book and commentary documents embed + them byte-for-byte, so `book.chapters[n]` is exactly the chapter document served + at its own path and a client needs only one parser for all three levels. + """ + + def __init__(self, root: Path, books: BookRegistry, schemas_dir: Path) -> None: self.root = root self.books = books - self.schema = read_json(schema_path) + self.chapter_schema = read_json(schemas_dir / "commentary-chapter.schema.json") - def write(self, module: ModuleDescriptor, exported: NativeExport) -> dict[str, Any]: + def write(self, module: ModuleDescriptor, exported: NativeExport) -> tuple[dict, dict]: module_id = slug(module.name) module_root = self.root / module_id - chapter_indexes: dict[int, list[dict[str, Any]]] = defaultdict(list) + chapter_files: dict[int, list[Path]] = defaultdict(list) + chapter_counts: dict[int, list[tuple[int, int]]] = defaultdict(list) entry_count = 0 with CommentaryChapterSpool() as chapters: for source in exported.entries: - verse = source.get("verse") or {} - chapter = int(verse.get("chapter", 0) or 0) - verse_number = int(verse.get("verse", 0) or 0) - # The public commentary API is chapter-addressable. SWORD modules may - # also expose book introductions with chapter zero; those records do - # not have a chapter endpoint and intentionally remain unpublished. - if chapter <= 0 or verse_number < 0: - continue - try: - book = self.books.from_entry(source) - except ValueError: - continue - content = public_content(source) - if not content.get("text") and not content.get("html"): - continue - osis = str(verse.get("osis", "")) - related = [] - for reference in extract_osis_references( - str(source.get("raw", "")), str(source.get("html", "")) - ): - normalized = self.books.reference(reference) - if normalized: - related.append(normalized) - anchor = { - "book": book.number, - "chapter": chapter, - "verse": verse_number, - } - if osis: - anchor["osis"] = osis - label = book.name - if chapter: - label += f" {chapter}" - if verse_number: - label += f":{verse_number}" - entry: dict[str, Any] = { - "book": book.number, - "chapter": chapter, - "verse": verse_number, - "name": label, - "anchor": anchor, - **content, - } - if related: - entry["references"] = related - chapters.append(entry) + entry = self._entry(source) + if entry is not None: + chapters.append(entry) for book_number, chapter_number in chapters.coordinates(): - chapter_seen: set[tuple[int, str]] = set() - chapter_entries: list[dict[str, Any]] = [] - for entry in chapters.entries(book_number, chapter_number): - unique = (int(entry["verse"]), str(entry.get("text", ""))) - if unique in chapter_seen: - continue - chapter_seen.add(unique) - chapter_entries.append(entry) - chapter_entries.sort(key=lambda item: (item["verse"], item["name"])) + chapter_entries = self._chapter_entries(chapters, book_number, chapter_number) + if not chapter_entries: + continue book = self.books.by_number[book_number] document = { "schema": "getbible-commentary-chapter-v1", @@ -95,44 +58,159 @@ def write(self, module: ModuleDescriptor, exported: NativeExport) -> dict[str, A "chapter": chapter_number, "entries": chapter_entries, } - validate(document, self.schema) - write_json(module_root / str(book_number) / f"{chapter_number}.json", document) - chapter_indexes[book_number].append( - { - "chapter": chapter_number, - "entry_count": len(chapter_entries), - "url": f"{book_number}/{chapter_number}.json", - } - ) + validate(document, self.chapter_schema) + path = module_root / str(book_number) / f"{chapter_number}.json" + write_json(path, document) + chapter_files[book_number].append(path) + chapter_counts[book_number].append((chapter_number, len(chapter_entries))) entry_count += len(chapter_entries) + book_files: list[Path] = [] books_index: list[dict[str, Any]] = [] - for book_number in sorted(chapter_indexes): + for book_number in sorted(chapter_files): book = self.books.by_number[book_number] - chapter_index = chapter_indexes[book_number] - chapter_numbers = [record["chapter"] for record in chapter_index] - write_json( - module_root / f"{book_number}.json", + path = module_root / f"{book_number}.json" + if path.name in RESERVED_DOCUMENTS: + raise RuntimeError(f"Book document collides with a reserved name: {path.name}") + write_composed_json( + path, { - "schema": "getbible-commentary-book-index-v1", + "schema": "getbible-commentary-book-v1", "commentary": module_id, "language": module.language, "book": book_number, "name": book.name, - "chapters": chapter_index, }, + "chapters", + chapter_files[book_number], ) + book_files.append(path) books_index.append( { "book": book_number, "name": book.name, - "chapters": chapter_numbers, - "url": f"{book_number}.json", - "chapter_url_template": f"{book_number}/{{chapter}}.json", + "chapters": [number for number, _ in chapter_counts[book_number]], + "entry_count": sum(count for _, count in chapter_counts[book_number]), } ) - metadata = { + write_json( + module_root / "books.json", + { + "schema": "getbible-commentary-books-v1", + "commentary": module_id, + "language": module.language, + "name": module.description, + "book_url_template": "{book}.json", + "chapter_url_template": "{book}/{chapter}.json", + "book_count": len(books_index), + "books": books_index, + }, + ) + + complete = self.root / f"{module_id}.json" + write_composed_json( + complete, + { + "schema": "getbible-commentary-v1", + "commentary": module_id, + "language": module.language, + "name": module.description, + }, + "books", + book_files, + ) + + chapter_count = sum(len(records) for records in chapter_counts.values()) + metadata = self._metadata( + module, module_id, len(books_index), chapter_count, entry_count, complete.stat().st_size + ) + write_json(module_root / "metadata.json", metadata) + record = { + "id": module_id, + "name": module.description, + "language": module.language, + "license": module.license, + "book_count": len(books_index), + "chapter_count": chapter_count, + "entry_count": entry_count, + "bytes": metadata["bytes"], + } + return record, metadata + + def _entry(self, source: dict[str, Any]) -> dict[str, Any] | None: + verse = source.get("verse") or {} + chapter = int(verse.get("chapter", 0) or 0) + verse_number = int(verse.get("verse", 0) or 0) + # Chapter zero carries a book introduction and verse zero a chapter + # introduction. Both are published: chapter zero at {book}/0.json, and + # verse zero as the first entry of its chapter. + if chapter < 0 or verse_number < 0: + return None + try: + book = self.books.from_entry(source) + except ValueError: + return None + content = public_content(source) + if not content["text"]: + return None + label = book.name + if chapter: + label += f" {chapter}" + if verse_number: + label += f":{verse_number}" + anchor: dict[str, Any] = { + "book": book.number, + "chapter": chapter, + "verse": verse_number, + } + osis = str(verse.get("osis", "")) + if osis: + anchor["osis"] = osis + entry: dict[str, Any] = { + "book": book.number, + "chapter": chapter, + "verse": verse_number, + "name": label, + "anchor": anchor, + **content, + } + related = [] + for reference in extract_osis_references( + str(source.get("raw", "")), str(source.get("html", "")) + ): + normalized = self.books.reference(reference) + if normalized: + related.append(normalized) + if related: + entry["references"] = related + return entry + + @staticmethod + def _chapter_entries( + chapters: CommentaryChapterSpool, book_number: int, chapter_number: int + ) -> list[dict[str, Any]]: + seen: set[tuple[int, str]] = set() + collected: list[dict[str, Any]] = [] + for entry in chapters.entries(book_number, chapter_number): + unique = (int(entry["verse"]), str(entry.get("text", ""))) + if unique in seen: + continue + seen.add(unique) + collected.append(entry) + collected.sort(key=lambda item: (item["verse"], item["name"])) + return collected + + @staticmethod + def _metadata( + module: ModuleDescriptor, + module_id: str, + book_count: int, + chapter_count: int, + entry_count: int, + complete_bytes: int, + ) -> dict[str, Any]: + return { "schema": "getbible-commentary-metadata-v1", "id": module_id, "module": module.name, @@ -143,9 +221,12 @@ def write(self, module: ModuleDescriptor, exported: NativeExport) -> dict[str, A "driver": module.driver, "source_type": module.first("sourcetype"), "versification": module.first("versification", "KJV"), + "book_count": book_count, + "chapter_count": chapter_count, "entry_count": entry_count, - "book_count": len(books_index), + "bytes": complete_bytes, "books_url": "books.json", + "book_url_template": "{book}.json", "chapter_url_template": "{book}/{chapter}.json", "source": "CrossWire SWORD", "source_module_url": ( @@ -166,6 +247,3 @@ def write(self, module: ModuleDescriptor, exported: NativeExport) -> dict[str, A "Converted to GetBible static JSON; wording is supplied by the source module." ), } - write_json(module_root / "metadata.json", metadata) - write_json(module_root / "books.json", books_index) - return metadata diff --git a/src/study_builder/content.py b/src/study_builder/content.py index db8d796..ee993d2 100644 --- a/src/study_builder/content.py +++ b/src/study_builder/content.py @@ -1,22 +1,21 @@ +# SPDX-License-Identifier: GPL-2.0-only from __future__ import annotations import html import re +from html.parser import HTMLParser from typing import Any -import bleach - -ALLOWED_TAGS = { - "a", - "b", +# The public API publishes plain text only. Markup is never republished, so the +# builder needs no HTML sanitizer and the generated API carries no markup that a +# consuming application could inject into a page. +_SUPPRESSED_TAGS = {"script", "style"} +_BREAK_TAGS = { "blockquote", "br", - "code", "dd", "div", - "dl", "dt", - "em", "h1", "h2", "h3", @@ -24,28 +23,13 @@ "h5", "h6", "hr", - "i", "li", "ol", "p", - "span", - "strong", - "sub", - "sup", "table", - "tbody", - "td", - "th", - "thead", "tr", - "u", "ul", } -ALLOWED_ATTRIBUTES = { - "a": ["href", "title"], - "*": ["class", "dir", "lang", "title"], -} -ALLOWED_PROTOCOLS = {"http", "https", "mailto", "sword"} _OSIS_REF = re.compile( r"(?PA comment on creation.
", - "verse": { - "osis": "Gen.1.1", - "testament": 1, - "book": 1, - "chapter": 1, - "verse": 1, - }, - } - ], - ) +def entry(osis, book, chapter, verse, text, *, raw=None, html=""): + return { + "record_type": "entry", + "key": osis, + "raw": raw if raw is not None else text, + "plain": text, + "html": html, + "verse": { + "osis": osis, + "testament": 1 if book <= 39 else 2, + "book": book if book <= 39 else book - 39, + "chapter": chapter, + "verse": verse, + }, + } + + +def write(tmp_path, project_root, module, entries): writer = CommentaryWriter( tmp_path, BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", + project_root / "schemas", ) - summary = writer.write(commentary_module, export) - chapter_path = tmp_path / "testcom/1/1.json" - chapter = json.loads(chapter_path.read_text(encoding="utf-8")) - assert summary["chapter_url_template"] == "{book}/{chapter}.json" - assert (chapter["book"], chapter["chapter"]) == (1, 1) - assert chapter["entries"][0]["verse"] == 1 - assert chapter["entries"][0]["anchor"]["osis"] == "Gen.1.1" - assert chapter["entries"][0]["references"][0]["book"] == 43 + return writer.write(module, NativeExport(metadata={"record_type": "module"}, entries=entries)) -def test_commentary_consumes_source_entries_once(tmp_path, project_root, commentary_module) -> None: - entries = OnePassEntries( +def test_commentary_matches_v3_book_chapter_verse_contract( + tmp_path, project_root, commentary_module +) -> None: + record, metadata = write( + tmp_path, + project_root, + commentary_module, [ - { - "key": "Genesis 1:1", - "raw": "First chapter", - "plain": "First chapter", - "html": "", - "verse": { - "osis": "Gen.1.1", - "testament": 1, - "book": 1, - "chapter": 1, - "verse": 1, - }, - }, - { - "key": "Genesis 2:1", - "raw": "Second chapter", - "plain": "Second chapter", - "html": "", - "verse": { - "osis": "Gen.2.1", - "testament": 1, - "book": 1, - "chapter": 2, - "verse": 1, - }, - }, - ] + entry( + "John.1.1", + 43, + 1, + 1, + "A comment on creation.", + raw='A comment on creation.
", + ) + ], ) - export = NativeExport(metadata={}, entries=entries) + chapter = json.loads((tmp_path / "testcom/43/1.json").read_text(encoding="utf-8")) + assert metadata["chapter_url_template"] == "{book}/{chapter}.json" + assert (chapter["book"], chapter["chapter"]) == (43, 1) + assert chapter["entries"][0]["verse"] == 1 + assert chapter["entries"][0]["anchor"]["osis"] == "John.1.1" + assert chapter["entries"][0]["references"][0]["book"] == 43 + assert record["entry_count"] == 1 - summary = CommentaryWriter( - tmp_path, - BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", - ).write(commentary_module, export) - assert entries.iterated - assert summary["entry_count"] == 2 - assert (tmp_path / "testcom/1/1.json").is_file() - assert (tmp_path / "testcom/1/2.json").is_file() +def test_commentary_publishes_text_without_markup( + tmp_path, project_root, commentary_module +) -> None: + write( + tmp_path, + project_root, + commentary_module, + [entry("Gen.1.1", 1, 1, 1, "Plain words.", html="Plain words.
")], + ) + chapter = json.loads((tmp_path / "testcom/1/1.json").read_text(encoding="utf-8")) + published = chapter["entries"][0] + assert published["text"] == "Plain words." + assert "html" not in published + assert "html" not in (tmp_path / "testcom/1/1.json").read_text(encoding="utf-8") -def test_commentary_skips_entries_without_a_scripture_chapter( +def test_commentary_publishes_book_and_chapter_introductions( tmp_path, project_root, commentary_module ) -> None: - export = NativeExport( - metadata={}, - entries=[ - { - "key": "Genesis", - "raw": "Book introduction", - "plain": "Book introduction", - "html": "", - "verse": { - "osis": "Gen", - "testament": 1, - "book": 1, - "chapter": 0, - "verse": 0, - }, - }, - { - "key": "Genesis 1", - "raw": "Chapter introduction", - "plain": "Chapter introduction", - "html": "", - "verse": { - "osis": "Gen.1", - "testament": 1, - "book": 1, - "chapter": 1, - "verse": 0, - }, - }, + write( + tmp_path, + project_root, + commentary_module, + [ + entry("Dan.0.0", 27, 0, 0, "About the book of Daniel."), + entry("Dan.1.0", 27, 1, 0, "About chapter one."), + entry("Dan.1.1", 27, 1, 1, "On the first verse."), ], ) + introduction = json.loads((tmp_path / "testcom/27/0.json").read_text(encoding="utf-8")) + assert introduction["chapter"] == 0 + assert introduction["entries"][0]["name"] == "Daniel" + assert introduction["entries"][0]["text"] == "About the book of Daniel." - summary = CommentaryWriter( - tmp_path, - BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", - ).write(commentary_module, export) + chapter = json.loads((tmp_path / "testcom/27/1.json").read_text(encoding="utf-8")) + assert [item["verse"] for item in chapter["entries"]] == [0, 1] + assert chapter["entries"][0]["name"] == "Daniel 1" - chapter = json.loads((tmp_path / "testcom/1/1.json").read_text(encoding="utf-8")) - assert summary["entry_count"] == 1 - assert not (tmp_path / "testcom/1/0.json").exists() - assert [entry["name"] for entry in chapter["entries"]] == ["Genesis 1"] + books = json.loads((tmp_path / "testcom/books.json").read_text(encoding="utf-8")) + assert books["books"][0]["chapters"] == [0, 1] -def test_commentary_canonicalizes_interleaved_source_chapters( +def test_book_and_commentary_documents_embed_their_parts_verbatim( tmp_path, project_root, commentary_module ) -> None: - def source(osis: str, testament: int, book: int, chapter: int, verse: int, text: str): - return { - "key": osis, - "raw": text, - "plain": text, - "html": "", - "verse": { - "osis": osis, - "testament": testament, - "book": book, - "chapter": chapter, - "verse": verse, - }, - } + write( + tmp_path, + project_root, + commentary_module, + [ + entry("Gen.1.1", 1, 1, 1, "First."), + entry("Gen.2.1", 1, 2, 1, "Second."), + entry("John.1.1", 43, 1, 1, "Third."), + ], + ) + chapters = [ + json.loads((tmp_path / f"testcom/1/{number}.json").read_text(encoding="utf-8")) + for number in (1, 2) + ] + book = json.loads((tmp_path / "testcom/1.json").read_text(encoding="utf-8")) + assert book["schema"] == "getbible-commentary-book-v1" + assert book["chapters"] == chapters + + complete = json.loads((tmp_path / "testcom.json").read_text(encoding="utf-8")) + books = [ + json.loads((tmp_path / f"testcom/{number}.json").read_text(encoding="utf-8")) + for number in (1, 43) + ] + assert complete["schema"] == "getbible-commentary-v1" + assert complete["books"] == books + + +def test_commentary_reports_counts_and_bulk_size(tmp_path, project_root, commentary_module) -> None: + record, metadata = write( + tmp_path, + project_root, + commentary_module, + [ + entry("Gen.1.1", 1, 1, 1, "First."), + entry("Gen.2.1", 1, 2, 1, "Second."), + entry("John.1.1", 43, 1, 1, "Third."), + ], + ) + assert (record["book_count"], record["chapter_count"], record["entry_count"]) == (2, 3, 3) + assert record["bytes"] == (tmp_path / "testcom.json").stat().st_size + assert metadata["bytes"] == record["bytes"] + +def test_commentary_consumes_source_entries_once(tmp_path, project_root, commentary_module) -> None: entries = OnePassEntries( [ - source("2Macc.7.1", 1, 48, 7, 1, "Maccabees first"), - source("Job.2.1", 1, 18, 2, 1, "Job second chapter"), - source("Job.1.2", 1, 18, 1, 2, "Job second verse"), - source("2Macc.7.1", 1, 48, 7, 1, "Maccabees first"), - source("Job.1.1", 1, 18, 1, 1, "Job first verse"), + entry("Gen.1.1", 1, 1, 1, "First chapter"), + entry("Gen.2.1", 1, 2, 1, "Second chapter"), ] ) - export = NativeExport(metadata={}, entries=entries) + write(tmp_path, project_root, commentary_module, entries) + assert entries.passes == 1 + assert (tmp_path / "testcom/1/1.json").is_file() + assert (tmp_path / "testcom/1/2.json").is_file() - summary = CommentaryWriter( - tmp_path, - BookRegistry(project_root / "conf/book_registry.json"), - project_root / "schemas/commentary-chapter.schema.json", - ).write(commentary_module, export) - books = json.loads((tmp_path / "testcom/books.json").read_text(encoding="utf-8")) - job = json.loads((tmp_path / "testcom/18.json").read_text(encoding="utf-8")) - job_one = json.loads((tmp_path / "testcom/18/1.json").read_text(encoding="utf-8")) - maccabees_seven = json.loads((tmp_path / "testcom/81/7.json").read_text(encoding="utf-8")) - - assert entries.iterated - assert summary["entry_count"] == 4 - assert [item["book"] for item in books] == [18, 81] - assert [item["chapter"] for item in job["chapters"]] == [1, 2] - assert [item["verse"] for item in job_one["entries"]] == [1, 2] - assert list(job_one["entries"][0]) == ["book", "chapter", "verse", "name", "anchor", "text"] - assert len(maccabees_seven["entries"]) == 1 +def test_commentary_skips_entries_without_a_bible_coordinate( + tmp_path, project_root, commentary_module +) -> None: + record, _ = write( + tmp_path, + project_root, + commentary_module, + [ + {"key": "Preface", "raw": "front matter", "plain": "front matter", "html": ""}, + entry("Gen.1.1", 1, 1, 1, "Kept."), + ], + ) + assert record["entry_count"] == 1 diff --git a/tests/test_content.py b/tests/test_content.py index 5e0c19a..a6e81d2 100644 --- a/tests/test_content.py +++ b/tests/test_content.py @@ -1,21 +1,32 @@ -from study_builder.content import clean_html, extract_osis_references, public_content +from study_builder.content import extract_osis_references, public_content, strip_markup -def test_html_sanitizer_removes_scripts_and_unsafe_links() -> None: - cleaned = clean_html( - "Hello world
" - 'bad' +def test_markup_stripper_drops_scripts_and_keeps_readable_text() -> None: + text = strip_markup( + 'Safe text
' + 'link' ) - assert "A word
"}) + assert content == {"text": "A word"} -def test_structural_html_without_text_is_not_public_content() -> None: +def test_public_content_falls_back_to_markup_when_stripped_text_is_empty() -> None: + content = public_content({"plain": "", "html": "Only in the rendered form
"}) + assert content == {"text": "Only in the rendered form"} + + +def test_structural_markup_without_text_is_not_public_content() -> None: assert public_content({"plain": "", "html": 'logos: a word
", } ], + metadata={"feature": "GreekDef"}, ) - summary = DictionaryWriter( - tmp_path, project_root / "schemas/dictionary-entry.schema.json" - ).write(greek_dictionary_module, export) - path = tmp_path / "strongsgreek/G3056.json" - document = json.loads(path.read_text(encoding="utf-8")) - assert summary["strong_prefix"] == "G" + document = json.loads((tmp_path / "strongsgreek/G3056.json").read_text(encoding="utf-8")) + assert record["strong_prefix"] == "G" assert document["id"] == "G3056" assert document["occurrence"] == 1 assert document["aliases"] == ["03056", "G3056"] - assert document["references"] == ["John.1.1"] + assert document["text"] == "logos: a word" + assert "html" not in document + assert document["references"] == [{"osis": "John.1.1", "book": 43, "chapter": 1, "verse": 1}] + assert metadata["index_url"] == "index.json" + + +def test_dictionary_index_is_sorted_and_slim( + tmp_path, project_root, greek_dictionary_module +) -> None: + write( + tmp_path, + project_root, + greek_dictionary_module, + [ + {"key": "Zeta", "raw": "last", "plain": "last", "html": ""}, + {"key": "Alpha", "raw": "first", "plain": "first", "html": ""}, + ], + ) + index = json.loads((tmp_path / "strongsgreek/index.json").read_text(encoding="utf-8")) + assert index["schema"] == "getbible-dictionary-index-v1" + assert [record["key"] for record in index["entries"]] == ["Alpha", "Zeta"] + assert index["entries"][0] == {"id": "k-Alpha", "key": "Alpha", "search": "alpha"} + assert index["entry_url_template"] == "{entry}.json" + + +def test_dictionary_links_resolve_in_both_directions( + tmp_path, project_root, greek_dictionary_module +) -> None: + write( + tmp_path, + project_root, + greek_dictionary_module, + [ + { + "key": "KADESH", + "raw": 'also Meribah', + "plain": "A place in the wilderness.", + "html": "", + }, + { + "key": "MERIBAH", + "raw": "Waters of strife.", + "plain": "Waters of strife.", + "html": "", + }, + ], + ) + kadesh = json.loads((tmp_path / "strongsgreek/k-KADESH.json").read_text(encoding="utf-8")) + meribah = json.loads((tmp_path / "strongsgreek/k-MERIBAH.json").read_text(encoding="utf-8")) + assert kadesh["see_also"] == [{"id": "k-MERIBAH", "key": "MERIBAH"}] + assert "backlinks" not in kadesh + assert meribah["backlinks"] == [{"id": "k-KADESH", "key": "KADESH"}] + assert "see_also" not in meribah def test_dictionary_preserves_repeated_keys_as_distinct_definitions( tmp_path, project_root, greek_dictionary_module ) -> None: - export = NativeExport( - metadata={"feature": "GreekDef"}, - entries=[ + record, _ = write( + tmp_path, + project_root, + greek_dictionary_module, + [ {"key": "03056", "raw": "First", "plain": "First", "html": ""}, {"key": "03056", "raw": "Second", "plain": "Second", "html": ""}, ], + metadata={"feature": "GreekDef"}, ) - summary = DictionaryWriter( - tmp_path, project_root / "schemas/dictionary-entry.schema.json" - ).write(greek_dictionary_module, export) - first = json.loads((tmp_path / "strongsgreek/G3056.json").read_text(encoding="utf-8")) second = json.loads((tmp_path / "strongsgreek/G3056--2.json").read_text(encoding="utf-8")) - keys = json.loads((tmp_path / "strongsgreek/keys.json").read_text(encoding="utf-8")) + index = json.loads((tmp_path / "strongsgreek/index.json").read_text(encoding="utf-8")) assert first["text"] == "First" assert first["occurrence"] == 1 assert second["text"] == "Second" assert second["occurrence"] == 2 - assert [entry["url"] for entry in keys] == ["G3056.json", "G3056--2.json"] - assert summary["entry_count"] == 2 - assert summary["unique_key_count"] == 1 + assert [item["id"] for item in index["entries"]] == ["G3056", "G3056--2"] + assert index["entries"][1]["occurrence"] == 2 + assert record["entry_count"] == 2 + assert record["unique_key_count"] == 1 + + +def test_whole_dictionary_embeds_entries_in_index_order( + tmp_path, project_root, greek_dictionary_module +) -> None: + record, _ = write( + tmp_path, + project_root, + greek_dictionary_module, + [ + {"key": "Zeta", "raw": "last", "plain": "last", "html": ""}, + {"key": "Alpha", "raw": "first", "plain": "first", "html": ""}, + ], + ) + complete = json.loads((tmp_path / "strongsgreek.json").read_text(encoding="utf-8")) + entries = [ + json.loads((tmp_path / f"strongsgreek/k-{name}.json").read_text(encoding="utf-8")) + for name in ("Alpha", "Zeta") + ] + assert complete["schema"] == "getbible-dictionary-v1" + assert complete["entries"] == entries + assert record["bytes"] == (tmp_path / "strongsgreek.json").stat().st_size diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py new file mode 100644 index 0000000..e8dbb45 --- /dev/null +++ b/tests/test_pipeline.py @@ -0,0 +1,67 @@ +import pytest + +from study_builder.pipeline import ( + BASE_URLS, + CATALOG_TEMPLATES, + RESERVED_MODULE_IDS, + BuildPipeline, + PipelineConfig, +) + + +@pytest.fixture +def pipeline(tmp_path, project_root) -> BuildPipeline: + return BuildPipeline( + PipelineConfig( + root=project_root, + work_dir=tmp_path / "work", + dist_dir=tmp_path / "dist", + policy_path=project_root / "conf/module_policy.json", + books_path=project_root / "conf/book_registry.json", + schemas_dir=project_root / "schemas", + engine_manifest_path=project_root / "conf/getbiblesword.json", + engine_schema_path=project_root / "schemas/getbiblesword-ndjson-v1.schema.json", + ) + ) + + +def test_reserved_identifiers_cover_every_root_document() -> None: + # A module directory and its whole-module document share the v1 root with these. + for name in ("commentaries", "dictionaries", "build", "hashes", "schema"): + assert name in RESERVED_MODULE_IDS + + +def test_every_resource_publishes_a_base_url_and_url_templates() -> None: + for kind in ("commentaries", "dictionaries"): + assert BASE_URLS[kind].endswith("/v1/") + assert CATALOG_TEMPLATES[kind] + for template in CATALOG_TEMPLATES[kind].values(): + assert template.startswith("{" + kind.removesuffix("ies") + "y}") + + +def test_schemas_are_published_beside_the_data(pipeline, tmp_path) -> None: + root = tmp_path / "generated" + pipeline._publish_schemas(root, "commentaries") + published = {path.name for path in (root / "schema").glob("*.json")} + assert published == { + "commentary.json", + "commentary-book.json", + "commentary-books.json", + "commentary-chapter.json", + } + + dictionary_root = tmp_path / "generated-dictionaries" + pipeline._publish_schemas(dictionary_root, "dictionaries") + published = {path.name for path in (dictionary_root / "schema").glob("*.json")} + assert published == {"dictionary.json", "dictionary-entry.json", "dictionary-index.json"} + + +def test_published_schema_ids_match_their_served_paths(pipeline, tmp_path) -> None: + from study_builder.util import read_json + + for kind in ("commentaries", "dictionaries"): + root = tmp_path / kind + pipeline._publish_schemas(root, kind) + for path in sorted((root / "schema").glob("*.json")): + expected = f"{BASE_URLS[kind].removesuffix('v1/')}schema/v1/{path.name}" + assert read_json(path)["$id"] == expected diff --git a/tests/test_pipeline_build.py b/tests/test_pipeline_build.py new file mode 100644 index 0000000..a6625c6 --- /dev/null +++ b/tests/test_pipeline_build.py @@ -0,0 +1,156 @@ +"""Exercise the whole publication assembly without CrossWire or the extractor.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from study_builder import pipeline as pipeline_module +from study_builder.models import ModuleDescriptor, NativeExport +from study_builder.pipeline import BuildPipeline, PipelineConfig + + +def descriptor(name: str, driver: str, category: str, **extra: str) -> ModuleDescriptor: + fields = { + "description": (f"{name} Description",), + "lang": ("en",), + "moddrv": (driver,), + "category": (category,), + "distributionlicense": ("Public Domain",), + "version": ("1.0",), + } + fields.update({key: (value,) for key, value in extra.items()}) + return ModuleDescriptor(name=name, fields=fields, conf_path=f"mods.d/{name.lower()}.conf") + + +COMMENTARY = descriptor("Clarke", "zCom", "Commentaries") +DICTIONARY = descriptor("Easton", "RawLD", "Lexicons / Dictionaries") + +EXPORTS: dict[str, NativeExport] = { + "Clarke": NativeExport( + metadata={"classification": "commentary"}, + entries=[ + { + "key": "Dan.0.0", + "raw": "Introduction to Daniel.", + "plain": "Introduction to Daniel.", + "html": "", + "verse": {"osis": "Dan.0.0", "testament": 1, "book": 27, "chapter": 0, "verse": 0}, + }, + { + "key": "Dan.1.1", + "raw": "In the third year.", + "plain": "In the third year.", + "html": "In the third year.
", + "verse": {"osis": "Dan.1.1", "testament": 1, "book": 27, "chapter": 1, "verse": 1}, + }, + ], + ), + "Easton": NativeExport( + metadata={"classification": "dictionary_or_lexicon"}, + entries=[ + { + "key": "KADESH", + "raw": 'see ZIN and Num.20.1', + "plain": "Holy.", + "html": "", + }, + {"key": "ZIN", "raw": "a low palm tree", "plain": "A low palm tree.", "html": ""}, + ], + ), +} + + +class StubInstaller: + def __init__(self, *args, **kwargs) -> None: + pass + + def install(self, module: ModuleDescriptor) -> Path: + return Path("/nonexistent") / module.name + + +class StubExporter: + def __init__(self, *args, **kwargs) -> None: + pass + + def export(self, installation: Path, name: str) -> NativeExport: + return EXPORTS[name] + + +@pytest.fixture +def built(tmp_path, project_root, monkeypatch): + monkeypatch.setattr(pipeline_module, "ModuleInstaller", StubInstaller) + monkeypatch.setattr(pipeline_module, "SwordExporter", StubExporter) + monkeypatch.setattr(BuildPipeline, "_catalog", lambda self: [COMMENTARY, DICTIONARY]) + monkeypatch.setattr( + pipeline_module.GetBibleSwordManager, "ensure", lambda self, path=None: Path("/stub") + ) + config = PipelineConfig( + root=project_root, + work_dir=tmp_path / "work", + dist_dir=tmp_path / "dist", + policy_path=project_root / "conf/module_policy.json", + books_path=project_root / "conf/book_registry.json", + schemas_dir=project_root / "schemas", + engine_manifest_path=project_root / "conf/getbiblesword.json", + engine_schema_path=project_root / "schemas/getbiblesword-ndjson-v1.schema.json", + ) + report = BuildPipeline(config).run() + return report, tmp_path / "dist" + + +def test_build_publishes_both_resources_under_v1(built) -> None: + report, dist = built + assert report.built == {"commentaries": ["Clarke"], "dictionaries": ["Easton"]} + assert not report.failed + assert (dist / "commentaries/v1/clarke.json").is_file() + assert (dist / "commentaries/v1/clarke/27/0.json").is_file() + assert (dist / "dictionaries/v1/easton/index.json").is_file() + assert (dist / "dictionaries/v1/easton/k-KADESH.json").is_file() + + +def test_catalog_url_templates_resolve_from_the_version_root(built) -> None: + _, dist = built + catalog = json.loads((dist / "commentaries/v1/commentaries.json").read_text(encoding="utf-8")) + assert catalog["base_url"] == "https://commentaries.getbible.net/v1/" + assert catalog["module_count"] == 1 + record = catalog["commentaries"][0] + assert record["id"] == "clarke" + assert "about" not in record and "copyright" not in record + + for template, replacements in ( + (catalog["chapter_url_template"], {"commentary": "clarke", "book": "27", "chapter": "1"}), + (catalog["book_url_template"], {"commentary": "clarke", "book": "27"}), + (catalog["commentary_url_template"], {"commentary": "clarke"}), + (catalog["books_url_template"], {"commentary": "clarke"}), + (catalog["metadata_url_template"], {"commentary": "clarke"}), + ): + relative = template + for key, value in replacements.items(): + relative = relative.replace("{" + key + "}", value) + assert (dist / "commentaries/v1" / relative).is_file(), relative + + +def test_hashes_manifest_covers_every_other_document(built) -> None: + _, dist = built + root = dist / "dictionaries/v1" + manifest = json.loads((root / "hashes.json").read_text(encoding="utf-8")) + assert manifest["algorithm"] == "sha256" + published = {path.relative_to(root).as_posix() for path in root.rglob("*.json")} + assert set(manifest["files"]) == published - {"hashes.json"} + assert all(len(digest) == 64 for digest in manifest["files"].values()) + + +def test_schemas_are_served_beside_the_data(built) -> None: + _, dist = built + assert (dist / "commentaries/v1/schema/commentary-chapter.json").is_file() + assert (dist / "dictionaries/v1/schema/dictionary-entry.json").is_file() + assert not (dist / "commentaries/v1/schema/dictionary-entry.json").exists() + + +def test_no_document_publishes_markup(built) -> None: + _, dist = built + for path in dist.rglob("*.json"): + assert '"html"' not in path.read_text(encoding="utf-8"), path diff --git a/tests/test_util.py b/tests/test_util.py index 6ecd2a2..9ea6fc1 100644 --- a/tests/test_util.py +++ b/tests/test_util.py @@ -1,6 +1,6 @@ import json -from study_builder.util import replace_tree, write_hash_sidecars, write_json +from study_builder.util import hash_tree, replace_tree, write_composed_json, write_json def test_json_writes_are_stable_and_hashed(tmp_path) -> None: @@ -9,9 +9,36 @@ def test_json_writes_are_stable_and_hashed(tmp_path) -> None: first = (root / "data.json").read_bytes() write_json(root / "data.json", {"hello": "world"}) assert (root / "data.json").read_bytes() == first - hashes = write_hash_sidecars(root) + hashes = hash_tree(root) assert len(hashes["data.json"]) == 64 - assert len((root / "data.json.sha").read_text().strip()) == 40 + assert not list(root.glob("*.sha")) + + +def test_hash_tree_can_exclude_its_own_manifest(tmp_path) -> None: + write_json(tmp_path / "data.json", {"hello": "world"}) + write_json(tmp_path / "hashes.json", {"files": {}}) + assert set(hash_tree(tmp_path, exclude={"hashes.json"})) == {"data.json"} + + +def test_composed_document_embeds_members_verbatim(tmp_path) -> None: + parts = [] + for index in (1, 2): + path = tmp_path / f"part{index}.json" + write_json(path, {"index": index, "nested": {"values": [index, index + 1]}}) + parts.append(path) + composed = tmp_path / "composed.json" + write_composed_json(composed, {"schema": "test", "name": "all"}, "members", parts) + + document = json.loads(composed.read_text(encoding="utf-8")) + assert document["schema"] == "test" + assert document["members"] == [json.loads(path.read_text(encoding="utf-8")) for path in parts] + assert composed.read_text(encoding="utf-8").endswith("]\n}\n") + + +def test_composed_document_handles_an_empty_member_list(tmp_path) -> None: + composed = tmp_path / "composed.json" + write_composed_json(composed, {"schema": "test"}, "members", []) + assert json.loads(composed.read_text(encoding="utf-8")) == {"schema": "test", "members": []} def test_replace_tree_preserves_only_new_generation(tmp_path) -> None: From 3b878cb04769fe1802ee62fbf48fc5d9fbdbc885 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?eW=C9=98yn?= <5607939+Llewellynvdm@users.noreply.github.com> Date: Tue, 11 Aug 2026 14:55:03 +0000 Subject: [PATCH 2/4] Make the study API origins production ready The single example server block was a starting point, not a deployment. This replaces it with the origin configuration, a deploy engine, and a live verifier for both hosts, so bringing the API up is a known procedure rather than an exercise left to the reader. docs/nginx/ holds the real configuration, split so the two hosts differ only in server_name, root, and certificate paths, and the locations they share live in one snippet. It sets two cache tiers (discovery documents are short-lived, the corpus is not), serves the precompressed variants the deploy writes, restricts the API to safe methods, answers preflight cheaply, and returns JSON for 404, 405, and 429 rather than an HTML error page a client has to special-case. It also carries nosniff, a default-src 'none' policy, cross-origin CORP, and HSTS, and exposes ETag to browsers so their own revalidation works. scripts/deploy_static_api.sh pulls, verifies the whole tree against hashes.json, compresses only what changed, syncs one version directory at a time, and reloads. Verification runs before the sync, so a build that fails its digests never reaches the live root and the previous one keeps serving. It can require a valid GPG signature, which ties the origin to the build key. The deploy pulls into a persistent checkout and syncs with rsync rather than cloning, because nginx derives ETags from mtime and size: rewriting every file each month would change every ETag and force every client to re-download a corpus that had not changed. Git and rsync write only what differs, and the compressed variants are stamped with their document's mtime for the same reason. The live root now holds only version directories, so no repository metadata reaches the origin at all. scripts/verify_live_api.sh reads the catalog and follows it, asserting every promise the API makes: paths resolve, both cache tiers are right, compressed variants are served, revalidation returns 304, the CORS and security headers are present, and failures are JSON. Nothing is hardcoded to a module, so it works against any build and is safe to run from a monitor. tests/nginx_config_check.sh wires it together in CI: generate a real tree, put it in a repository, deploy it, serve it through the shipped configuration, and run the verifier against it. That covers what the Python tests cannot reach. docs/deployment.md documents the server layout, the caching model and why ETags survive a deploy, the atomicity trade-off, CDN and real-IP handling, the security posture, rollback, and monitoring. --- .github/workflows/ci.yml | 9 + README.md | 18 + docs/deployment.md | 275 +++++++++++++++ docs/nginx.conf | 45 --- docs/nginx/commentaries.getbible.net.conf | 80 +++++ docs/nginx/dictionaries.getbible.net.conf | 74 +++++ docs/nginx/getbible-api-http.conf | 96 ++++++ docs/nginx/snippets/getbible-api-headers.conf | 24 ++ docs/nginx/snippets/getbible-api-v1.conf | 121 +++++++ docs/target-repositories.md | 9 +- scripts/deploy_static_api.sh | 314 ++++++++++++++++++ scripts/verify_live_api.sh | 173 ++++++++++ tests/nginx_config_check.sh | 144 ++++++++ tests/support/build_sample_tree.py | 58 ++++ 14 files changed, 1393 insertions(+), 47 deletions(-) create mode 100644 docs/deployment.md delete mode 100644 docs/nginx.conf create mode 100644 docs/nginx/commentaries.getbible.net.conf create mode 100644 docs/nginx/dictionaries.getbible.net.conf create mode 100644 docs/nginx/getbible-api-http.conf create mode 100644 docs/nginx/snippets/getbible-api-headers.conf create mode 100644 docs/nginx/snippets/getbible-api-v1.conf create mode 100755 scripts/deploy_static_api.sh create mode 100755 scripts/verify_live_api.sh create mode 100755 tests/nginx_config_check.sh create mode 100755 tests/support/build_sample_tree.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 01af92a..84a837b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,3 +33,12 @@ jobs: study-builder --help study-builder engine --help study-builder build --help + - name: Lint deployment scripts + run: | + sudo apt-get update -qq + sudo apt-get install -y -qq shellcheck + shellcheck --severity=style scripts/*.sh + - name: Check the origin configuration parses + run: | + sudo apt-get install -y -qq nginx + bash tests/nginx_config_check.sh diff --git a/README.md b/README.md index d676aa8..db229a2 100644 --- a/README.md +++ b/README.md @@ -224,6 +224,24 @@ The static output is the system of record. Nginx and a CDN can serve direct lookups without an application process, database connection pool, or request throttling bottleneck. +## Deployment + +A production origin is a pull, a verify, a compress, and a sync: + +```bash +scripts/deploy_static_api.sh \ + --repo git@github.com:getbible/commentaries.git \ + --root /var/www/getbible/commentaries \ + --require-signature + +scripts/verify_live_api.sh https://commentaries.getbible.net +``` + +The whole tree is checked against `hashes.json` before it reaches the live root, +so a failed build leaves the previous one serving. `docs/nginx/` holds the origin +configuration for both hosts and `docs/deployment.md` describes the server +layout, the caching model, the CDN and security posture, and rollback. + ## Local development ```bash diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..6955074 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,275 @@ +# Deploying the study APIs + +Two origins, each serving one generated repository as static JSON. There is no +application process, no database, and no request-time logic — a request is a +file read, which is why the whole design leans on the filesystem and the CDN +rather than on code. + +| Origin | Serves | Live root | +| --- | --- | --- | +| `commentaries.getbible.net` | `getbible/commentaries` | `/var/www/getbible/commentaries` | +| `dictionaries.getbible.net` | `getbible/dictionaries` | `/var/www/getbible/dictionaries` | + +Everything below applies identically to both. Where a command names one, run the +same command with the other's repository and root. + +## What the builder guarantees + +The deploy depends on four properties. If you change the builder, keep them, or +the deployment stops being safe: + +1. **Output is byte-stable.** A module that has not changed rebuilds to an + identical file. Content documents carry no timestamp — only `build.json` and + the catalog do — so an unchanged corpus produces an unchanged tree. +2. **`v1/hashes.json` describes the whole tree.** It holds a SHA-256 for every + other document, so it is both the integrity manifest and the list of paths + the builder owns. +3. **The builder owns only version directories.** Anything else in the + publication repository — README, licence, workflows — is the repository's + own and is never served. +4. **Every document is plain text JSON.** No HTML is published anywhere, which + is why the origin can send `Content-Security-Policy: default-src 'none'` and + why no consumer has to sanitize a response. + +## Server layout + +``` +/var/lib/getbible/commentaries persistent Git checkout (working copy) +/var/www/getbible/commentaries live root nginx serves — version dirs only +/etc/nginx/conf.d/getbible-api-http.conf +/etc/nginx/snippets/getbible-api-headers.conf +/etc/nginx/snippets/getbible-api-v1.conf +/etc/nginx/sites-available/commentaries.getbible.net.conf +``` + +The live root is **not** a Git checkout. The checkout stays in `/var/lib`, and +only the version directories are copied across. That keeps repository metadata +off the origin entirely rather than relying on a rule to hide it. + +## First-time setup + +Requires nginx 1.25.1 or newer (for `http2 on`), git, rsync, python3, gzip, and +ideally brotli. + +```bash +apt-get install -y nginx git rsync python3 brotli +# Brotli for nginx is a separate module; without it only .gz is served, which +# still works — it is roughly 15-20% larger on JSON than brotli. +apt-get install -y libnginx-mod-http-brotli # where packaged + +install -d -m 755 /var/www/getbible /var/lib/getbible /var/www/acme + +cp docs/nginx/getbible-api-http.conf /etc/nginx/conf.d/ +cp docs/nginx/snippets/*.conf /etc/nginx/snippets/ +cp docs/nginx/*.getbible.net.conf /etc/nginx/sites-available/ +ln -sf /etc/nginx/sites-available/commentaries.getbible.net.conf /etc/nginx/sites-enabled/ +ln -sf /etc/nginx/sites-available/dictionaries.getbible.net.conf /etc/nginx/sites-enabled/ +``` + +If the brotli module is installed, uncomment the three `brotli_static on;` lines +in `snippets/getbible-api-v1.conf`. + +Issue certificates before the first `nginx -t`, since the server blocks +reference them: + +```bash +certbot certonly --webroot -w /var/www/acme \ + -d commentaries.getbible.net -d dictionaries.getbible.net +nginx -t && systemctl reload nginx +``` + +## Deploying + +```bash +scripts/deploy_static_api.sh \ + --repo git@github.com:getbible/commentaries.git \ + --root /var/www/getbible/commentaries \ + --require-signature \ + --verify-url https://commentaries.getbible.net/v1/commentaries.json +``` + +The script does five things, and the order is the point — nothing reaches the +live root until it has been proven good: + +1. **Pull.** `git reset --hard` into the persistent checkout. Git rewrites only + files whose content changed, so every unchanged document keeps its mtime. +2. **Verify.** Every digest in `hashes.json` is recomputed, and any `.json` in + the tree that the manifest does not list is an error. A build that fails here + never reaches the live root; the previous deploy keeps serving. +3. **Compress.** `.gz` and `.br` are written beside each document, but only + where missing or stale, and each variant is stamped with its document's + mtime. A monthly rebuild that changes two modules recompresses two modules. +4. **Sync.** `rsync --delete --delay-updates`, one version directory at a time. + Scoping the delete to a single version means no failure of this script can + remove anything outside a version directory. +5. **Reload.** `nginx -t` then reload. New workers start with an empty + open-file cache, which is what makes the new documents visible at once. + +`--require-signature` refuses any commit without a valid GPG signature. The +build workflow signs its commits when the publication secrets are present, so on +a production origin this should always be on: it means the origin serves only +what the build key signed. + +Use `--dry-run` to see the exact change set without touching the live root. + +### Automating it + +```ini +# /etc/systemd/system/getbible-commentaries-deploy.service +[Unit] +Description=Deploy the GetBible commentary API +After=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/local/bin/deploy_static_api.sh \ + --repo git@github.com:getbible/commentaries.git \ + --root /var/www/getbible/commentaries \ + --require-signature \ + --verify-url https://commentaries.getbible.net/v1/commentaries.json +``` + +```ini +# /etc/systemd/system/getbible-commentaries-deploy.timer +[Unit] +Description=Check for a new commentary API build + +[Timer] +OnCalendar=*-*-* 05:30:00 +RandomizedDelaySec=30m +Persistent=true + +[Install] +WantedBy=timers.target +``` + +The builder runs monthly, so a daily timer simply finds nothing to do most days +— the pull is a no-op, verification passes, and rsync transfers nothing. + +## The caching model + +Two tiers, because the documents have two very different change rates: + +| Documents | `max-age` | Why | +| --- | --- | --- | +| `{resource}.json`, `build.json`, `hashes.json` | 300s | Rewritten by every build | +| Everything else | 86400s | Changes only when a source module changes | + +Both carry `stale-while-revalidate` and `stale-if-error`, so a slow or briefly +unreachable origin degrades into serving slightly stale JSON rather than errors. + +Nothing is marked `immutable`, deliberately. Paths are not content-hashed, so a +document's URL is stable across rebuilds; promising immutability would strand +clients on an old copy after a correction. + +### Why ETags survive a deploy + +nginx derives a static file's ETag from its mtime and size. A deploy that +rewrote every file — a fresh clone, or `cp -r` — would change every mtime and so +every ETag, and several thousand clients would re-download a corpus that had not +changed. Git and rsync both write only what actually differs and rsync preserves +mtimes, so unchanged documents keep their ETag and revalidation stays a 304. + +This is why the script pulls into a persistent checkout instead of cloning, and +why the compressed variants are stamped with `touch -r`. + +### Atomicity + +`rsync --delay-updates` stages the whole change set and swaps it in at the end, +which narrows the window where a client could see a mix of old and new documents +to a fast sequence of renames. It does not eliminate it. + +That is a deliberate trade. Eliminating it entirely means deploying to a new +release directory and swapping a symlink — which changes every inode on every +deploy and throws away the ETag stability above. For a read-only corpus where +old and new documents are each individually valid, and where a build lands +monthly, stable ETags are worth far more than a perfectly atomic switch. + +## Behind a CDN + +Put a CDN in front of both origins. The two cache tiers are already what a CDN +wants, and the bulk documents in particular should be served from the edge. + +Restore the real client IP before the rate limits apply, or they will limit the +CDN rather than the client — the commented `set_real_ip_from` block in each +server file is where that goes. Refresh the address ranges from the CDN's +published list on a schedule; do not freeze them into the config. + +If the origin sits only behind a CDN, consider raising the `limit_req` rate: +the limits shipped here assume the origin is directly reachable. + +## Security posture + +- **Read-only.** Anything other than `GET`, `HEAD`, or `OPTIONS` gets a JSON + 405. `client_max_body_size` is 1k — nothing is ever uploaded. +- **No markup anywhere.** `default-src 'none'`, `sandbox`, and `nosniff` mean a + response that somehow was not JSON is inert in a browser. +- **No repository metadata on the origin.** The live root holds only version + directories; the dotfile deny rule is a second line, not the first. +- **Signed input.** `--require-signature` ties the origin to the build key. +- **Verified input.** The whole tree is checked against `hashes.json` before and + after it goes live. +- **No version disclosure.** `server_tokens off`. +- **Abuse limits.** Per-IP request and connection limits, with a tighter + connection cap and a rate cap on the bulk documents so one offline sync cannot + monopolise the origin's upstream. + +## Verifying a live origin + +```bash +scripts/verify_live_api.sh https://commentaries.getbible.net +scripts/verify_live_api.sh https://dictionaries.getbible.net +``` + +The script reads the catalog and follows it, so it works against any build +without being told which modules exist. It asserts every promise the API makes: +paths resolve, both cache tiers are right, precompressed variants are served, +revalidation returns 304, CORS and the security headers are present, and +failures come back as JSON. It exits non-zero on the first broken promise, so it +is safe to run from a monitor. + +## Rolling back + +The live root records what it is serving: + +```bash +cat /var/www/getbible/commentaries/.revision +``` + +Generated repositories are replace-only, so the clean rollback is to revert the +bad commit in the publication repository and deploy again: + +```bash +git -C /var/lib/getbible/commentaries revert --no-edit