Publish plain text only; add dictionary cross-references

.github/workflows/build.yml

@@ -88,8 +88,8 @@ jobs:
           RESOURCE: ${{ github.event_name == 'schedule' && 'all' || inputs.resource }}
           REFRESH: ${{ github.event_name == 'workflow_dispatch' && inputs.refresh || 'false' }}
           PUBLISH: ${{ steps.publication.outputs.enabled }}
-          STUDY_BUILDER_COMMENTARIES_REPO: ${{ secrets.GETBIBLE_COMMENTARIES_REPO || 'git@github.com:getbible/v1_commentaries.git' }}
-          STUDY_BUILDER_DICTIONARIES_REPO: ${{ secrets.GETBIBLE_DICTIONARIES_REPO || 'git@github.com:getbible/v1_dictionaries.git' }}
+          STUDY_BUILDER_COMMENTARIES_REPO: ${{ secrets.GETBIBLE_COMMENTARIES_REPO || 'git@github.com:getbible/commentaries.git' }}
+          STUDY_BUILDER_DICTIONARIES_REPO: ${{ secrets.GETBIBLE_DICTIONARIES_REPO || 'git@github.com:getbible/dictionaries.git' }}
           STUDY_BUILDER_SIGN_COMMITS: ${{ steps.publication.outputs.enabled }}
         shell: bash
         run: |

.github/workflows/ci.yml

@@ -33,3 +33,12 @@ jobs:
           study-builder --help
           study-builder engine --help
           study-builder build --help
+      - name: Lint deployment scripts
+        run: |
+          sudo apt-get update -qq
+          sudo apt-get install -y -qq shellcheck
+          shellcheck --severity=style scripts/*.sh tests/*.sh
+      - name: Install, deploy, and serve the origin configuration
+        run: |
+          sudo apt-get install -y -qq nginx
+          sudo --preserve-env=PATH bash tests/nginx_config_check.sh

AGENTS.md

@@ -10,7 +10,9 @@ publication workflow.
 - Runtime: Python 3.12.
 - Extractor: the separately released `getbiblesword` executable, pinned in
   `conf/getbiblesword.json` and invoked only as a subprocess.
-- Outputs: static JSON trees for `v1_commentaries` and `v1_dictionaries`.
+- Outputs: static JSON trees under `v1/` in `getbible/commentaries` and
+  `getbible/dictionaries`. The version lives in the folder, not the repository
+  name, so a future `v2/` can be published beside it.
 
 This repository does not build or link the CrossWire SWORD C++ engine. Changes to
 that engine belong in `getbible/getbiblesword`. Do not reintroduce a local C++
@@ -32,21 +34,42 @@ Never use a `utf8` convenience field as the authoritative value. Decode `base64`
 verify it, then create the public text projection. Unknown additive fields must be
 retained in the internal source record. Validated entries remain disk-backed and
 writers stream them; do not restore whole-module entry or commentary collections in
-memory. A missing footer, failed digest, failed artifact, unsupported major contract,
-or extractor error blocks all publication.
+memory. Composed documents are streamed from the documents they embed, never built
+up as one object. A missing footer, failed digest, failed artifact, unsupported major
+contract, or extractor error blocks all publication.
 
 ## API stability
 
-Commentary files remain addressable by GetBible book number and chapter. Dictionary
-Strong's keys remain compatible with Bible API v3 (`G3056`, `H0430`). Any breaking
-path or document change requires a new API version; do not silently mutate v1.
+The published API is plain text. No document may reintroduce an `html` member, and
+the builder must not grow an HTML sanitizer; the value of the text-only contract is
+that no consumer has to sanitize a response.
+
+Commentary files remain addressable by GetBible book number and chapter. Chapter `0`
+is a book introduction and verse `0` a chapter introduction; neither may be dropped.
+Book and whole-commentary documents embed their parts byte-for-byte, so
+`book.chapters[n]` must stay identical to the chapter document served on its own —
+`scripts/validate_build.py` asserts this and it is the property clients rely on.
+
+Dictionary Strong's keys remain compatible with Bible API v3 (`G3056`, `H0430`).
 Repeated dictionary keys retain the unsuffixed direct path for their first
 definition; later definitions use deterministic `--2`, `--3`, and subsequent
-suffixes and must all remain discoverable through `keys.json`.
+suffixes and must all remain discoverable through `index.json`, which stays sorted
+by its `search` term. Cross-references between words resolve only to keys that
+exist in the same dictionary.
+
+A module identifier may never collide with a document at the `v1/` root; see
+`RESERVED_MODULE_IDS`. Any breaking path or document change requires a new API
+version; do not silently mutate v1.
 
 Generated repositories are replace-only outputs. A partial `--module` build may be
 used for tests but must never be pushed.
 
+## Commits
+
+Commits in this repository are authored in the maintainer's name. Do not add a
+`Co-Authored-By` trailer, a session link, an assistant name, or any other
+tool attribution to a commit message, tag, or pull request.
+
 ## Verification
 
 Run before publishing changes:

README.md

@@ -5,21 +5,24 @@
 `v1_study_builder` converts policy-approved CrossWire SWORD commentary and
 dictionary modules into two independently deployable static JSON APIs:
 
-- `https://commentaries.getbible.net/v1/` from `getbible/v1_commentaries`
-- `https://dictionaries.getbible.net/v1/` from `getbible/v1_dictionaries`
+- `https://commentaries.getbible.net/v1/` from `getbible/commentaries`
+- `https://dictionaries.getbible.net/v1/` from `getbible/dictionaries`
 
 The Bible API v3 builder remains unchanged. Study Builder deliberately uses the
 same book numbers, chapters, verses, and Strong's keys so a client can move from a
 Bible response to commentary or dictionary data with a direct path lookup.
 
+Every document is plain text. Nothing in either API publishes HTML, so a consuming
+application never has to sanitize a response before rendering it.
+
 ## Repository boundaries
 
 | Repository | Responsibility | Runtime |
 | --- | --- | --- |
 | `getbible/getbiblesword` | Official SWORD C++ extraction and deterministic NDJSON | Released Linux executable |
 | `getbible/v1_study_builder` | Download policy, strict contract validation, normalization, schemas, and publication | Python 3.12 at build time |
-| `getbible/v1_commentaries` | Generated commentary JSON under `v1/` | Nginx/CDN only |
-| `getbible/v1_dictionaries` | Generated dictionary JSON under `v1/` | Nginx/CDN only |
+| `getbible/commentaries` | Generated commentary JSON under `v1/` | Nginx/CDN only |
+| `getbible/dictionaries` | Generated dictionary JSON under `v1/` | Nginx/CDN only |
 
 Study Builder does not contain C++, link `libsword`, use a Python SWORD binding, or
 parse a module's binary driver format. `getbiblesword` is a separately versioned
@@ -65,30 +68,42 @@ and independently checks all of the rules that protect publication:
 - exact stream SHA-256 over every line before the footer, including LF;
 - exact footer record/entry/artifact/byte counts and `success: true`.
 
-Raw bytes remain authoritative. The adapter derives safe text/HTML for the public
-API only after verification and retains the original contract records internally.
+Raw bytes remain authoritative. The adapter derives the public plain text only
+after verification and retains the original contract records internally.
 Validated entries are held in a compressed, disk-backed spool. Commentary entries
 are then normalized into disk-backed chapter buckets and emitted in canonical
 GetBible book/chapter order; this supports source modules whose versification orders
 canonical or deuterocanonical books differently. Dictionary definitions are written
-one at a time. This keeps memory bounded for large modules without weakening the
-contract or the all-or-nothing publication rule. Any missing footer, checksum
-failure, failed diagnostic, extractor error, or classification mismatch stops the
-complete build before publication.
+one at a time. Book, whole-commentary, and whole-dictionary documents are streamed
+from the documents they contain rather than assembled in memory. This keeps memory
+bounded for large modules without weakening the contract or the all-or-nothing
+publication rule. Any missing footer, checksum failure, failed diagnostic,
+extractor error, or classification mismatch stops the complete build before
+publication.
 
 ## Commentary API
 
 ```text
 GET https://commentaries.getbible.net/v1/commentaries.json
+GET https://commentaries.getbible.net/v1/{commentary}.json
 GET https://commentaries.getbible.net/v1/{commentary}/metadata.json
 GET https://commentaries.getbible.net/v1/{commentary}/books.json
 GET https://commentaries.getbible.net/v1/{commentary}/{book}.json
 GET https://commentaries.getbible.net/v1/{commentary}/{book}/{chapter}.json
 ```
 
-The chapter path is the primary high-volume endpoint. `book` is the GetBible API
-v3 numeric identifier: Genesis is `1`, Matthew `40`, and Revelation `66`. Each
-entry contains its natural Bible coordinate:
+`book` is the GetBible API v3 numeric identifier: Genesis is `1`, Daniel `27`,
+Matthew `40`, and Revelation `66`. Deuterocanonical books continue to `83`.
+
+The three content levels are self-similar. A chapter document is one member of a
+book document, which is one member of a whole-commentary document, embedded
+byte-for-byte. One client parser therefore handles all three:
+
+```text
+{commentary}/{book}/{chapter}.json    one chapter, the high-volume endpoint
+{commentary}/{book}.json             every chapter of that book
+{commentary}.json                    every book of that commentary
+```
 
 ```json
 {
@@ -106,24 +121,41 @@ entry contains its natural Bible coordinate:
       "name": "John 1:1",
       "anchor": {"book": 43, "chapter": 1, "verse": 1, "osis": "John.1.1"},
       "text": "...",
-      "html": "<p>...</p>"
+      "references": [{"osis": "Gen.1.1", "book": 1, "chapter": 1, "verse": 1}]
     }
   ]
 }
 ```
 
-Book and chapter introductions use chapter or verse `0`; they are not discarded.
+Introductions are published, not discarded. A book introduction is chapter `0`,
+so Clarke's introduction to Daniel is `clarke/27/0.json`. A chapter introduction
+is verse `0`, and appears as the first entry of its own chapter document.
+
+`books.json` reports which books and chapters a commentary covers, and
+`metadata.json` reports its licence, counts, and the byte size of the
+whole-commentary document so a client can decide before requesting it.
 
 ## Dictionary API
 
 ```text
 GET https://dictionaries.getbible.net/v1/dictionaries.json
+GET https://dictionaries.getbible.net/v1/{dictionary}.json
 GET https://dictionaries.getbible.net/v1/{dictionary}/metadata.json
-GET https://dictionaries.getbible.net/v1/{dictionary}/keys.json
+GET https://dictionaries.getbible.net/v1/{dictionary}/index.json
 GET https://dictionaries.getbible.net/v1/{dictionary}/{entry}.json
-GET https://dictionaries.getbible.net/v1/{dictionary}/indexes/{sha256-prefix}.json
 ```
 
+Searching a dictionary takes two requests. `index.json` lists every word once,
+sorted by an accent-insensitive lowercase `search` term, so a client can fetch it
+once and then search, prefix-match, or binary-search entirely in memory:
+
+```json
+{"id": "k-KADESH", "key": "KADESH", "search": "kadesh"}
+```
+
+The record's `id` is the path of the word itself — `{entry}.json` — so a hit in
+the index resolves to exactly one document with no further lookup.
+
 Strong's paths match Bible API v3 tokens directly:
 
 ```text
@@ -133,17 +165,48 @@ H0430 -> https://dictionaries.getbible.net/v1/strongshebrew/H0430.json
 
 Greek keys use `G` plus the unpadded number; Hebrew keys use `H0` plus the
 unpadded number. Other dictionary keys receive deterministic, path-safe IDs.
-`keys.json` maps source keys and aliases, while 256 SHA-256-prefix shards provide
-smaller lookup indexes for constrained clients.
+
+Each word document carries the dictionary's own link graph, so a client can
+navigate in either direction without rebuilding an index:
+
+```json
+{
+  "schema": "getbible-dictionary-entry-v1",
+  "dictionary": "easton",
+  "id": "k-KADESH",
+  "key": "KADESH",
+  "occurrence": 1,
+  "aliases": ["KADESH"],
+  "text": "Holy; a place in the wilderness of Zin.",
+  "see_also": [{"id": "k-MERIBAH", "key": "MERIBAH"}],
+  "backlinks": [{"id": "k-ZIN", "key": "ZIN"}],
+  "references": [{"osis": "Num.20.1", "book": 4, "chapter": 20, "verse": 1}]
+}
+```
+
+`see_also` lists the words this entry points at and `backlinks` the words that
+point back. Only targets that resolve to a real key in the same dictionary are
+published. Scripture references stay in `references`, in the same shape the
+commentary API uses.
 
 Some SWORD dictionaries legitimately contain more than one definition for the
 same public key. The first definition keeps the canonical direct path, and later
 definitions receive deterministic `--2`, `--3`, and subsequent suffixes. For
 example, Easton's repeated `KADESH` records are available as `k-KADESH.json` and
-`k-KADESH--2.json`. Every definition appears in `keys.json` with an `occurrence`
+`k-KADESH--2.json`. Every definition appears in `index.json` with an `occurrence`
 value. Dictionary metadata reports both the total `entry_count` and the distinct
 `unique_key_count`.
 
+`{dictionary}.json` is the complete dictionary in index order, for offline
+clients that would otherwise request every word individually.
+
+## Integrity and schemas
+
+Each API root publishes `hashes.json`, a SHA-256 digest of every other generated
+document, which is also the manifest of the paths a build owns. The JSON Schemas
+for every document type are served beside the data under `v1/schema/`, so each
+schema `$id` resolves to the document that defines it.
+
 ## Build flow
 
 ```mermaid
@@ -152,15 +215,37 @@ flowchart TD
     B --> C["NDJSON v1 subprocess stream"]
     C --> D["Independent stream + artifact validator"]
     D --> E["Python API adapter + JSON Schema"]
-    E --> F["Atomic static v1 trees + hash sidecars"]
-    F --> G["v1_commentaries, when publication secrets exist"]
-    F --> H["v1_dictionaries, when publication secrets exist"]
+    E --> F["Atomic static v1 trees + SHA-256 manifest"]
+    F --> G["commentaries, when publication secrets exist"]
+    F --> H["dictionaries, when publication secrets exist"]
 ```
 
 The static output is the system of record. Nginx and a CDN can serve direct
 lookups without an application process, database connection pool, or request
 throttling bottleneck.
 
+## Deployment
+
+A production origin is a pull, a verify, a compress, and a sync:
+
+```bash
+scripts/deploy_static_api.sh \
+    --repo git@github.com:getbible/commentaries.git \
+    --root /var/www/getbible/commentaries \
+    --require-signature
+
+scripts/verify_live_api.sh https://commentaries.getbible.net
+```
+
+The whole tree is checked against `hashes.json` before it reaches the live root,
+so a failed build leaves the previous one serving.
+
+`docs/nginx/` holds the origin configuration for both hosts; install it with
+`scripts/install_nginx_config.sh`, which adapts it to the host's nginx version,
+brotli availability, and IPv6 support rather than leaving those as footguns.
+`docs/deployment.md` describes the server layout, the caching model, the CDN and
+security posture, rollback, and monitoring.
+
 ## Local development
 
 ```bash
@@ -232,8 +317,8 @@ Publication secret set:
 | `GETBIBLE_SSH_KEY` | SSH private key with write access to both outputs |
 | `GETBIBLE_SSH_PUB` | Matching public key |
 
-The default output remotes are `getbible/v1_commentaries` and
-`getbible/v1_dictionaries`. Optional `GETBIBLE_COMMENTARIES_REPO` and
+The default output remotes are `getbible/commentaries` and
+`getbible/dictionaries`. Optional `GETBIBLE_COMMENTARIES_REPO` and
 `GETBIBLE_DICTIONARIES_REPO` secrets may select staging remotes.
 
 ## Redistribution policy