Collapse repeated comments and enforce document size limits

.gitignore

@@ -1,4 +1,4 @@
-.venv/
+.venv*/
 __pycache__/
 *.py[cod]
 .pytest_cache/

README.md

@@ -71,9 +71,10 @@ and independently checks all of the rules that protect publication:
 Raw bytes remain authoritative. The adapter derives the public plain text only
 after verification and retains the original contract records internally.
 Validated entries are held in a compressed, disk-backed spool. Commentary entries
-are then normalized into disk-backed chapter buckets and emitted in canonical
-GetBible book/chapter order; this supports source modules whose versification orders
-canonical or deuterocanonical books differently. Dictionary definitions are written
+are then normalized into disk-backed chapter buckets, collapsed so that a comment
+attached to a verse range is stored once rather than once per verse, and emitted in
+canonical GetBible book/chapter order; this supports source modules whose versification
+orders canonical or deuterocanonical books differently. Dictionary definitions are written
 one at a time. Book, whole-commentary, and whole-dictionary documents are streamed
 from the documents they contain rather than assembled in memory. This keeps memory
 bounded for large modules without weakening the contract or the all-or-nothing
@@ -118,15 +119,54 @@ byte-for-byte. One client parser therefore handles all three:
       "book": 43,
       "chapter": 1,
       "verse": 1,
-      "name": "John 1:1",
-      "anchor": {"book": 43, "chapter": 1, "verse": 1, "osis": "John.1.1"},
+      "osis": "John.1.1",
       "text": "...",
       "references": [{"osis": "Gen.1.1", "book": 1, "chapter": 1, "verse": 1}]
     }
   ]
 }
 ```
 
+### One comment, stored once
+
+A SWORD commentary attaches a comment to a verse *range*, and the extractor reports
+that same text once for every verse in the range. Publishing an entry per verse
+stored the identical paragraph dozens of times — Augustine's exposition of a psalm
+reappeared under all 176 verses of Psalm 119, and the whole-commentary documents grew
+into the hundreds of megabytes without carrying any more text.
+
+Each distinct comment is therefore published **once**, anchored at the lowest verse it
+covers. When it covers more than one verse, `verses` lists every verse it applies to:
+
+```json
+{
+  "book": 19,
+  "chapter": 119,
+  "verse": 1,
+  "verses": [1, 2, 3, 4, 5, 6, 7, 8],
+  "osis": "Ps.119.1",
+  "text": "..."
+}
+```
+
+Resolving a verse is one rule: **an entry covers `verses` when that member is present,
+and `verse` alone when it is not.**
+
+```js
+const forVerse = (chapter, n) =>
+  chapter.entries.find(e => (e.verses ?? [e.verse]).includes(n))
+```
+
+Nothing is dropped by this — every verse the source commented on still resolves to its
+comment. Grouping stops at the chapter boundary, because a chapter document is the
+addressable unit and has to stand alone, so a comment spanning a chapter break is
+published in both chapters.
+
+An entry carries no `name` and no `anchor` object. Both only restated values already
+present on the entry or its chapter: `name` is the book name with `chapter:verse`, and
+`anchor` repeated `book`, `chapter`, and `verse` verbatim. `osis` — the source module's
+own key for the anchor verse — is kept as a plain member.
+
 Introductions are published, not discarded. A book introduction is chapter `0`,
 so Clarke's introduction to Daniel is `clarke/27/0.json`. A chapter introduction
 is verse `0`, and appears as the first entry of its own chapter document.
@@ -135,6 +175,30 @@ is verse `0`, and appears as the first entry of its own chapter document.
 `metadata.json` reports its licence, counts, and the byte size of the
 whole-commentary document so a client can decide before requesting it.
 
+`metadata.json` also carries a `storage` block, which is the build's own measurement
+of this module rather than anything a client needs:
+
+```json
+{
+  "source_entry_count": 168447,
+  "source_text_bytes": 402653184,
+  "text_bytes": 41943040,
+  "repetition_ratio": 9.6,
+  "chapter_bytes": 44040192,
+  "book_bytes": 44564480,
+  "commentary_bytes": 45088768,
+  "published_bytes": 133693440
+}
+```
+
+`repetition_ratio` is how many times the average byte of source text was repeated
+across the verse range it was attached to — the multiplier the collapse removes. The
+three `*_bytes` members are what each level of the API costs on disk.
+
+No generated document may exceed `--max-document-bytes` (default 95 MB, just under the
+100 MB a Git remote refuses). The build fails and names the file rather than producing
+a tree that is rejected at push time, hours later. Set it to `0` to disable the check.
+
 ## Dictionary API
 
 ```text

docs/deployment.md

@@ -30,6 +30,9 @@ the deployment stops being safe:
 4. **Every document is plain text JSON.** No HTML is published anywhere, which
    is why the origin can send `Content-Security-Policy: default-src 'none'` and
    why no consumer has to sanitize a response.
+5. **No document exceeds `--max-document-bytes`** (95 MB by default). The build
+   fails naming the offending file rather than publishing a tree the publication
+   remote would reject, so the origin never sees a half-pushed corpus.
 
 ## Server layout
 

schemas/commentary-chapter.schema.json

@@ -21,22 +21,27 @@
   "$defs": {
     "entry": {
       "type": "object",
-      "required": ["book", "chapter", "verse", "name", "anchor", "text"],
+      "description": "One comment. A source module attaches a comment to a verse range and repeats it for every verse in that range; it is published once here, anchored at the lowest verse it covers, with `verses` listing every verse it applies to when that is more than one.",
+      "required": ["book", "chapter", "verse", "text"],
       "properties": {
         "book": {"type": "integer", "minimum": 1, "maximum": 83},
         "chapter": {"type": "integer", "minimum": 0},
-        "verse": {"type": "integer", "minimum": 0},
-        "name": {"type": "string"},
-        "anchor": {
-          "type": "object",
-          "required": ["book", "chapter", "verse"],
-          "properties": {
-            "book": {"type": "integer", "minimum": 1, "maximum": 83},
-            "chapter": {"type": "integer", "minimum": 0},
-            "verse": {"type": "integer", "minimum": 0},
-            "osis": {"type": "string"}
-          },
-          "additionalProperties": false
+        "verse": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "The lowest verse this comment covers."
+        },
+        "verses": {
+          "type": "array",
+          "description": "Every verse this comment covers, including `verse`. Absent when it covers only `verse`.",
+          "items": {"type": "integer", "minimum": 0},
+          "minItems": 2,
+          "uniqueItems": true
+        },
+        "osis": {
+          "type": "string",
+          "minLength": 1,
+          "description": "The source module's own key for the anchor verse."
         },
         "text": {"type": "string"},
         "references": {"$ref": "#/$defs/references"}

scripts/validate_build.py

@@ -30,6 +30,24 @@ def _assert_composed(composed: list[Any], parts: list[Path], where: str) -> None
             raise RuntimeError(f"{where} does not match the document served at {path}")
 
 
+def _assert_no_repeated_text(chapter: dict[str, Any], where: Path) -> None:
+    """The whole point of the collapse: one comment is stored once, not once per verse.
+
+    Two distinct comments may still land on the same verse — a source module can emit
+    more than one record for a verse — so verse coverage is deliberately not asserted
+    to be disjoint. What must hold is that no text repeats.
+    """
+    seen: set[str] = set()
+    for entry in chapter["entries"]:
+        text = entry["text"]
+        if text in seen:
+            raise RuntimeError(f"{where} publishes the same comment more than once")
+        seen.add(text)
+        verses = entry.get("verses", [entry["verse"]])
+        if entry["verse"] != min(verses):
+            raise RuntimeError(f"{where} anchors an entry above the lowest verse it covers")
+
+
 def validate_commentary(root: Path, complete_path: Path) -> dict[str, Any]:
     metadata = read_json(root / "metadata.json")
     books = read_json(root / "books.json")
@@ -54,8 +72,9 @@ def validate_commentary(root: Path, complete_path: Path) -> dict[str, Any]:
     if chapter.get("schema") != "getbible-commentary-chapter-v1" or not chapter.get("entries"):
         raise RuntimeError("Commentary chapter produced no entries")
     first = chapter["entries"][0]
-    if not all(name in first for name in ("book", "chapter", "verse", "anchor", "text")):
+    if not all(name in first for name in ("book", "chapter", "verse", "text")):
         raise RuntimeError("Commentary entry is not linked to a Bible API coordinate")
+    _assert_no_repeated_text(chapter, chapter_paths[0])
     _reject_markup(chapter, "chapter")
 
     complete = read_json(complete_path)

src/study_builder/cli.py

@@ -11,7 +11,7 @@
 from study_builder.http import HttpClient
 from study_builder.pipeline import BuildPipeline, PipelineConfig
 from study_builder.policy import ModulePolicy
-from study_builder.util import reset_directory
+from study_builder.util import DOCUMENT_CEILING_BYTES, reset_directory
 
 
 def repository_root() -> Path:
@@ -65,6 +65,14 @@ def parser() -> argparse.ArgumentParser:
             "git@github.com:getbible/dictionaries.git",
         ),
     )
+    build.add_argument(
+        "--max-document-bytes",
+        type=int,
+        default=int(
+            os.environ.get("STUDY_BUILDER_MAX_DOCUMENT_BYTES", DOCUMENT_CEILING_BYTES) or 0
+        ),
+        help="Fail the build rather than publish a document above this size (0 disables)",
+    )
     build.add_argument("--commentaries-branch", default="main")
     build.add_argument("--dictionaries-branch", default="main")
     build.add_argument("--pull", action="store_true", help="Clone/pull target repositories")
@@ -122,6 +130,7 @@ def _build(args: argparse.Namespace) -> int:
         dictionaries_repo=args.dictionaries_repo,
         commentaries_branch=args.commentaries_branch,
         dictionaries_branch=args.dictionaries_branch,
+        max_document_bytes=args.max_document_bytes,
     )
     report = BuildPipeline(config).run()
     print(json.dumps(report.as_dict(), ensure_ascii=False, indent=2))