Collapse repeated comments and enforce document size limits - #5
Merged
Conversation
A SWORD commentary attaches one comment to a verse range, and the extractor reports that same text once for every verse in the range. Writing an entry per verse stored the identical paragraph dozens of times: an exposition of a psalm reappeared under all 176 verses of Psalm 119, and the whole-commentary documents reached 481 MB for Matthew Henry and 306 MB for FreAug without carrying any more text than the source held. That is what the publication push was rejected for. Each distinct comment is now published once, anchored at the lowest verse it covers, with `verses` listing every verse it applies to when that is more than one. Nothing is dropped: every verse the source commented on still resolves to its comment, and resolving one is a single rule -- an entry covers `verses` when that member is present and `verse` alone when it is not. Grouping stops at the chapter boundary so a chapter document still stands alone, and references from every verse in a range are unioned onto the published entry rather than taken from whichever verse happened to be seen first. Entries also lose `name` and `anchor`. Both only restated values already present on the entry or its chapter: `name` was the book name with `chapter:verse`, and `anchor` repeated `book`, `chapter`, and `verse` verbatim. `osis` stays, as a plain member, because it is the source module's own key rather than a restatement. The build now measures itself instead of leaving this to be guessed at. metadata.json carries a `storage` block reporting source against published bytes, the repetition ratio the collapse removed, and what each of the three levels costs; the build report collects the same per module, so a CI run reports it. No generated document may exceed --max-document-bytes, 95 MB by default and just under the 100 MB a Git remote refuses. The build now fails naming the offending file rather than producing a tree that is rejected at push time hours later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change optimizes commentary storage by deduplicating comments attached to verse ranges and adds safeguards to prevent oversized documents from being published. SWORD commentaries attach a single comment to a verse range but report it once per verse in that range, leading to massive duplication. The fix stores each distinct comment once, anchored at the lowest verse it covers, with a
versesarray listing all verses it applies to.Key Changes
_chapter_entries()to collapse identical comments across verse ranges into a single entry with aversesarray, reducing storage by 9-10x for typical commentariesenforce_document_ceiling()utility that fails the build immediately when a generated document exceeds configurable limits (default 95 MB), preventing rejected pushes hours laternameandanchorobject fields from commentary entries;osisis now a direct property. Updated schema to reflect the newversesfield and document the deduplication behaviorstorageblock to metadata tracking source vs. published entry counts, byte sizes at each level, andrepetition_ratioto measure deduplication effectiveness--max-document-bytesflag to control the ceiling (0 disables), with environment variable override supportImplementation Details
write_composed_json()now returns the written document size for ceiling enforcementCommentaryWriterandDictionaryWriteracceptmax_document_bytesparameter