Publish plain text only; add dictionary cross-references - #4
Merged
Conversation
The generated API could only be read one chapter at a time, duplicated every
entry as both text and HTML, and silently discarded book introductions. This
reshapes all three, keeping output under v1/ so the version stays in the folder
rather than the repository name.
Commentaries gain book and whole-commentary documents. The three levels are
self-similar: a chapter document is one member of a book document, which is one
member of a whole-commentary document, embedded byte-for-byte, so one client
parser handles all three. Composed documents stream from the documents they
contain rather than being assembled in memory.
Book introductions (chapter 0) and chapter introductions (verse 0) are published
again, at {book}/0.json and as the first entry of their chapter. The README had
promised this while the writer dropped them.
Dictionaries replace keys.json and the 256 SHA-256 shards with a single sorted
index.json. A client cannot know which SHA-256 shard holds a word without
already hashing it, so the shards enabled no search at all. Records carry an
accent-insensitive search term and drop the url that was always derivable from
the id. Entries now carry the dictionary's own link graph as see_also and
backlinks, resolved over two passes because forward and reverse links are only
knowable once every key has an identifier. Whole-dictionary documents are
published for offline clients.
The html member is gone from every document and both schemas. It duplicated
text almost everywhere, and with no markup republished the builder needs no HTML
sanitizer, so bleach is dropped and the API carries nothing a consumer must
sanitize before rendering. Entries whose stripped text is empty fall back to text
derived from the rendered form instead of vanishing.
Also: unify references to the object form in dictionaries as well as
commentaries; slim the catalogs and hoist url templates to the envelope; guard
module identifiers against colliding with a root document; retire the per-file
.sha sidecars, halving the file count, in favour of hashes.json, which doubles as
the manifest of builder-owned paths; serve the schemas beside the data so every
$id resolves; and retarget output at getbible/commentaries and
getbible/dictionaries.
The single example server block was a starting point, not a deployment. This replaces it with the origin configuration, a deploy engine, and a live verifier for both hosts, so bringing the API up is a known procedure rather than an exercise left to the reader. docs/nginx/ holds the real configuration, split so the two hosts differ only in server_name, root, and certificate paths, and the locations they share live in one snippet. It sets two cache tiers (discovery documents are short-lived, the corpus is not), serves the precompressed variants the deploy writes, restricts the API to safe methods, answers preflight cheaply, and returns JSON for 404, 405, and 429 rather than an HTML error page a client has to special-case. It also carries nosniff, a default-src 'none' policy, cross-origin CORP, and HSTS, and exposes ETag to browsers so their own revalidation works. scripts/deploy_static_api.sh pulls, verifies the whole tree against hashes.json, compresses only what changed, syncs one version directory at a time, and reloads. Verification runs before the sync, so a build that fails its digests never reaches the live root and the previous one keeps serving. It can require a valid GPG signature, which ties the origin to the build key. The deploy pulls into a persistent checkout and syncs with rsync rather than cloning, because nginx derives ETags from mtime and size: rewriting every file each month would change every ETag and force every client to re-download a corpus that had not changed. Git and rsync write only what differs, and the compressed variants are stamped with their document's mtime for the same reason. The live root now holds only version directories, so no repository metadata reaches the origin at all. scripts/verify_live_api.sh reads the catalog and follows it, asserting every promise the API makes: paths resolve, both cache tiers are right, compressed variants are served, revalidation returns 304, the CORS and security headers are present, and failures are JSON. Nothing is hardcoded to a module, so it works against any build and is safe to run from a monitor. tests/nginx_config_check.sh wires it together in CI: generate a real tree, put it in a repository, deploy it, serve it through the shipped configuration, and run the verifier against it. That covers what the Python tests cannot reach. docs/deployment.md documents the server layout, the caching model and why ETags survive a deploy, the atomicity trade-off, CDN and real-IP handling, the security posture, rollback, and monitoring.
Standing the configuration up against a real distribution nginx rather than a purpose-built test config exposed four things that would each have broken a first deployment. A stock nginx.conf already declares sendfile, gzip, tcp_nopush, ssl_protocols, and ssl_prefer_server_ciphers at http level, and nginx treats a second declaration in the same context as a fatal error — so the http-context drop-in refused to start on a stock Ubuntu box. conf.d/ now carries only the shared memory zones and the log format, which genuinely cannot live in a server block; everything else moved to snippets/getbible-api-server.conf, where it overrides the http-level settings instead of colliding with them. The access log was named .json, outside the stock /etc/logrotate.d/nginx glob of /var/log/nginx/*.log, so it would never have rotated and would eventually have filled the disk. It is now .access.log. "http2 on" needs nginx 1.25.1, but Ubuntu 24.04 LTS ships 1.24, so a first nginx -t would have failed on the LTS release most likely to be hosting this. brotli_static and IPv6 listeners have the same shape of problem. Rather than leave three version footguns in a file people copy by hand, scripts/install_nginx_config.sh detects each one, adapts the shipped files, stages them, and runs nginx -t before installing. It also refuses to proceed when a certificate is missing, naming the certbot command to run, because nginx otherwise fails with a filename and no indication of what to do. The ACME challenge now resolves from the stock /var/www/html rather than a dedicated webroot nothing serves yet, which removes the chicken-and-egg between issuing the first certificate and installing the configuration that serves the challenge for renewals. Dotfiles return 404 instead of deny: the caller sees the same JSON 404 as any unknown path, which confirms nothing, and a scanned origin no longer fills its error log with lines that would bury real failures. tests/nginx_config_check.sh now drives the real installer against the distribution's own nginx.conf and serves both hosts, which is what caught all of the above; the previous hand-rolled test config could not have. It also asserts the access log stays inside the logrotate glob. CI runs it as root and shellchecks the tests as well as the scripts. Also: deploy_static_api.sh reports an unreachable --verify-url instead of exiting bare, and docs/deployment.md covers the installer, the configuration split and why it exists, certificate issuance and renewal, and log rotation.
Deploy verification only rejected an unexpected .json, so a stray .html or .js committed into a version directory would have passed and been served from the API's own hostname. hashes.json is the complete list of what the builder publishes, so anything else in the tree is unaccounted for: every file is now checked, with only the .gz and .br variants this script writes exempted. Also: --help printed a line of the script itself, because the header comment block had outgrown the fixed line range it was extracted with; it is now bounded by the block. And document that --require-signature verifies against the deploying user's GPG keyring, so the build key has to be imported and trusted on each origin first or every deploy fails closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change removes HTML from the public API entirely, publishing only plain text. It also adds cross-reference linking within dictionaries, allowing entries to point to related words. The builder now performs two passes over dictionary entries to assign stable identifiers and resolve links.
Key Changes
Plain text only: Removed the
bleachHTML sanitizer dependency. Thestrip_markup()function now extracts readable text from source markup without republishing any HTML. All schemas and documents have been updated to removehtmlfields.Dictionary cross-references: Added
link_candidates()function to extract cross-reference targets from source markup (SWORD links, markup targets, and Strong's "see" references). Dictionary entries now includesee_alsoandbacklinksarrays to navigate the word graph.Two-pass dictionary writing:
DictionaryWriternow stages entries in a first pass to assign stable identifiers and collect targets, then resolves links in a second pass before writing. This allows forward and reverse links to be computed once all keys are known.Dictionary index: Added a new
index.jsondocument listing all entries sorted by accent-insensitive search term, enabling efficient client-side dictionary search.Composed documents: Added
write_composed_json()to create bulk documents that embed their parts byte-for-byte, allowing offline clients to fetch a single file while online clients use individual documents.Repository naming: Updated references from
v1_commentariesandv1_dictionariestocommentariesanddictionaries, with the version living in thev1/folder rather than the repository name.Schema updates: Added new schemas for dictionary index, commentary books index, and whole-module documents. Updated existing schemas to reflect plain text only and new cross-reference fields.
Utility functions: Added
search_key()for accent-insensitive search term folding,hash_tree()to digest all generated documents, and removed SHA sidecar generation.Implementation Details
_Stageddataclass holds entry metadata during the two-pass dictionary build process._MarkupStripperuses Python'sHTMLParserto extract text while suppressing script/style content and normalizing whitespace around block elements.(record, metadata)instead of a single summary dict.