Docs: split the README into an introduction and a reference manual

[flavorjones]

Motivation

The README ran to 681 lines and covered everything from the pitch to the log schema, and discussion #86 called it verbose. Each file in docs/ held rationale, how-to steps and settings tables together, and some facts appeared in several files: the OpenMP bound was in the README, docs/DEPLOYMENT.md and docs/IMAGEMAGICK.md, and the timeout rule was in both docs/DEPLOYMENT.md and docs/TUNING.md.

Nothing tied a page to the code it described. docs/DESIGN.md records that an earlier API description went stale and its errors spread back into code comments.

Details

Layout. The README is a human introduction: what Hot Cell is, an Active Storage quick start, a short custom-operation example, and one line per recommended alert, each linking to a reference page. Everything else is a reference manual under docs/, one topic per page, written in Google developer documentation style. docs/DESIGN.md is split into docs/design/ and docs/contributing/, with the invariant and experiment numbers unchanged. docs/contributing/ is for people working on Hot Cell itself: contributing, keeping the docs current, the experiments and overhead measurements, and the decision records. CONTRIBUTING.md and adr/README.md point there.

Frontmatter. Each page under docs/ starts with Open Knowledge Format (OKF) frontmatter: type, title, a one-line description, and sources, the files or directories that the page describes. Each section's index.md lists its pages with their descriptions, so an agent can choose a page without opening the others.

Tooling. Three rake tasks in rakelib/docs.rake:

• rake docs:stale lists each page whose sources changed since BASE (default origin/master) while the page did not. It never fails. A new Docs CI job prints the list as warnings on a pull request.
• rake docs:index regenerates the page list between the <!-- index --> markers in each index.md. Pages with an order in their frontmatter come first, in that order; the rest follow by file name.
• rake docs:check fails on a page without type, title or description, on a source that does not exist, and on an out-of-date index. The Docs job runs it on every push, and so does rake.

New docs_test.rb files in hotcell-core, hotcell-server and activestorage-hotcell-server fail when the codes and kill-causes table, the log events table, the cell defaults tables or the shipped operation limits table disagree with the code.

During development. After changing code, run rake docs:stale, read each page it names, and fix what the change made wrong. Edit sources when a page starts or stops describing a file, and run rake docs:index after adding a page or changing a title or description. docs/contributing/docs.md says the same. AGENTS.md includes that page and gives the writing rules for each kind of page.

Corrections. The README's Active Storage initializer did not call HotCell.register "active_storage", which every shipped client needs. docs/DEPLOYMENT.md said that an unset HotCell.root runs callers in process; perform_in_hotcell raises HotCell::CellNotConfigured. Both are fixed.

Additional information

The commits are meant to be reviewed one at a time:

1. Moves every section, unchanged, into its new file, and repoints the code comments that cited the old paths. A script assigned every source line to exactly one destination; only the tables of contents were dropped.
2. Rewrites the README and the reference pages, merges duplicates, and adds reference that no page had: response codes, HotCell::Operation, Input and Output, and the client's boot checks. The design pages keep their prose; only headings and links changed.
3. Adds the frontmatter, the rake tasks, the CI job, the tests and AGENTS.md.
4. Restores details that the rewrite dropped, found by an adversarial review and by comparing every old sentence with the new pages.
5. Rewrites the design pages to describe the design as it is. It drops withdrawn invariant 1 without renumbering the rest, a deferred check, a socket check that was tried and withdrawn, and the history of reversed decisions.
6. Adds the docs/contributing/ section.
7. Orders the reference and design indexes, renames awkward page titles, rewrites the design introduction, and spells the name Hot Cell in prose.
8. Renames the contributor section to Contributing to Hot Cell, in docs/contributing/, and gives its pages type: Contributing.
9. Presents the reference manual, Design, and Contributing to Hot Cell as sibling sections: an index no longer lists the indexes below it, and the README and AGENTS.md list each one.
10. Corrects the statements that Copilot's review found wrong: scratch sizing, how a full scratch is classified, file_size being removable, the supervisor reading a request, and the operation example's output size.

The README is 331 lines; most of what remains is the Kamal configuration, which a reader copies.

[Copilot]

Copilot review overview

🟡 Changes recommended

Several operational rules and API examples contradict the implementation and could cause failures or unsafe scratch sizing.

Review effort: Balanced
Findings: 13

Open (13)

• Clarify permanent verdicts can be fixed by deployment tuning · New
• Qualify claim that supervisor never reads requests · New
• Correct nonexistent scratch subdirectory layout · New
• Account for multiple files in scratch sizing · New
• Avoid marking fd_path output as staged via Output#path · New
• Document worker.undispatchable request inspection exception · New
• Do not promise retries for all full-filesystem failures · New
• Avoid underprovisioning scratch for multi-file requests · New
• Include all concurrent request files in scratch sizing · New
• Document per-file rather than uncapped file_size limits · New
• Qualify full-filesystem failure classification · New
• Size scratch for aggregate per-request usage · New
• Describe operation limits as conditional failure thresholds · New

What changed in this PR

Splits the monolithic documentation into a concise README and source-linked reference manual, with validation tooling and synchronization tests.

Changes:

• Adds topic-focused reference and design pages.
• Adds documentation indexing, staleness checks, CI, and table synchronization tests.
• Updates code comments and contributor guidance to reference the new structure.

Pull request overview

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File	Description
.github/​workflows/​ci.yml	Adds documentation CI.
AGENTS.md	Documents the reference workflow.
CHANGELOG.md	Records the documentation restructure.
CONTRIBUTING.md	Updates documentation guidance.
README.md	Becomes the introduction and quick start.
Rakefile	Adds documentation validation to defaults.
activestorage-hotcell-server/​test/​docs_test.rb	Verifies documented operation limits.
bin/​conformance	Updates documentation references.
docs/​DEPLOYMENT.md	Removes the former deployment guide.
docs/​DESIGN.md	Removes the monolithic design document.
docs/​IMAGEMAGICK.md	Replaced by the new reference page.
docs/​LOGS.md	Replaced by observability documentation.
docs/​TUNING.md	Replaced by the rewritten tuning page.
docs/​active-storage.md	Documents Active Storage operations.
docs/​cell-settings.md	Documents cell configuration.
docs/​client-api.md	Documents the client API.
docs/​codes.md	Documents response classifications.
docs/​concepts.md	Defines core terminology.
docs/​conformance.md	Documents image conformance checks.
docs/​container.md	Documents container configuration.
docs/​design/​descriptors.md	Records descriptor design rationale.
docs/​design/​experiments.md	Preserves numbered experiments.
docs/​design/​index.md	Indexes design documentation.
docs/​design/​invariants.md	Preserves numbered invariants.
docs/​design/​overhead.md	Records overhead findings.
docs/​design/​threat-model.md	Separates the threat model.
docs/​design/​worker-isolation.md	Documents worker isolation.
docs/​imagemagick.md	Rewrites ImageMagick guidance.
docs/​index.md	Indexes the reference manual.
docs/​observability.md	Consolidates logs and metrics guidance.
docs/​operation-api.md	Documents operation and descriptor APIs.
docs/​request-lifecycle.md	Documents request processing.
docs/​scratch.md	Documents scratch layouts and sizing.
docs/​tuning.md	Rewrites tuning guidance.
examples/​gate	Updates documentation references.
hotcell-client/​lib/​hot_cell/​client.rb	Updates client API reference.
hotcell-client/​test/​describe_survival_test.rb	Updates design reference.
hotcell-core/​lib/​hot_cell/​codes.rb	Updates isolation reference.
hotcell-core/​test/​docs_test.rb	Verifies documented response codes.
hotcell-server/​lib/​hot_cell/​log.rb	Updates observability reference.
hotcell-server/​lib/​hot_cell/​operation.rb	Updates operation reference.
hotcell-server/​lib/​hot_cell/​slot.rb	Updates isolation reference.
hotcell-server/​lib/​hot_cell/​supervisor.rb	Updates design and logging references.
hotcell-server/​test/​docs_test.rb	Verifies events and defaults.
hotcell-server/​test/​log_test.rb	Updates log schema reference.
rakelib/​docs.rake	Adds index, check, and stale tasks.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.