perf(docm): size the docs cache from recorded checkout sizes - #103
Merged
Merged
Conversation
Every doctor run re-walked every file under every materialized checkout to recompute a number that barely moves. A checkout is a detached worktree pinned at a fixed commit, so its size is known at the one moment it is cheap to take: right after resolve materializes the tree, verifies it clean, and while the library lock is already held. WorktreeMeta gains an optional `bytes`, recorded there and back-filled by upgrade for a cache that predates it. doctor substitutes the records into the walk rather than composing a total out of parts, so the shared object stores, sidecars and anything a broken cache left behind are all still counted, and a record outliving its checkout adds nothing. The record is a snapshot, so it goes stale when a checkout grows, and growth the repository ignores escapes the cleanliness sweep too. `docm list --refresh` re-measures under the library lock, doctor names the flag, and `docm list` now shows size per version - which is what says where to point the filters issue #31 wants. The sweep also drops to one git process per checkout: porcelain v2 carries the full HEAD oid in its --branch header, so the commit comparison reads off the same output as the dirty check. Refs #102 Claude-Session: https://claude.ai/code/session_01TXnj77YPcNHmatshb47AJN Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…again Review found the record was written unconditionally, so every `docm path` and `docm info` - which re-resolve on each invocation and almost always find the checkout already at its pinned commit - paid a full walk of the tree. That put a walk on the commands run most to take one off `devkit doctor`, which runs rarely. A checkout this resolve neither created nor re-pointed now keeps the number it already recorded. `repaired` alone cannot say which case it is, so the directory is probed before ensure_at. Also from review: - `human_size` had landed between `walk` and its doc comment, filing the directory-read and jwalk reasoning on a string formatter. Split back. - `docm list --refresh` under `--project` returned before the refresh ran, accepting a mutating flag and doing nothing. Now refused by clap. - One unparsable sidecar aborted the whole listing. `--refresh` is what a reader reaches for when the cache is already suspect, so a failure is a line on stderr and the table still prints. - `dir_size_with_known` substituted a recorded root without walking it, which its own doc said could not happen. The root is always walked now. - v2 marks untracked and ignored entries with one character where v1 wrote `??` and `!!`. Doubled back, so the column stays two wide. - The doctor test proved "the rest of the cache is still walked" only because the fixture object store outweighs a two-file checkout. It now plants a directory of known size that no record covers. - `inspect` ending on a failed `git status` is a behaviour change from the separate `rev-parse`; said so at the site. Refs #102 Claude-Session: https://claude.ai/code/session_01TXnj77YPcNHmatshb47AJN Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
AbysmalBiscuit
marked this pull request as ready for review
September 18, 2026 17:33
AbysmalBiscuit
force-pushed
the
102-docm-record-checkout-sizes
branch
from
September 18, 2026 17:35
03d014a to
5be9d25
Compare
Refresh cached libraries even when the current manifest omits them, so the refresh command covers the cache counted by doctor. Share size measurement with backfill, correct inspection documentation, and cover refresh output, scope, and flag validation through the CLI. Co-authored-by: Codex <noreply@openai.com>
AbysmalBiscuit
force-pushed
the
102-docm-record-checkout-sizes
branch
from
September 18, 2026 17:44
5be9d25 to
831317b
Compare
AbysmalBiscuit
added a commit
that referenced
this pull request
Sep 21, 2026
🤖 I have created a release *beep* *boop* --- ## [0.14.6](v0.14.5...v0.14.6) (2026-09-21) ### Features * **mcp:** add [mcp] enabled switch ([#117](#117)) ([74250a8](74250a8)) * **rules:** inject the rules that govern a file before it is written ([#108](#108)) ([52bca00](52bca00)) ### Bug Fixes * **cloud:** pin setup installs to plugin versions ([#120](#120)) ([3afad97](3afad97)) * **common:** trust the platform certificate store ([#119](#119)) ([a4bdda9](a4bdda9)) * **locks:** hold a claude code fork's writes as its session ([#121](#121)) ([a6c0fe1](a6c0fe1)) ### Performance Improvements * **docm:** size the docs cache from recorded checkout sizes ([#103](#103)) ([2450c4d](2450c4d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR (human written)
Closes #102
Refs #31
Agent summary
Every
devkit doctorrun re-walked every file under every materialized checkout to recompute a number that barely moves.devkit_docs::doctor_summarycalleddir_sizeon the whole cache root, and that was the only place the number came from.Most of what it walked is stable. A library directory is one shared
repo.gitplus a detached worktree per resolved version; the worktrees sit at fixed commits and are verified clean, and only the object store actually grows as new versions are materialized. On a blobless clone of tokio with two version trees,repo.gitwas 9.0 MB of a 19.1 MB total, so roughly half the walked bytes were identical on every run.What changed
cache::WorktreeMetagains an optionalbytes, recorded inresolve::resolveright afterensure_atwrites the tree andassert_cleanverifies it, while the library lock is already held and the tree is still hot in page cache.upgrade::attendback-fills it once for a cache written before this, under the lock it already takes, andneeds_attentionmatches exactly what the back-fill will do, so a cache it cannot finish does not ask for the lock again on every command.doctor_summarysubstitutes the records into the walk through a newdisk::dir_size_with_known, rather than composing a total out of parts. The cache is still walked whole, so the shared object stores, sidecars, lock files and whatever a broken cache has left lying around all still land in the total. A record that outlived its checkout adds nothing, because the substitution happens where the walk meets the entry.docm listnow shows size per version in both the table and--json. That is the number that says which version is the fat one, which is what #31 needs in order to decide where to point its filters.The staleness trade
A record is a snapshot of a tree pinned at a fixed commit, so it goes stale when a checkout grows, and the growth most likely to matter escapes the cleanliness sweep as well:
git status --porcelaindoes not report ignored files, so atarget/ornode_modules/landing in a checkout shows up in neither the record nor the local-modifications check. That gap is not new, but nothing here pretends otherwise.docm list --refreshre-measures every checkout under its library lock and writes the numbers back, and doctor names the flag in its docs-cache row so a reader who doubts the number knows where to go. Refreshing lives onlistrather than ondoctorbecause doctor diagnoses and does not mutate the cache, and its sweep deliberately takes no library lock.Also
inspectspawned two git processes per checkout,git status --porcelainthengit rev-parse HEAD.git status --porcelain=v2 --branchanswers both in one: its header carries the full HEAD oid, so the recorded-commit comparison reads off the same output as the dirty check. Entry lines are reduced to the status letters and the path the way v1 rendered them, unrecognized record types yield nothing so a format git grows later cannot become a phantom modification, and(initial)is reported as a missing commit rather than compared against a sha.assert_cleanonly needs dirtiness and stays on v1.Measured on this box that saves about 1 ms per checkout, which is the spawn cost of
rev-parseitself. The case for it is Windows, where process creation is far more expensive, and where CI runs.Verification
devrun task test(2262 tests, 2259 passing),test-doc,lintclean,fmtapplied.Three failures are pre-existing on
mainin this container and unrelated to the diff, confirmed by running the suite on a clean tree:devkit::install_links a_failed_pass_is_partial_and_retries_once_the_cooldown_elapsesand twodevkit-common git::testspermission fixtures, all of which need a non-root user for anything to be denied.🤖 Generated with Claude Opus 5 via Claude Code
https://claude.ai/code/session_01TXnj77YPcNHmatshb47AJN
Generated by Claude Code