Skip to content

perf(docm): size the docs cache from recorded checkout sizes - #103

Merged
AbysmalBiscuit merged 3 commits into
mainfrom
102-docm-record-checkout-sizes
Sep 18, 2026
Merged

AbysmalBiscuit merged 3 commits into
mainfrom
102-docm-record-checkout-sizes

Conversation

@AbysmalBiscuit

Copy link
Copy Markdown
Owner

TL;DR (human written)


Closes #102
Refs #31


Agent summary

Every devkit doctor run re-walked every file under every materialized checkout to recompute a number that barely moves. devkit_docs::doctor_summary called dir_size on the whole cache root, and that was the only place the number came from.

Most of what it walked is stable. A library directory is one shared repo.git plus a detached worktree per resolved version; the worktrees sit at fixed commits and are verified clean, and only the object store actually grows as new versions are materialized. On a blobless clone of tokio with two version trees, repo.git was 9.0 MB of a 19.1 MB total, so roughly half the walked bytes were identical on every run.

What changed

cache::WorktreeMeta gains an optional bytes, recorded in resolve::resolve right after ensure_at writes the tree and assert_clean verifies it, while the library lock is already held and the tree is still hot in page cache. upgrade::attend back-fills it once for a cache written before this, under the lock it already takes, and needs_attention matches exactly what the back-fill will do, so a cache it cannot finish does not ask for the lock again on every command.

doctor_summary substitutes the records into the walk through a new disk::dir_size_with_known, rather than composing a total out of parts. The cache is still walked whole, so the shared object stores, sidecars, lock files and whatever a broken cache has left lying around all still land in the total. A record that outlived its checkout adds nothing, because the substitution happens where the walk meets the entry.

docm list now shows size per version in both the table and --json. That is the number that says which version is the fat one, which is what #31 needs in order to decide where to point its filters.

The staleness trade

A record is a snapshot of a tree pinned at a fixed commit, so it goes stale when a checkout grows, and the growth most likely to matter escapes the cleanliness sweep as well: git status --porcelain does not report ignored files, so a target/ or node_modules/ landing in a checkout shows up in neither the record nor the local-modifications check. That gap is not new, but nothing here pretends otherwise.

docm list --refresh re-measures every checkout under its library lock and writes the numbers back, and doctor names the flag in its docs-cache row so a reader who doubts the number knows where to go. Refreshing lives on list rather than on doctor because doctor diagnoses and does not mutate the cache, and its sweep deliberately takes no library lock.

Also

inspect spawned two git processes per checkout, git status --porcelain then git rev-parse HEAD. git status --porcelain=v2 --branch answers both in one: its header carries the full HEAD oid, so the recorded-commit comparison reads off the same output as the dirty check. Entry lines are reduced to the status letters and the path the way v1 rendered them, unrecognized record types yield nothing so a format git grows later cannot become a phantom modification, and (initial) is reported as a missing commit rather than compared against a sha. assert_clean only needs dirtiness and stays on v1.

Measured on this box that saves about 1 ms per checkout, which is the spawn cost of rev-parse itself. The case for it is Windows, where process creation is far more expensive, and where CI runs.

Verification

devrun task test (2262 tests, 2259 passing), test-doc, lint clean, fmt applied.

Three failures are pre-existing on main in this container and unrelated to the diff, confirmed by running the suite on a clean tree: devkit::install_links a_failed_pass_is_partial_and_retries_once_the_cooldown_elapses and two devkit-common git::tests permission fixtures, all of which need a non-root user for anything to be denied.


🤖 Generated with Claude Opus 5 via Claude Code

https://claude.ai/code/session_01TXnj77YPcNHmatshb47AJN


Generated by Claude Code

AbysmalBiscuit and others added 2 commits September 18, 2026 07:43
Every doctor run re-walked every file under every materialized checkout to
recompute a number that barely moves. A checkout is a detached worktree
pinned at a fixed commit, so its size is known at the one moment it is
cheap to take: right after resolve materializes the tree, verifies it
clean, and while the library lock is already held.

WorktreeMeta gains an optional `bytes`, recorded there and back-filled by
upgrade for a cache that predates it. doctor substitutes the records into
the walk rather than composing a total out of parts, so the shared object
stores, sidecars and anything a broken cache left behind are all still
counted, and a record outliving its checkout adds nothing.

The record is a snapshot, so it goes stale when a checkout grows, and
growth the repository ignores escapes the cleanliness sweep too. `docm
list --refresh` re-measures under the library lock, doctor names the flag,
and `docm list` now shows size per version - which is what says where to
point the filters issue #31 wants.

The sweep also drops to one git process per checkout: porcelain v2 carries
the full HEAD oid in its --branch header, so the commit comparison reads
off the same output as the dirty check.

Refs #102

Claude-Session: https://claude.ai/code/session_01TXnj77YPcNHmatshb47AJN

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…again

Review found the record was written unconditionally, so every `docm path`
and `docm info` - which re-resolve on each invocation and almost always
find the checkout already at its pinned commit - paid a full walk of the
tree. That put a walk on the commands run most to take one off `devkit
doctor`, which runs rarely. A checkout this resolve neither created nor
re-pointed now keeps the number it already recorded. `repaired` alone
cannot say which case it is, so the directory is probed before ensure_at.

Also from review:

- `human_size` had landed between `walk` and its doc comment, filing the
  directory-read and jwalk reasoning on a string formatter. Split back.
- `docm list --refresh` under `--project` returned before the refresh ran,
  accepting a mutating flag and doing nothing. Now refused by clap.
- One unparsable sidecar aborted the whole listing. `--refresh` is what a
  reader reaches for when the cache is already suspect, so a failure is a
  line on stderr and the table still prints.
- `dir_size_with_known` substituted a recorded root without walking it,
  which its own doc said could not happen. The root is always walked now.
- v2 marks untracked and ignored entries with one character where v1 wrote
  `??` and `!!`. Doubled back, so the column stays two wide.
- The doctor test proved "the rest of the cache is still walked" only
  because the fixture object store outweighs a two-file checkout. It now
  plants a directory of known size that no record covers.
- `inspect` ending on a failed `git status` is a behaviour change from the
  separate `rev-parse`; said so at the site.

Refs #102

Claude-Session: https://claude.ai/code/session_01TXnj77YPcNHmatshb47AJN

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@AbysmalBiscuit
AbysmalBiscuit marked this pull request as ready for review September 18, 2026 17:33
@AbysmalBiscuit
AbysmalBiscuit force-pushed the 102-docm-record-checkout-sizes branch from 03d014a to 5be9d25 Compare September 18, 2026 17:35
Refresh cached libraries even when the current manifest omits them,
so the refresh command covers the cache counted by doctor.

Share size measurement with backfill, correct inspection documentation,
and cover refresh output, scope, and flag validation through the CLI.

Co-authored-by: Codex <noreply@openai.com>
@AbysmalBiscuit
AbysmalBiscuit force-pushed the 102-docm-record-checkout-sizes branch from 5be9d25 to 831317b Compare September 18, 2026 17:44
@AbysmalBiscuit
AbysmalBiscuit merged commit 2450c4d into main Sep 18, 2026
12 checks passed
@AbysmalBiscuit
AbysmalBiscuit deleted the 102-docm-record-checkout-sizes branch September 18, 2026 17:51
AbysmalBiscuit added a commit that referenced this pull request Sep 21, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.14.6](v0.14.5...v0.14.6)
(2026-09-21)


### Features

* **mcp:** add [mcp] enabled switch
([#117](#117))
([74250a8](74250a8))
* **rules:** inject the rules that govern a file before it is written
([#108](#108))
([52bca00](52bca00))


### Bug Fixes

* **cloud:** pin setup installs to plugin versions
([#120](#120))
([3afad97](3afad97))
* **common:** trust the platform certificate store
([#119](#119))
([a4bdda9](a4bdda9))
* **locks:** hold a claude code fork's writes as its session
([#121](#121))
([a6c0fe1](a6c0fe1))


### Performance Improvements

* **docm:** size the docs cache from recorded checkout sizes
([#103](#103))
([2450c4d](2450c4d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(docm): size the docs cache from meta.toml instead of re-walking every checkout

1 participant