Skip to content

Report APFS clone sharing and estimate private-byte savings - #3

Closed
huntharo wants to merge 20 commits into
mainfrom
feat/apfs-clone-accounting
Closed

huntharo wants to merge 20 commits into
mainfrom
feat/apfs-clone-accounting

Conversation

@huntharo

@huntharo huntharo commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

APFS clones have separate inodes, so Disktree's hardlink deduplication does not recognize their shared blocks. An 8 MiB file and a cp -c clone remain 16 MiB in the scan, but deleting one copy may release no file data. Previously, review and permanent-delete confirmation projected the full listed size as savings.

This ports the clone accounting from huntharo/diskhound#4 while preserving Disktree's allocated-byte totals and treemap weights. The accounting implementation is based on upstream main at 6c8d4ce, with the feature changes in aa4f168 and Windows lint fix in 786ae6d. Current head b0e48fb also incorporates the expanded main-only queue/rendering fixes from fork PR #9. APFS measurement, estimates and removal authorization remain unchanged by that integration. Upstream #53 provides native bulk metadata reads and a smaller node layout; clone sharing and private-byte estimates remain additional functionality here.

  • macOS enumeration reads clone IDs, real APFS device IDs, and sharing flags in the native bulk listing. Only candidates pay for private-size/reference-count queries. A bounded per-scan cache has 64 mutex shards, holds no lock during filesystem queries, and reuses only confirmed full clones within their observed reference count. Reuse also checks the bulk full-sharing flag and all-fork allocation; hardlink aliases do not spend another clone-reference slot. Modified and unknown observations are measured individually.
  • Sharing reports keep listed allocation, repeated data-fork allocation, and private data-fork bytes separate. Groups are keyed by real device and clone ID. Missing real identity cannot group unrelated volumes. Group and exceptional hardlink maps are bounded; reaching a cap reports partial accounting and drops unproved shared credit.
  • Estimates evaluate accepted marks together. Complete clone groups credit their shared data once. A retained hardlink prevents crediting its inode, and followed links cannot credit their targets. Modified or unknown sharing and non-data-fork allocation produce a range; only its conservative lower estimate enters the scanned volume's meter. Tile weights and listed byte totals remain unchanged, including when the user switches to file-count tiles.
  • The main-only marking/rendering fixes submitted separately as upstream PR #105 are incorporated here; fork PR Fix quadratic marking, path resolution, and removal review #9 is closed as superseded. Accounting uses its batch resolver and caches display estimates against the immutable tree, root, and accepted paths. Single marked subtrees use precomputed summaries. Removal guards and worker authorization remain fresh; no deletion authorization is cached.
  • Unsupported optional attributes fall back to ordinary bulk listing; a rejected reference-count attribute retries without it. Apparent-size scans skip APFS clone attributes. Snapshots, open files, and changes after the scan can retain space; the final outcome measures actual volume free-space change. Snapshot inventory remains outside this port.

Current selected-set behavior

Captured from committed aa4f168 on 2026-10-14 using the same synthetic fixture as the matched comparison below. No removal was performed.

Accepted marks Listed allocation marked Estimated space freed
Two clone copies, retaining the original 16 MiB 0 B
Original plus both clone copies 24 MiB 8 MiB once

Original retained:

Current review: two clone copies remain 16 MiB listed with zero estimated savings

Complete clone group selected across separate marks:

Current review: 24 MiB listed across two marks estimates 8 MiB freed once

Accounting and scaling evidence

Durable accounting evidence includes two failing baseline regressions and passing correctness/budget tests. The baseline claimed 4 KiB for one retained hardlink name and zero for a complete 80-byte synthetic clone group; the fixes return zero and 80 bytes, respectively.

For the growing selected-set workload, 128 to 1,024 groups means 256 to 2,048 marked directories and 512 to 4,096 file entries. Accounting work grows from 1,790 to 14,334 operations (8.01x), below both the 16x ratio budget and absolute budget. Full-clone cache tests require 128 measurement callbacks for 128 groups x 128 members and 1,024 callbacks for 1,024 groups x 1,024 members. These deterministic counts are not elapsed-time benchmark results.

Integrated main-only queue and rendering fixes

Upstream PR #105 is the separate, ready main-only review unit, consolidated into commit 11ff1d98 with unchanged reviewed runtime code. It supersedes closed fork PR #9. This APFS branch already incorporates the equivalent fixes from their earlier commits; the comparison currently includes that shared performance work. Rebasing after tobi#105 lands will remove it from the APFS diff. Evidence commit IDs below identify retained historical snapshots, including failing budgets before the fixes and passing budgets afterward. Expanded evidence preserves the baselines, raw logs and fixture qualifications.

A mark is one independently queued file or folder, regardless of descendant count. The 1,024-mark cases are large-queue stress workloads relevant to programmatic control; there is no usage evidence establishing them as common manual use. These changes add no MCP endpoint.

The original fixes batch path resolution, cache display crumbs, and sort guard-normalized paths for nesting. The expansion replaces the private mark hash index with a native-component BTreeSet, visits ancestors instead of scanning all marks, absorbs descendants with one stable compaction, and builds Review status indexes from each fresh plan. Disjoint scalar construction is O(M D log M), parent absorption is O(M + K log M) plus ancestor lookup, and Review row-status work is expected O(B + C + L), with L <= 1,200. Exact membership trades expected O(1) for O(log M). Removal guards are still freshly evaluated, and the APFS display-estimate cache does not authorize removal.

Work at 1,024 marks Before expansion Expanded fixes
Queue construction, all scalar additions 1,048,576 logical units 9,216
One parent absorption 528,899 logical units 2,058
Blocked Review status, per render 524,800 comparisons 3,072 status units
Covered Review status, per render 525,824 comparisons 3,074 status units

The counters measure logical requests/visits, not CPU instructions or internal B-tree comparisons; the complexity bound includes the logarithmic index factor. Separate file/folder insertion budgets fail before and pass after. Both Review cases also run at 8,192 entries, using 10,592 status units per render while building exactly 1,200 rows. Blocked fixtures queue paths outside the root; covered fixtures deliberately supply overlapping raw queues that normal manual marking would absorb. No removal is attempted.

Three-run medians against pre-expansion PR #9 at 1,024 marks: constructing the whole queue improves from 176.13 to 6.41 ms, and one parent absorption from 15.60 to 0.47 ms. Four forced Review draws/eight observed renders improve from 1,003.88 to 883.69 ms for blocked rows and 1,110.66 to 1,009.63 ms for covered rows. Row layout and fresh guard evaluation still dominate much of Review's time; eliminating its quadratic join does not establish a frame-rate guarantee.

Actual input replay and denominator

The earlier 6.60-second result measured four forced Explore draws/eight prepares on main, not a scan, single click, or time to select 1,024 paths. The fresh main-versus-expanded replay below uses the same fixed on-disk fixture: 1,025 folders, each containing one file. Space adds one more folder; hover and C then use M+1 marks. Each cell is the three-repetition median, main → expanded main-only fixes:

Already queued Space Hover afterward Open Review afterward
1 3.79 → 3.76 ms 3.94 → 3.87 ms 3.47 → 3.43 ms
8 5.14 → 5.01 ms 5.28 → 5.14 ms 5.82 → 5.74 ms
128 41.54 → 16.13 ms 40.85 → 16.26 ms 62.99 → 38.19 ms
1,024 1,584.73 → 96.49 ms 1,575.83 → 97.78 ms 1,786.18 → 300.71 ms

Small queues show no meaningful speedup in this fixture. These are synthetic test-profile observations on the isolated main-only snapshots, not release-app per-click guarantees or timings of this combined APFS branch. Inputs cause two or three harness renders and zero scan refreshes. Costs recur during interaction, including pointer movement within a hovered tile; they are not paid only once per scan. The opt-in replay also passes on this combined branch.

Remaining audit cases

The broader audit is a historical record from before this queue expansion. Its construction, absorption and Review joins are fixed above. Depth-dependent breadcrumb prefix walks, pairwise Linux mount setup, and the finite case-sensitive node_modules classification case remain deferred. The mount probe used synthetic Linux metadata on macOS; the classification case has at most 2,048 case variants and is not generic arbitrary-folder quadratic behavior.

Actual temporary flat/nested scans showed linear listed-entry and aggregate-node visits, with per-directory sorting O(F log F). Normal squarify input is bounded to 96 children plus one tail, despite superlinear uncapped row rescans. Windows MFT extension-record membership remains a source-only, unmeasured O(E²) candidate in extension count, not file count. Fresh guards, sorting, row layout, path-length costs, and O(M) vector compaction for a scalar unmark remain; arbitrary sequences of scalar edits are not promised one-pass total work. Existing budgets protect the exercised paths rather than every application input.

Matched before/after screenshots

Captured on 2026-10-14 from unmodified upstream 6c8d4ce and the rebased PR e9b4996, using the same synthetic APFS fixture, window size, selection, and permanent-removal mode. The fixture contains an 8 MiB original, two cp -c clones in copies, and a 4 MiB ordinary file. Only copies is marked; the original is retained. These screenshots show the review state; no removal was performed.

Observation Before: upstream After: this PR
Listed scan allocation 28 MiB 28 MiB
Marked clone copies 16 MiB 16 MiB
Projected / estimated space freed 16 MiB approximately 0 B
Clone allocation reported separately Unavailable 24 MiB, with 16 MiB counted again

Before: upstream projects the full marked allocation as freed space.

Before: upstream review projects 16 MiB freed for the two clone copies

After: the marked allocation stays 16 MiB, and estimated savings exclude the shared blocks.

After: clone-aware review retains 16 MiB marked and estimates approximately 0 B freed

Matched treemap screenshots: listed sizes stay unchanged

Before — upstream:

Before: upstream treemap lists 28 MiB without clone sharing information

After — this PR:

After: treemap retains 28 MiB and reports clone sharing separately

Images were uploaded as GitHub attachments with gh 2.102.0 and gh pr edit --attach. Local originals and exact source revisions are retained for a future upstream submission.

Full-home before/after validation

The performance results in this section and the earlier ~/github timing comparison below are historical measurements of 77c91ad, before integration with upstream's native bulk scanner. They do not measure the current rebased implementation.

Six sequential scans of the accessible portion of the home directory compare unmodified main (158f9cc) with this PR (77c91ad). Both release builds used the same resolved dependency versions, Rust 1.97.1, default ScanOptions, and the same macOS 26.6.1 / Apple-silicon host. Source snapshots, build output, and benchmark reports were placed outside home. Run order was before, after, after, before, before, after; this was a live filesystem without cache flushing or a frozen snapshot.

The full-home workload is slower and uses more memory with this PR. The small ~/github measurement below does not predict this workload, where approximately half the scanned file entries are clones.

Metric Main before PR after Change
Median scan time, three runs 116.72 s 168.17 s +44.1%
Scan-time range 116.10–122.08 s 166.25–169.95 s
Median peak resident memory 3.476 GiB 4.939 GiB +42.1%
File entries, last run of each mode 18,291,797 18,291,801 +4
Directories, last run of each mode 3,970,802 3,970,805 +3
Listed allocation, last run of each mode 1,409.925293 GiB 1,409.926685 GiB +1.43 MiB
Scan errors 169–171 152 on every run Coverage remains incomplete

The initial before scan listed about 1.11 GiB less than later before scans, so that initial size difference is not solely a before/after scanner difference. The final pair differs by only four file entries, three directories, and 1.43 MiB. All 50 retained diagnostics in each run are Operation not permitted; the scanner caps error detail at 50, so these logs cannot fully explain the differing total error counts. These figures do not establish identical coverage or complete home-directory usage.

Clone accounting in the last PR run:

Figure Result
Cloned-file entries 9,081,529
Listed allocation in potentially shared files 635.06 GiB
Allocation counted again in detected full-clone groups 362.05 GiB
Reported private allocation in cloned files 435.82 MiB
Detected clones with unknown private size 0
Clone-group limit reached No

The clone count and 388,747,776,000-byte repeated-allocation figure were identical across all three PR runs. Clone private allocation varied from 430.76 to 435.82 MiB as the live tree changed.

Listed allocation remains about 1,409.93 GiB. Subtracting the detected full-clone repetition yields 1,047.88 GiB, but that is not an exact physical-space figure: partial sharing, references outside the scan, snapshots, and inaccessible paths remain relevant. Applying the private-only savings formula to the whole scanned tree gives 775.29 GiB; this is an accounting diagnostic, not an amount the application permits deleting. The home directory is protected, and actual marked-path estimates use only accepted removal targets. No home files were deleted, so these runs do not measure actual reclaimed space.

All home scan results
Run Mode Scan seconds Peak RSS, GiB File entries Directories Listed bytes Errors
1 before 122.084 3.475 18,286,927 3,970,536 1,512,708,968,448 170
2 after 166.247 4.948 18,291,801 3,970,805 1,513,892,573,184 152
3 after 168.173 4.905 18,291,801 3,970,805 1,513,893,552,128 152
4 before 116.717 3.476 18,291,794 3,970,793 1,513,895,047,168 169
5 before 116.104 3.477 18,291,797 3,970,802 1,513,895,755,776 171
6 after 169.945 4.939 18,291,801 3,970,805 1,513,897,250,816 152

Scan time runs from ScanHandle::spawn to receipt of the completed tree, with a 10 ms polling interval. Peak RSS is measured by /usr/bin/time -l. Each process uses the same harness against its exact source snapshot; reporting happens after the scan timer stops. Private paths from permission diagnostics and arbitrary home-directory names are omitted here.

Evidence

Real APFS fixtures on macOS 26.6.1, Apple silicon:

Case Listed bytes Sharing result
8 MiB original + one full clone 16 MiB 8 MiB counted again; 0 private bytes
Modify one clone; add a hardlink and a followed symlink Allocated-byte accounting retained Two clone files; private allocation detected; no full-clone duplicate allocation
Demo: original + two clones + 4 MiB ordinary file 28 MiB 24 MiB in clones; 16 MiB counted again
Mark the demo's two clone copies 16 MiB listed ≈ 0 B estimated savings
Replace a marked clone with an ordinary file and refresh 8 MiB listed Estimate updates from 0 B to 8 MiB

The treemap preserves listed sizes and reports sharing separately:

APFS scan totals and selection estimate

Review retains the marked size while showing the private-byte estimate:

APFS clone removal review

The permanent-delete confirmation uses the same estimate:

APFS clone deletion confirmation

Validation

  • cargo xtask lint: passed.
  • cargo xtask test: 266 passed (89 app/window, 174 core, 3 xtask); 2 existing core tests and the opt-in interaction timing test ignored.
  • cargo test -p disktree-app marked_interaction_timings -- --ignored --nocapture --test-threads=1: passed explicitly on the combined branch on retry; the first attempt aborted when GPUI detected filesystem-watcher thread activity as nondeterministic. Local replay timings are not used for the isolated comparisons above.
  • cargo build -p disktree-app: passed; the committed build was visually checked against the APFS fixture.
  • Real APFS coverage includes complete/modified clones, hardlinks, followed links and private resource-fork allocation. Synthetic tests cover missing metadata/identity, limits, cross-directory/set grouping, cache budgets and volume attribution. The Node budget remains 104 bytes.
  • The window test marks separate clone paths, switches tile metric, draws review and confirmation, verifies warm plan reuse and selection changes, then replaces the scan tree and checks refreshed savings.
  • No new home-directory benchmark was run. Earlier performance numbers below and in the home section are historical, and do not measure the cache or accepted-set changes in aa4f168.

Historical developer-tree scan

A release-mode scan of the current ~/github tree produced identical totals in both modes: 159,883 files and 26,266,083,328 listed bytes. Enabled runs detected 37,502 clone files, with no unknown private sizes.

Mode Three successive alternating runs Median
Clone accounting disabled 1.862 / 1.554 / 1.519 s 1.554 s
Clone accounting enabled 1.314 / 1.088 / 0.952 s 1.088 s

This compares this branch's bulk enumeration plus clone accounting against its ordinary-listing fallback; it does not isolate the cost of private-size calls or compare against DiskHound. It is a cached developer-tree measurement, not a whole-volume benchmark.

Reproduce the scanner comparison
cargo build --release -p disktree-core --example scan_accounting
target/release/examples/scan_accounting ~/github --no-clones
target/release/examples/scan_accounting ~/github

Clone metadata is enabled by default. The example's --no-clones option sets ScanOptions::apfs_clone_metadata to false.

@huntharo
huntharo force-pushed the feat/apfs-clone-accounting branch from b63cd8b to e9b4996 Compare October 14, 2026 16:17
Keep remote selections outside local clone estimates and retain both SSH modules and the batched marks resolver. Default the new serialized clone option for existing SSH requests. Align the chooser test with canonical roots, use a fixed test palette, and guard Unix-only SSH test helpers on Windows.
Keep the remote-root test independent of host path syntax. The fixture stays nonexistent locally so acceptance still proves local directory validation is skipped on Windows as well as Unix.
Keep final hardlink totals and received progress counters checked on every platform. Require live preview records from the Unix streaming walker while asserting their absence from the Windows completed-tree fallback.
huntharo added a commit to pwrdrvr/PwrDisk that referenced this pull request Oct 30, 2026
Port clone metadata, bounded per-scan reuse, accepted-set savings, and separate UI estimates from huntharo#3 at 87c02c7. Reuse the marking and review performance fixes already in PwrDisk main. Preserve PwrDisk branding, packaging, remote selection boundaries, and fresh removal guards.
@huntharo huntharo closed this Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant