Repository navigation
Conversation
huntharo
force-pushed
the
feat/apfs-clone-accounting
branch
from
October 14, 2026 16:17
b63cd8b to
e9b4996
Compare
Keep remote selections outside local clone estimates and retain both SSH modules and the batched marks resolver. Default the new serialized clone option for existing SSH requests. Align the chooser test with canonical roots, use a fixed test palette, and guard Unix-only SSH test helpers on Windows.
Keep the remote-root test independent of host path syntax. The fixture stays nonexistent locally so acceptance still proves local directory validation is skipped on Windows as well as Unix.
Keep final hardlink totals and received progress counters checked on every platform. Require live preview records from the Unix streaming walker while asserting their absence from the Windows completed-tree fallback.
huntharo
added a commit
to pwrdrvr/PwrDisk
that referenced
this pull request
Oct 30, 2026
Port clone metadata, bounded per-scan reuse, accepted-set savings, and separate UI estimates from huntharo#3 at 87c02c7. Reuse the marking and review performance fixes already in PwrDisk main. Preserve PwrDisk branding, packaging, remote selection boundaries, and fresh removal guards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
APFS clones have separate inodes, so Disktree's hardlink deduplication does not recognize their shared blocks. An 8 MiB file and a
cp -cclone remain 16 MiB in the scan, but deleting one copy may release no file data. Previously, review and permanent-delete confirmation projected the full listed size as savings.This ports the clone accounting from huntharo/diskhound#4 while preserving Disktree's allocated-byte totals and treemap weights. The accounting implementation is based on upstream
mainat6c8d4ce, with the feature changes inaa4f168and Windows lint fix in786ae6d. Current headb0e48fbalso incorporates the expanded main-only queue/rendering fixes from fork PR #9. APFS measurement, estimates and removal authorization remain unchanged by that integration. Upstream #53 provides native bulk metadata reads and a smaller node layout; clone sharing and private-byte estimates remain additional functionality here.Current selected-set behavior
Captured from committed
aa4f168on 2026-10-14 using the same synthetic fixture as the matched comparison below. No removal was performed.Original retained:
Complete clone group selected across separate marks:
Accounting and scaling evidence
Durable accounting evidence includes two failing baseline regressions and passing correctness/budget tests. The baseline claimed 4 KiB for one retained hardlink name and zero for a complete 80-byte synthetic clone group; the fixes return zero and 80 bytes, respectively.
For the growing selected-set workload, 128 to 1,024 groups means 256 to 2,048 marked directories and 512 to 4,096 file entries. Accounting work grows from 1,790 to 14,334 operations (8.01x), below both the 16x ratio budget and absolute budget. Full-clone cache tests require 128 measurement callbacks for 128 groups x 128 members and 1,024 callbacks for 1,024 groups x 1,024 members. These deterministic counts are not elapsed-time benchmark results.
Integrated main-only queue and rendering fixes
Upstream PR #105 is the separate, ready main-only review unit, consolidated into commit
11ff1d98with unchanged reviewed runtime code. It supersedes closed fork PR #9. This APFS branch already incorporates the equivalent fixes from their earlier commits; the comparison currently includes that shared performance work. Rebasing after tobi#105 lands will remove it from the APFS diff. Evidence commit IDs below identify retained historical snapshots, including failing budgets before the fixes and passing budgets afterward. Expanded evidence preserves the baselines, raw logs and fixture qualifications.A mark is one independently queued file or folder, regardless of descendant count. The 1,024-mark cases are large-queue stress workloads relevant to programmatic control; there is no usage evidence establishing them as common manual use. These changes add no MCP endpoint.
The original fixes batch path resolution, cache display crumbs, and sort guard-normalized paths for nesting. The expansion replaces the private mark hash index with a native-component
BTreeSet, visits ancestors instead of scanning all marks, absorbs descendants with one stable compaction, and builds Review status indexes from each fresh plan. Disjoint scalar construction is O(M D log M), parent absorption is O(M + K log M) plus ancestor lookup, and Review row-status work is expected O(B + C + L), with L <= 1,200. Exact membership trades expected O(1) for O(log M). Removal guards are still freshly evaluated, and the APFS display-estimate cache does not authorize removal.The counters measure logical requests/visits, not CPU instructions or internal B-tree comparisons; the complexity bound includes the logarithmic index factor. Separate file/folder insertion budgets fail before and pass after. Both Review cases also run at 8,192 entries, using 10,592 status units per render while building exactly 1,200 rows. Blocked fixtures queue paths outside the root; covered fixtures deliberately supply overlapping raw queues that normal manual marking would absorb. No removal is attempted.
Three-run medians against pre-expansion PR #9 at 1,024 marks: constructing the whole queue improves from 176.13 to 6.41 ms, and one parent absorption from 15.60 to 0.47 ms. Four forced Review draws/eight observed renders improve from 1,003.88 to 883.69 ms for blocked rows and 1,110.66 to 1,009.63 ms for covered rows. Row layout and fresh guard evaluation still dominate much of Review's time; eliminating its quadratic join does not establish a frame-rate guarantee.
Actual input replay and denominator
The earlier 6.60-second result measured four forced Explore draws/eight prepares on main, not a scan, single click, or time to select 1,024 paths. The fresh main-versus-expanded replay below uses the same fixed on-disk fixture: 1,025 folders, each containing one file. Space adds one more folder; hover and C then use M+1 marks. Each cell is the three-repetition median, main → expanded main-only fixes:
Small queues show no meaningful speedup in this fixture. These are synthetic test-profile observations on the isolated main-only snapshots, not release-app per-click guarantees or timings of this combined APFS branch. Inputs cause two or three harness renders and zero scan refreshes. Costs recur during interaction, including pointer movement within a hovered tile; they are not paid only once per scan. The opt-in replay also passes on this combined branch.
Remaining audit cases
The broader audit is a historical record from before this queue expansion. Its construction, absorption and Review joins are fixed above. Depth-dependent breadcrumb prefix walks, pairwise Linux mount setup, and the finite case-sensitive
node_modulesclassification case remain deferred. The mount probe used synthetic Linux metadata on macOS; the classification case has at most 2,048 case variants and is not generic arbitrary-folder quadratic behavior.Actual temporary flat/nested scans showed linear listed-entry and aggregate-node visits, with per-directory sorting O(F log F). Normal
squarifyinput is bounded to 96 children plus one tail, despite superlinear uncapped row rescans. Windows MFT extension-record membership remains a source-only, unmeasured O(E²) candidate in extension count, not file count. Fresh guards, sorting, row layout, path-length costs, and O(M) vector compaction for a scalar unmark remain; arbitrary sequences of scalar edits are not promised one-pass total work. Existing budgets protect the exercised paths rather than every application input.Matched before/after screenshots
Captured on 2026-10-14 from unmodified upstream
6c8d4ceand the rebased PRe9b4996, using the same synthetic APFS fixture, window size, selection, and permanent-removal mode. The fixture contains an 8 MiB original, twocp -cclones incopies, and a 4 MiB ordinary file. Onlycopiesis marked; the original is retained. These screenshots show the review state; no removal was performed.Before: upstream projects the full marked allocation as freed space.
After: the marked allocation stays 16 MiB, and estimated savings exclude the shared blocks.
Matched treemap screenshots: listed sizes stay unchanged
Before — upstream:
After — this PR:
Images were uploaded as GitHub attachments with
gh2.102.0 andgh pr edit --attach. Local originals and exact source revisions are retained for a future upstream submission.Full-home before/after validation
The performance results in this section and the earlier
~/githubtiming comparison below are historical measurements of77c91ad, before integration with upstream's native bulk scanner. They do not measure the current rebased implementation.Six sequential scans of the accessible portion of the home directory compare unmodified
main(158f9cc) with this PR (77c91ad). Both release builds used the same resolved dependency versions, Rust 1.97.1, defaultScanOptions, and the same macOS 26.6.1 / Apple-silicon host. Source snapshots, build output, and benchmark reports were placed outside home. Run order was before, after, after, before, before, after; this was a live filesystem without cache flushing or a frozen snapshot.The full-home workload is slower and uses more memory with this PR. The small
~/githubmeasurement below does not predict this workload, where approximately half the scanned file entries are clones.The initial before scan listed about 1.11 GiB less than later before scans, so that initial size difference is not solely a before/after scanner difference. The final pair differs by only four file entries, three directories, and 1.43 MiB. All 50 retained diagnostics in each run are
Operation not permitted; the scanner caps error detail at 50, so these logs cannot fully explain the differing total error counts. These figures do not establish identical coverage or complete home-directory usage.Clone accounting in the last PR run:
The clone count and 388,747,776,000-byte repeated-allocation figure were identical across all three PR runs. Clone private allocation varied from 430.76 to 435.82 MiB as the live tree changed.
Listed allocation remains about 1,409.93 GiB. Subtracting the detected full-clone repetition yields 1,047.88 GiB, but that is not an exact physical-space figure: partial sharing, references outside the scan, snapshots, and inaccessible paths remain relevant. Applying the private-only savings formula to the whole scanned tree gives 775.29 GiB; this is an accounting diagnostic, not an amount the application permits deleting. The home directory is protected, and actual marked-path estimates use only accepted removal targets. No home files were deleted, so these runs do not measure actual reclaimed space.
All home scan results
Scan time runs from
ScanHandle::spawnto receipt of the completed tree, with a 10 ms polling interval. Peak RSS is measured by/usr/bin/time -l. Each process uses the same harness against its exact source snapshot; reporting happens after the scan timer stops. Private paths from permission diagnostics and arbitrary home-directory names are omitted here.Evidence
Real APFS fixtures on macOS 26.6.1, Apple silicon:
The treemap preserves listed sizes and reports sharing separately:
Review retains the marked size while showing the private-byte estimate:
The permanent-delete confirmation uses the same estimate:
Validation
cargo xtask lint: passed.cargo xtask test: 266 passed (89 app/window, 174 core, 3 xtask); 2 existing core tests and the opt-in interaction timing test ignored.cargo test -p disktree-app marked_interaction_timings -- --ignored --nocapture --test-threads=1: passed explicitly on the combined branch on retry; the first attempt aborted when GPUI detected filesystem-watcher thread activity as nondeterministic. Local replay timings are not used for the isolated comparisons above.cargo build -p disktree-app: passed; the committed build was visually checked against the APFS fixture.aa4f168.Historical developer-tree scan
A release-mode scan of the current
~/githubtree produced identical totals in both modes: 159,883 files and 26,266,083,328 listed bytes. Enabled runs detected 37,502 clone files, with no unknown private sizes.This compares this branch's bulk enumeration plus clone accounting against its ordinary-listing fallback; it does not isolate the cost of private-size calls or compare against DiskHound. It is a cached developer-tree measurement, not a whole-volume benchmark.
Reproduce the scanner comparison
Clone metadata is enabled by default. The example's
--no-clonesoption setsScanOptions::apfs_clone_metadatato false.