ScanWithAnalysis: Array dimensions exceeded on large NTFS volumes; MaxDepth cannot help #5

Summary

Get-FolderSizes / ScanWithAnalysis fails on very large NTFS volumes with:

Exception calling "ScanWithAnalysis" with "11" argument(s): "Array dimensions exceeded supported range."

Retrying with lower MaxDepth (5 → 3 → 2 → 1) does not help.

Environment

• UltraTree 1.0.1 (PowerShell Gallery)
• NinjaOne agent script calling Get-FolderSizes
• Volume: ~16.7 TB NTFS data drive (H:), nearly full / high file count
• Windows Server, NT AUTHORITY\SYSTEM

Why MaxDepth cannot help

In Module/src/UltraTree/Classes/MftScanner.cs.ps1 ScanWithAnalysis:

1. Loads every USN/MFT record into Dictionary<ulong, MftEntry>(2000000) (around lines 715–747)
2. Materializes all non-directory entries with .ToArray() (around line 780) before maxDepth is applied when folder refs become paths (around line 926)

So depth filtering happens only after the full file list exists in memory.

Likely throw site

var files = fileMap.Values.Where(e => !e.IsDirectory).ToArray();

On tens of millions of files, the second giant array hits the CLR array size cap (OverflowException: "Array dimensions exceeded supported range."). A huge Dictionary bucket array can hit the same limit.

Proposed fix (smallest first)

1. Stop the second giant array — Parallel.ForEach(fileMap.Values, …) and if (entry.IsDirectory) return; instead of .Where(...).ToArray()
2. Catch OverflowException around the MFT path and fall back to existing ScanFallback (already honors maxDepth)
3. Apply maxDepth while walking parent refs, not after the full file list exists
4. Longer term — keep directory entries for path rebuild + a bounded top-N file heap; do not retain every file MftEntry if only folder aggregates are required

Workaround today

Our NinjaOne integrator script falls back to gdu with a SQLite backend and a reduced disk report when UltraTree throws this overflow. An upstream fix would restore full UltraTree HTML (cleanup / file types / duplicates) on large file servers.