main.rs overrides fgumi's spill-file consolidation limit unconditionally:
cli.sort.max_temp_files = cli.sort.max_temp_files.or(Some(DEFAULT_MAX_TEMP_FILES)); // 256
That override existed because fgumi's default was a hardcoded 64 that bore no relationship to the descriptor budget the process actually had, so mako raised it to avoid consolidation — which rewrites already-sorted data and is pure overhead when the descriptors were available anyway. On a 1.33B-record WGS sort spilling 84 runs, that was 958s of a 2529s run.
fulcrumgenomics/fgumi#705 fixes this upstream: fgumi sort now sizes the limit from the process's soft RLIMIT_NOFILE (clamp(soft - 32, 16, 256)), and --max-temp-files gained an explicit auto value which is the default. Since mako flattens fgumi's Sort struct and dispatches through the same command, it inherits the sizing for free once it picks up that fgumi.
Why the override should go, not just become redundant
It is not merely superfluous after #705 — it is worse than the new default on a low-ulimit host. mako asks for 256 unconditionally. On a stock macOS host (ulimit -n = 256) the usable budget is 256 - 32 = 224, so a mako sort that spills more than 224 runs hits EMFILE, while plain fgumi sort on the same host correctly sizes to 224 and does not.
#705 adds a budget check that fires for pinned limits precisely because of this case, so mako will now warn about its own override rather than silently heading for the failure.
This is also a compile break
#705 changes Sort::max_temp_files from Option<usize> to a MaxTempFiles { Auto, Fixed(usize) } enum, matching the shape --max-memory already uses for its host-derived budget. So cli.sort.max_temp_files.or(Some(...)) will not compile against that fgumi — this needs action at the next fgumi bump regardless of the reasoning above.
Suggested change
Delete the override and the DEFAULT_MAX_TEMP_FILES constant, and let the flattened flag default to auto. Anyone who wants the old behaviour can still pass --max-temp-files 256 explicitly.
Worth doing in the same commit as the fgumi dependency bump, since the build will not pass otherwise.
main.rsoverrides fgumi's spill-file consolidation limit unconditionally:That override existed because fgumi's default was a hardcoded 64 that bore no relationship to the descriptor budget the process actually had, so mako raised it to avoid consolidation — which rewrites already-sorted data and is pure overhead when the descriptors were available anyway. On a 1.33B-record WGS sort spilling 84 runs, that was 958s of a 2529s run.
fulcrumgenomics/fgumi#705 fixes this upstream:
fgumi sortnow sizes the limit from the process's softRLIMIT_NOFILE(clamp(soft - 32, 16, 256)), and--max-temp-filesgained an explicitautovalue which is the default. Since mako flattens fgumi'sSortstruct and dispatches through the same command, it inherits the sizing for free once it picks up that fgumi.Why the override should go, not just become redundant
It is not merely superfluous after #705 — it is worse than the new default on a low-
ulimithost. mako asks for 256 unconditionally. On a stock macOS host (ulimit -n= 256) the usable budget is256 - 32 = 224, so a mako sort that spills more than 224 runs hitsEMFILE, while plainfgumi sorton the same host correctly sizes to 224 and does not.#705 adds a budget check that fires for pinned limits precisely because of this case, so mako will now warn about its own override rather than silently heading for the failure.
This is also a compile break
#705 changes
Sort::max_temp_filesfromOption<usize>to aMaxTempFiles { Auto, Fixed(usize) }enum, matching the shape--max-memoryalready uses for its host-derived budget. Socli.sort.max_temp_files.or(Some(...))will not compile against that fgumi — this needs action at the next fgumi bump regardless of the reasoning above.Suggested change
Delete the override and the
DEFAULT_MAX_TEMP_FILESconstant, and let the flattened flag default toauto. Anyone who wants the old behaviour can still pass--max-temp-files 256explicitly.Worth doing in the same commit as the fgumi dependency bump, since the build will not pass otherwise.