Skip to content

Return directory entries in name order (fixes #179) - #181

Merged
typeless merged 2 commits into
mainfrom
fix/179-readdir-order
Jul 27, 2026
Merged

Return directory entries in name order (fixes #179)#181
typeless merged 2 commits into
mainfrom
fix/179-readdir-order

Conversation

@typeless

Copy link
Copy Markdown
Owner

readdir order is where filesystem nondeterminism enters putup. Every consumer was responsible for canonicalizing it — a conventional invariant — and two of the five consumers did not: glob expansion (#171) and discover_variants (#175). The first put readdir order into %f and therefore into command identity.

read_directory now sorts entries by name on both platforms, and walk_directory inherits it because both implementations enumerate through read_directory. Enumeration order becomes a function of directory content at the one site that produces it, instead of at each consumer that remembers to ask. The guarantee is stated at the declaration, since that is what callers read.

The same boundary in tup is already ordered: listings come from node_dir_index (dir, name).

Demonstrated

This branch carries neither downstream fix, so the variant hint shows the boundary alone doing the work. Six build dirs enumerating (ls -U) as yankee mike charlie bravo alpha zeta:

Error: no build directory specified; found: build-alpha, build-bravo, build-charlie, build-mike, build-yankee, build-zeta

What this does not replace

Boundary sorting gives determinism, not canonical path order, for recursive walks: DFS pre-order over per-directory sorted entries is not lexicographic — sub.txt sorts before sub/a.txt as a string, while the walk emits sub/a.txt first because it descends sub on the way past. glob_expand's own sort (#174) is still what makes %f path-ordered, and #176's sort still states the intent locally rather than depending on a remote invariant.

Tests

Two scenarios: read_directory returns name order, and walk_directory enumerates every level in name order depth-first. The RED artifact depends on the filesystem — the tests create entries in reverse-lexicographic order, so they fail on a creation-ordered filesystem and a hash-ordered one alike. Observed here as { alpha, zeta, charlie, mike, yankee, bravo }.

The win32 half is compile-checked by CI only; it is the same one-line sort in the same position.

make test green (142313 assertions, 618 cases, 32 e2e shards); make tidy and make iwyu clean; bootstrap scripts unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA

readdir order is where filesystem nondeterminism enters putup. Every consumer
was responsible for canonicalizing it, which is a conventional invariant -- and
two of the five consumers did not: glob expansion (#171) and discover_variants
(#175), the first of which put readdir order into %f and therefore into command
identity.

read_directory now sorts its entries by name, on both platforms, and
walk_directory inherits it because both implementations enumerate through
read_directory. Enumeration order becomes a function of directory content at the
one site that produces it, rather than at each consumer that remembers to ask.
The guarantee is stated at the declaration, since that is what callers read.

The same boundary in tup is already ordered: listings come from
node_dir_index (dir, name).

Demonstrated without either downstream fix in this branch -- six build dirs
enumerating as yankee mike charlie bravo alpha zeta:

  Error: no build directory specified; found: build-alpha, build-bravo,
    build-charlie, build-mike, build-yankee, build-zeta

This does not make the consumer-side sorts redundant. Boundary sorting gives
determinism, not canonical path order, for recursive walks: DFS pre-order over
per-directory sorted entries is not lexicographic, since sub.txt sorts before
sub/a.txt as a string while the walk emits sub/a.txt first. glob_expand's own
sort is still what makes %f path-ordered.

The RED artifact depends on the filesystem: the test creates entries in
reverse-lexicographic order, so it fails on a creation-ordered filesystem and on
a hash-ordered one alike. Observed here as
{ alpha, zeta, charlie, mike, yankee, bravo }.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1073 M 0.49 s 14.1 k 0.8% 0.1% 0.502 s 30.7 MB (-0.2MB)
dry-run 1786 M 0.6 s 16.6 k 0.9% 0.1% 0.612 s 40 MB (+0.2MB)

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5818
Files changed 0
Files in index 6087
Graph edges 367913
Index size (bytes) 7264970
Implicit deps 330624
Hash computations 0
Hashes skipped (stat cache) 5818
Stat calls 5869
Parse time (ms) 493
Total time (ms) 598.7
Runner CPU AMD EPYC 7763 64-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.
Timing deltas suppressed: baseline ran on different hardware (AMD EPYC 9V74 80-Core Processor).

Binary size (Linux)

Binary .text .data .bss File
putup 497 KB (+0.6%) 1.6 KB (-5.9%) 98.7 KB 587.5 KB (+0.8%)

Test coverage (lines)

Overall Median file Min file Max file
86.6% 94.9% 0.0% src/platform/path-posix.cpp 100.0% include/pup/core/arena.hpp

103 files · 14995/17322 lines covered

Deltas vs main@1738aa715.

Updated for 389f021

…m too

The names were all lowercase with distinct initials. NTFS keys its directory
index case-insensitively, so that set already arrives sorted there and the test
would have passed with the production sort reverted -- on the one platform the
win32 half of this change exists for.

Mixed case makes byte order and NTFS's native order disagree: byte order gives
Bravo, Zulu, alpha, mike; a case-insensitive enumeration gives alpha, Bravo,
mike, Zulu. Same for the walk test, at both levels.

Checked by reverting the sort in read_directory: the test fails, which it did not
have to do before on every filesystem.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
@typeless
typeless merged commit 8af3a8f into main Jul 27, 2026
12 checks passed
typeless added a commit that referenced this pull request Jul 27, 2026
Three findings from an adversarial review of the enumeration-order PRs (#174,
#176, #181, #186).

compose_nested_project_subtree binary-searched `available` while push_back-ing
into it, so after the first append the range was unsorted and lower_bound's
precondition was violated: a directory already present could be reported absent
and appended again, and the final sort had no unique to remove it. A duplicate
there means sort_dirs_by_depth parses that Tupfile twice in one run, registering
its rules twice -- which now dies at the duplicate-key rejection with an error
naming a project that is perfectly valid. Which projects hit it was decided by
interning-order accidents, i.e. exactly what this PR series was removing. The
search now covers only the sorted prefix, which is sufficient because the rel_ids
within one call are distinct, and the sort dedups so the invariant no longer rests
on that being true.

PR #186 deleted <, >, <= and >= on StringId so that every comparison has to state
whether it means handle order or name order. It left two routes open, both
verified by compilation against the real header:

  auto x = (a <=> b) < 0;                                  // compiled
  struct Node { StringId name; int x;                      // compiled: member-wise
    auto operator<=>(Node const&) const = default; };      // built-in enum <=>

The second is the dangerous one: a defaulted spaceship on any struct holding a
StringId sorts by handle without a single character at the call site saying so.
Deleting operator<=> rejects both; equality is untouched. Neither route is used
anywhere in the tree today, so this closes the class before it grows a member.

prepare_job_launch emitted a job's exported variables in interning-handle order.
create_env_block (process-win32.cpp:102) writes that sequence into the environment
block verbatim, and Win32 expects it sorted alphabetically. Now emitted by name.

Not claimed: the review reported observing handle order in a child's `env` output.
That evidence is confounded -- commands run under sh, which re-exports its own
environment in its own order, so the block order putup passes is not observable
that way. The Win32 block requirement stands on its own and is why this changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant