Skip to content

Fold the rule's directory into command identity (fixes #167) - #173

Merged
typeless merged 1 commit into
mainfrom
fix/167-command-identity-directory
Jul 27, 2026
Merged

Fold the rule's directory into command identity (fixes #167)#173
typeless merged 1 commit into
mainfrom
fix/167-command-identity-directory

Conversation

@typeless

Copy link
Copy Markdown
Owner

Fixes #167.

The bug

Command text is Tupfile-relative, so the same rule in sibling directories renders the same string. compute_command_identity hashed that text plus sticky Variable values and nothing else, so two distinct rules shared one identity. find_by_identity is a lower_bound, so one twin silently won every lookup.

Identity is the cross-build join key for expand_implicit_deps, preserve_old_implicit_edges, merge_out_of_scope_commands, remove_stale_outputs, detect_new_commands and reconcile_input_set — the collision misrouted all of them.

Lost header dependencies, in a plain full build. Two directories compiling a same-named source with the same command line is an ordinary layout:

b's command text b/main.o after editing b/dep.h
identical to a's unchanged — rebuild silently lost
distinct (-O0 added) recompiled, correct

No scoping, no globs, no -B. Byte-identical object after the header changed, with no diagnostic.

Index truncation. A scoped build dropped every out-of-scope command, because merge_out_of_scope_commands saw the twin's identity already present and concluded the command was carried forward. Afterwards a full build detected nothing at all — not out-of-scope edits, not entirely new directories. That was #167 as originally filed; it turned out to be a symptom, and the -B immunity I first reported was a coincidence of the repro (variant builds render distinct %o paths, so the text differs).

The fix

Fold the rule's source directory into the hash. The join key must denote which rule this is, and a Tupfile-relative rendering under-determines that — the hash was missing a term of the rule's definition. Identity still deliberately excludes the input set; #166 handled that side separately.

INDEX_VERSION 14 → 15. Without the bump every stored identity stops matching at once, so remove_stale_outputs would delete every output and rebuild it; the bump discards v14 indexes cleanly instead. Verified by patching a live index's version byte back to 14 — one rebuild, then settles.

The version entry also records that v14 predates the OrderOnly edges persisted for #166. That content change shipped without a bump, and this gate is what closes it — so a future revert of this PR must not silently drop it.

From the systemic review of this change

Filed separately (all pre-existing, none fixed here)

Tests

Both observed RED before the fix:

  • unit (test_graph.cpp) — same text, different source_dir, must differ. RED showed byte-identical hashes.
  • E2E (identity_collision) — two directories with byte-identical Tupfiles and sources; editing only b/config.h must move b/program to "Version 2" while a stays at 1. RED showed b stuck at 1.

Verification

615 cases / 142,305 assertions / 32 shards; format, tidy, iwyu clean; no-op stability holds; no bootstrap drift. The original #167 repro now keeps the index at 11 files/2 commands where it fell to 7/1, sees the out-of-scope edit and the new directory, and settles.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HDzr9JV1ofTZKCL1H2qDPg

Command text is Tupfile-relative, so the same rule in sibling directories
renders the same string. compute_command_identity hashed that text plus the
sticky Variable values and nothing else, so those distinct rules shared one
identity. find_by_identity is a lower_bound, so one twin silently won every
lookup.

Identity is the cross-build join key for expand_implicit_deps,
preserve_old_implicit_edges, merge_out_of_scope_commands, remove_stale_outputs,
detect_new_commands and reconcile_input_set, so the collision misrouted all of
them. Two observed consequences:

  A scoped build dropped every out-of-scope command from the saved index --
  merge_out_of_scope_commands saw the twin's identity already present and
  concluded the command was carried forward. Afterwards a full build detected
  nothing at all: not the out-of-scope edits, not entirely new directories.

  Implicit header edges attached to the wrong twin. Two directories compiling a
  same-named source with the same command line is an ordinary layout, and
  editing one's header left its object byte-identical, in a plain full build
  with no scoping, globs or variant directory involved.

The hash was simply missing a term of the rule's definition; identity still
deliberately excludes the input set, which #166 handled on the state side.

INDEX_VERSION 14 -> 15. Without the bump every stored identity would stop
matching at once, so remove_stale_outputs would delete every output and rebuild
it; the bump discards v14 indexes cleanly instead. The entry also records that
v14 predates the OrderOnly edges persisted for #166 -- that content change
shipped without a bump, and this gate is what closes it, so a future revert
must not silently drop it.

Also from the systemic review of this change:

  reconcile_input_set no longer admits LinkType::Group. Group edges run
  file -> group and the group -> command hop is a separate OrderOnly edge, so
  find_command_by_id rejected every one of them: the arm was dead code that read
  as coverage. Removed and filed as #169 rather than replaced with a two-hop
  walk I could not demonstrate fixing anything.

  merge_out_of_scope_commands' header said it copies "Normal/Sticky edges"; it
  has carried everything except Implicit since #166, OrderOnly included, and
  that one is load-bearing for removal routing after a scoped build.

Filed separately, all pre-existing: #170 (identity injectivity is assumed by
every consumer and never enforced -- this bug would have been an error, not a
stale object), #171 (glob results sort by StringId handle, so %f order follows
readdir order and command lines are not reproducible across machines), #172
(remove_stale_outputs runs after all detection, so its own deletions are never
routed to consumers).

Verified: 615 cases / 142,305 assertions / 32 shards. The former #167 repro now
keeps the index at 11 files/2 commands where it fell to 7/1, sees the
out-of-scope edit and the new directory, and settles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HDzr9JV1ofTZKCL1H2qDPg
@github-actions

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1073 M 0.48 s 14.1 k 0.8% (+0.2pp) 0.1% (+0.1pp) 0.506 s 30.6 MB (-0.2MB)
dry-run 1785 M (+0.2%) 0.58 s 16.6 k 0.9% (+0.2pp) 0.1% (+0.1pp) 0.61 s 40.1 MB

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5818
Files changed 0
Files in index 6087
Graph edges 367913
Index size (bytes) 7264970
Implicit deps 330624
Hash computations 0
Hashes skipped (stat cache) 5818
Stat calls 5869
Parse time (ms) 489.3
Total time (ms) 599.6
Runner CPU AMD EPYC 9V74 80-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.
Timing deltas suppressed: baseline ran on different hardware (INTEL(R) XEON(R) PLATINUM 8573C).

Binary size (Linux)

Binary .text .data .bss File
putup 490.6 KB (+0.1%) 1.7 KB 98.7 KB 577.5 KB

Test coverage (lines)

Overall Median file Min file Max file
86.5% 94.9% 0.0% src/platform/path-posix.cpp 100.0% include/pup/core/arena.hpp

103 files · 14983/17313 lines covered

Deltas vs main@787968b7e.

Updated for 9b448ed

@typeless
typeless merged commit fa93fb0 into main Jul 27, 2026
12 checks passed
@typeless
typeless deleted the fix/167-command-identity-directory branch July 27, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Command identity omits the rule's directory, so identical rules in different directories collide (lost header deps, dropped index entries)

1 participant