[vyncint]

What this fixes

Catch stale repository source paths in Markdown and Rust line doc comments before readers follow them.

crates/removed/src/lib.rs -> rejected even when its parent is gone
[source]:crates/pkg/lib.rs -> checked as a repository path

What we fixed

• Check paths against Git's tracked files, so missing directories and untracked local files cannot hide broken references.
• Keep complete URI and longer-path tokens intact; an external URL cannot donate a local-looking suffix.
• Use the same extractor for ordinary references and declared generated outputs, including Markdown reference definitions without a space after the colon.
• Check the traversal diagnostic directly, so removing that rule fails its regression.

Verification

Checked locally at 603aa864:

• 43 independent whole-script controls and eight scanner mutations passed; the reference-definition regression fails on the previous head.
• The current tree passes with 62 references. The tree before docs: repoint the three source references the module split invalidated #1197 reports exactly its three stale references.
• Seven affected guards, shell syntax and diff checks passed.
• Hosted CI also passes on this head.

Caveats

The guard covers declared extensions under crates/, cuda-oxide-book/ and scripts/. It checks tracked Markdown and Rust line doc comments; it does not resolve external URLs, crate-relative paths or bare filenames. Scheme-prefixed tokens such as TODO:crates/file.rs remain URI tokens.

Partially addresses #1248; its requested docs/ root is outside this guard's coverage.

[nihalpasham]

Reviewed 971ff891: changes requested.

• The tracked-parent rule mistakes deleted or mistyped source directories for generated output:

crates/cuda-device/src/missing.rs      -> rejected
crates/cuda-device/deleted/mod.rs     -> accepted as "all resolve"

• A local untracked file can also hide a missing repository reference; an external URL suffix can be treated as a repository path.
• Define tracked-source versus generated-output handling explicitly, and bound repository-path matching. Add deletion/rename, typo, external-URL and untracked-file controls through the real scanner.

The current tree and known historical cases pass, but those controls do not establish the proposed policy. #1248 remains open.

[vyncint]

Revised on your review. missing source directory != proof of generated output was right, and the parent heuristic is gone rather than patched — it was inferring exactly the thing it should have been told.

The three points, each with the old behaviour beside the new

Same probe planted in a tracked prose file, run against the old guard and this one:

planted reference	old	new
crates/no-such-crate-xyz/src/lib.rs — missing parent directory	passes	catches
crates/mir-lowerr/src/.../func_abi.rs — misspelled parent	passes	catches
crates/zz-local-only.rs — exists on disk, untracked	passes	catches
a missing file under a live directory	catches	catches
a tracked file	passes	passes
the declared generated output	passes	passes

What changed

Resolution is now "tracked by git", not "present on disk". Those are different questions and only the first is reproducible: a local build artifact or somebody's stray file resolves for them and nobody else, so the filesystem test made the guard's answer depend on whose checkout it ran in. That is your third point, and it is the same change that fixes the first two.

Generated outputs are declared. GENERATED_OUTPUT_PATHS is the whole accommodation — one entry, crates/fuzzer/artifacts/summary.jsonl, which run_seed.py writes — and it is verified rather than trusted: an entry that becomes tracked, or that nothing refers to any more, is an error rather than a silent exemption. Measured before writing it: under a tracked-only rule exactly one path in the tree needed declaring, so the heuristic was buying one line of allowlist at the cost of the guard's main job.

Repository-path boundaries. The roots are verified against the index before each sweep, and a traversal segment is reported rather than resolved. That check earned its place immediately: docs/ was in my original root list and holds no tracked file in this repository, so the pattern anchored there had been matching nothing and would have kept matching nothing after any rename. It is dropped.

Controls

• Historical: run at 6f5f538a^, the commit before docs: repoint the three source references the module split invalidated #1197, it reports exactly the three references that PR was filed for.
• Five mutations of the guard, each caught by the self-test: tracked test disabled, everything treated as a generated output, pattern matching nothing, the .rs doc-comment filter dropped, and a stale accommodation entry added.
• Clean tree: 67 repo-anchored paths in prose, all tracked. All scripts/check-*.sh pass.

One fragility the historical control exposed, worth naming because it would have bitten quietly: the self-test used this script's own path as its tracked example, so it depended on the guard already being committed — false on the branch that adds it, and in any worktree. It now takes that example from the index.

Still not checked, and still stated rather than papered over: crate-relative and bare-basename spellings. Your #1197 follow-up fixed 14 references written that way and this would not have found them.

[nihalpasham]

The rewrite fixes the missing-directory and untracked-file cases. I’m adding two small repairs on this branch:

• Match complete path tokens, so URL suffixes and names such as file.rs.bak are not mistaken for source references.
• Check generated-output references through the same prose scanner.

external URL -> outside this guard’s scope
repo path    -> tracked-file or declared-output check

I’ll validate the repaired head and update the review.

[vyncint]

First, an operational apology. I force-pushed over 8ebb8643 at 07:52Z while pushing a revision of my own, not having fetched since you pushed it. I noticed on the next command and restored the branch to 8ebb8643 about a minute later; the head is your commit, byte for byte, and nothing of yours was lost. My fault for using --force-with-lease against a stale remote ref instead of fetching first.

Second: your version supersedes what I was about to push, and I have dropped mine rather than re-apply it. I had a sed-based URL exclusion with a hard-coded blob/main/ prefix; lexical tokenization is the right shape and mine would have been a worse answer to your own review point. Below is what I found testing yours, since the useful thing I can still do is try to break it.

Verification of 8ebb8643

control	result
clean tree	62 repo-anchored paths in prose
historical, at 6f5f538a^ (before #1197)	reports exactly #1197's three, nothing else
missing parent directory	catches
misspelled parent directory	catches
deletion/rename under a live directory	catches
untracked file that exists on disk	catches
traversal segment	catches
foreign URL whose tail looks like a path here	passes
declared generated output, tracked file	pass

Eight mutations, each asserted to have applied: tracked membership always satisfied, doc-comment filter matching every line, URI arm never matching, emphasis unwrapping removed, ./ handling removed, line-citation stripping removed, and the declared-output tracked-check disabled — seven caught by your self-test. The eighth is below.

The 67 -> 62 difference, in case it looked like lost coverage

Five paths my regex counted and yours does not. Four are the github.com/NVlabs/cuda-oxide/blob/main/... links in kernel-families.md, overlapping-transfers-and-compute.md and pinned_overlap/README.md — external references by your rule, deliberately.

The fifth is a defect in mine that yours fixes. crates/cargo-oxide/src/backend_source.rs:23 documents a cache path:

//! ~/.cargo/git/checkouts/cuda-oxide-<hash>/<sha>/crates/cuda-device/Cargo.toml

My pattern matched the tail of that line as crates/cuda-device/Cargo.toml. It happens to be tracked, so it never failed — but "longer paths never donate suffixes" in your token comment is doing real work on the current tree, not just in principle.

Two findings

1. The traversal control does not discriminate. Disabling the traversal branch entirely — if ".." in path.split("/") -> if False — leaves the self-test green.

("case.md", "crates/../scripts/no-such-source-reference-file.sh", 1)

Both branches produce one finding for that input: with the rule, "contains an unsupported traversal segment"; without it, the path is not a string git tracks, so the membership arm reports it as untracked. The control asserts the count, and the count is 1 either way. broken_references already yields the reason, so comparing reasons rather than lengths pins it. (I hit the same thing in my own version and concluded the branch was dead weight; that was the wrong conclusion — your message is better, the control just needs to assert it.)

2. A path immediately after a colon is skipped. No live instance in the tree, so this costs nothing today:

[spec]:crates/nope/src/lib.rs     missed      [spec]: crates/nope/src/lib.rs    caught
TODO:crates/nope/src/lib.rs       missed      TODO: crates/nope/src/lib.rs      caught

TODO: satisfies [A-Za-z][A-Za-z0-9+.-]*:, so the URI arm consumes the span; in the reference-definition form the token begins at the colon and fails fullmatch. Both are the spelling-without-a-space case, both latent. git grep finds zero of either in prose right now, so this is a note for whenever the tokenizer is next touched, not something I would change under you.

Happy to push either as a commit on top if you want them; I did not want to touch this branch again without being asked.

[nihalpasham]

Thanks @vyncint — the reference-link case needs a fix in my extractor. I will handle the Markdown delimiter before token classification and make the traversal control check its diagnostic.

[source]:crates/pkg/lib.rs -> repository path checked
[source]:https://host/... -> external URI kept whole

• The generated-output check will use the same corrected extraction.
• TODO:... remains a scheme-prefixed URI under the declared rule; guessing its purpose would break custom schemes.
• I will rerun the controls and CI on the follow-up commit.

[nihalpasham]

I rechecked the tracked-file rewrite and added the remaining extractor follow-up:

• Markdown reference definitions such as [source]:crates/pkg/lib.rs now use the same path check as ordinary links and generated-output references.
• External and custom-scheme URIs stay intact.
• The traversal control now checks its diagnostic, so removing the rule fails the test.

At 603aa864, 43 independent script controls, eight scanner mutations and all seven affected guards passed. The historical tree still reports exactly the three stale paths from #1197. Hosted CI also passes on this head.

This partially addresses #1248; docs/ paths remain outside coverage.