Skip to content

perf: scale readable ref diffs with changed notes#176

Open
junior-ricon wants to merge 5 commits into
mainfrom
junior/changed-note-ref-diff
Open

perf: scale readable ref diffs with changed notes#176
junior-ricon wants to merge 5 commits into
mainfrom
junior/changed-note-ref-diff

Conversation

@junior-ricon

Copy link
Copy Markdown
Contributor

Summary

  • validate each ref's manifest and note tree in batched passes
  • materialize only note blobs that affect the readable ref diff
  • define --out base/head trees as changed-note review artifacts
  • split the oversized changes/stage/commit/suppression BATS suite so existing across-file workers can schedule it concurrently
  • correct the test runner's misleading within-file parallelism claim

Validation

  • mise run test test/diff.bats — 12/12 passed
  • split 58-test group — 58/58 passed in 27.49s
  • mise run test test-task — 9/9 passed after correcting parallelism text
  • mise run test changes — 15/15 passed
  • all eight configured Codebase lints passed
  • git diff --check and Fold pre-commit/pre-push checks passed
  • one Fold-scale current-source ref-diff observation completed in 3.139s and materialized only changed artifacts

The full local suite ran once. It passed 400/402 before failing only on the two old parallelism-message assertions; those assertions were then corrected and their focused 9-test suite passed. The 29-minute full suite was not repeated for that wording-only correction. Hosted CI remains the complete post-push gate.

@junior-ricon
junior-ricon requested a review from quick-ricon July 26, 2026 04:38

@quick-ricon quick-ricon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one ref-diff correctness issue.

In lib/diff.sh:145-152, the manifest comparison stores only one readable path per ID (base[$1] = $2). Repeated IDs are accepted by the current materializer and can arise from the merge driver's documented pure-rename limitation. If the base maps one ID to both alpha.md and alias-alpha.md, and the head removes only alpha.md, the retained alias is the last path in both arrays. No candidate ID is emitted, so notes diff reports “No readable note changes” instead of the readable deletion.

There is a second form of the same selection invariant: when multiple IDs map to one readable path, selecting only the changed ID can materialize content that an unchanged colliding ID would overwrite in the full readable tree. The changed-note subset must include the full affected ID/path component, or reject these manifests explicitly rather than emit a misleading patch.

I reproduced the empty-patch case against exact head 0638992bbf9110484a9964872559381e77106393. Fix-it PR #177 derives candidates from exact changed rows and expands the affected alias component, with regression coverage for both shapes.

Apart from this finding, the batched full-tree mapping validation, missing-blob warnings, manifest-only rename path, changed-only --out contract, spaces/nested path quoting, and bounded Git process count held up in review. The mechanical test split preserves all 58 moved test names, and focused diff tests plus all eight lints passed locally.

…st-aliases

fix: preserve aliased manifest diff semantics

@johnson-ricon johnson-ricon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for three ref-diff contract issues found at exact head 3de63a20c69f48378d7aa2171b3d86d41511e36d against exact base 7e19f5f623d4fc4895599f6afad9e36599d6b21f.

  1. lib/diff.sh:145-162 reduces each manifest to exact row membership. That loses row order, but lib/diff.sh:197-208 materializes colliding IDs sequentially and the last row wins. Reordering A -> shared.md, B -> shared.md to B -> shared.md, A -> shared.md changes the full readable tree from B's content to A's content while the reducer selects no IDs and reports no change. This is an edge left by #177's exact-row repair. Fix-it #179 records and compares each path's ordered ID sequence before component expansion.

  2. lib/diff.sh:150-158 connects only exactly equal readable paths. A ref mapping one ID to shared and another to shared/child.md cannot be represented by full-tree materialization, which always fails at lib/diff.sh:205-208. If only the child ID changes, the reduced path omits the parent and emits a plausible patch instead of preserving that failure. Fix-it #178 validates every existing-blob mapping and rejects file/directory prefix collisions while retaining exact-path aliases.

  3. lib/diff.sh:167-181 rescans every edge to a fixed point. Reverse edge order can advance a long alias component only one hop per pass, making the new component repair quadratic in manifest size. A synthetic 3,001-ID component took 7.258s at this head; fix-it #180's union-find selects the same 3,001 IDs in 0.096s without additional subprocesses or blob reads.

I formed this view before reading the existing review, then checked Quick's review and #177. Quick correctly repaired duplicate-ID aliases and exact multiple-ID path collisions, but the repair still misses ordered overwrite semantics and adds the quadratic fixed-point walk.

The broader reduction is sound for ordinary one-ID/one-path manifests: changed blob IDs, manifest-only renames, missing-blob omission/warnings, NUL-delimited Git path discovery, complete-ref validation, and changed-only --out artifacts all held up. The mechanical BATS split was not part of my verdict.

Validation on the reviewed head: complete suite passed (404 BATS, 9 Python), focused ref-diff suite passed (14/14), all eight Codebase lints passed, mise run doctor passed with only the documented optional local-hook warning, readme build --check, bash -n, and git diff --check passed. Each fix-it separately passed the complete and focused suites plus the same lint/document/syntax checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants