Skip to content

check --since is as fast as undeclared: it reads only the files changed since REV (#85) - #87

Merged
rabbilyasar merged 1 commit into
mainfrom
rabbilyasar/bl-205-fast-since
Oct 26, 2026
Merged

rabbilyasar merged 1 commit into
mainfrom
rabbilyasar/bl-205-fast-since

Conversation

@rabbilyasar

Copy link
Copy Markdown
Owner

Closes #85

In one line: check --since REV --code-only, the CI replacement for undeclared, took 155 s on PostHog. It now takes 3.0 s, faster than undeclared (4.2 s), and finds the same new reads.

Decision (maintainer, 2026-10-27, recorded on #85): option 1

With --since, code references come only from the files changed or untracked since REV. An old undeclared read in an unchanged file is no longer listed in that mode. Plain check still lists and fails on every one.

Before / after

  • Before: check --since walked every file the service owns, then classified each undeclared read as new or old.
  • After: it reads only the changed files, then proves a read old if a changed file (deleted ones included) read the same name at REV.
  • Unchanged: the gate (a new read fails; an unproven one counts as new), plain check, and every command without --since.

What changed

  • since.changed_files(REV): git diff against the working tree, plus untracked files. A git failure raises (exit 1) instead of returning "nothing changed", which would be a false pass. git_utils.list_changed_files/list_untracked_files gained strict=True, and GitError.
  • evaluator.service_source_filter(service): the per-file rules of the full walk (ownership, additional_source_roots, default-excluded directories, exclude_files) as one predicate. The walk, the changed-file list and REV's side all use it. discover_code_references(service, files=…) reads only the files it's given.
  • since.read_before checks each changed file's content at REV with check's own discovery, so a BaseSettings field that was already there is old. The same 1 MB cap applies on both sides.
  • The report's since block gains code_scope: "changed_files". The human output says "N file(s) changed since REV scanned".

How to check it

Branch rabbilyasar/bl-205-fast-since, commit 4bffef2. In a git repository with EnvShield set up:

  1. Commit a file that reads os.getenv("OLD_FLAG"). envshield check --since HEAD --code-only --json → exit 0, files_scanned: 0, code_scope: "changed_files". envshield check --code-only → exit 1 (plain check still sees it).
  2. Append a comment to that file → --since lists OLD_FLAG as old (new: false), exit 0.
  3. Add new.py reading os.getenv("NEW_FLAG") → exit 1 on NEW_FLAG only.

Measured (fresh shallow clones, median of three runs)

Repository Files undeclared HEAD~10 HEAD check --since HEAD~10 --code-only Before
PostHog 61,936 4.2 s 3.0 s 155 s
Twenty 39,885 1.8 s 0.6 s 65 s
n8n 32,345 1.7 s 0.7 s 50 s
Dify 15,086 3.1 s 0.4 s 29 s
Langfuse 7,476 1.4 s 0.6 s 15 s
cal.com 7,712 0.7 s 0.5 s 8 s

Over HEAD~39, both commands report the same new reads on all six (Dify: USERNAME, from both). check also lists 4–18 old reads in changed files per repository; the slowest run was PostHog at 6.1 s. This mode reads no .env or manifest, so no configuration value is involved. The clones were deleted afterwards. Recorded in BACKLOG.md BL-205.

Risk

Low. It narrows what --since lists (old reads in unchanged files) without changing the gate, and fails closed on any git error. Undo: revert the PR.

Size: 344 lines added, 122 of them tests.

Checks, configuration and docs
  • Full test suite passes: uv run --frozen --extra dev pytest, 2278 passed, 1 skipped, also with GITHUB_ACTIONS=true.
  • ruff check ., ruff format --check . and envshield scan . --config .github/envshield.ci.yml are clean
  • UI or end-to-end change checked in a real browser: not applicable (CLI)
  • Configuration changes: none.
  • Docs and decision records updated: D-13, AGENTS.md (current phase; V6-2 · Consistent defaults, readable CI output, honest text #45 merged), BACKLOG.md (BL-205 fixed with measurements; the V6-2 · Consistent defaults, readable CI output, honest text #45 findings marked merged). README/website are deferred to the v5 release; this PR's narrowing of --since is on that list.
  • New tests: --since never walks the tree, plain check still does; a changed file counts only for its owner; exclude_files applies to changed files; an old read in an unchanged file isn't listed; a BaseSettings field in a changed file is proven old; git failures fail closed (listing, and content at REV).

🤖 Generated with Claude Code

The CI form, check --since REV --code-only, walked every file the
service owns and was 40-110x slower than undeclared on large
repositories (PostHog: 155 s). With --since, code references now come
only from files changed or untracked since REV, through the same
per-file rules as the full walk, and a read is old only if a changed
file read the same name at REV (check's own discovery). Listing the
changes fails closed when git can't answer. Plain check is unchanged.

PostHog now takes 3.0 s, faster than undeclared (4.2 s); same new reads
as undeclared on six large repositories. D-13 updated.
@rabbilyasar
rabbilyasar merged commit c017110 into main Oct 26, 2026
4 checks passed
@rabbilyasar
rabbilyasar deleted the rabbilyasar/bl-205-fast-since branch October 26, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check --since --code-only is 40–110× slower than undeclared on large repositories (BL-205)

1 participant