Skip to content

Finish whole-repository checks: opt-in rules stay off under group levels, a status line, budget and key errors said once - #49

Merged
tauanbinato merged 2 commits into
mainfrom
v0.32
Sep 29, 2026
Merged

tauanbinato merged 2 commits into
mainfrom
v0.32

Conversation

@tauanbinato

@tauanbinato tauanbinato commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

A whole-repository check of nodehaven-online (950 files, TypeScript, Python, SpacetimeDB) with 0.30/0.31 printed nothing for 70 seconds, spent its 1,000-request budget and left 377 files unchecked, advising a --max-requests that cannot raise the ceiling jevgate.toml sets. With this branch the same check, same config, empty cache, finishes: 930 requests, 73 s, $0.046, with a status line all along; a rerun from the cache takes 2.2 s.

What caused it

  • A group level turned on an opt-in rule. The project's jevgate.toml (written for 0.8) sets maintainability = "consider". A level for a group judged every rule of it, so hardcoded values — out of the default rules since 0.26, right 6 times in 37 on unseen projects — ran too. It asks about every module constant (one request per file, one question each) and every function with a literal, and many rechecks and locates follow: 1,846 requests for a whole check, where the three default maintainability rules need 943.
  • The budget was known to be short before anything was sent. The first pass alone planned 1,126 requests against a ceiling of 1,000; the check sent 1,000 and failed the rest.
  • No key gave 660 failed files, 38 of them blamed on the request budget with 0 requests sent.
  • No output while asking. Local work is 2.2 s for all 950 files; the rest is requests, paced at TypeSafe's 1,200 a minute (the project's concurrency = 4 makes it about 13 a second).

Changes

  • A group turns on the rules it runs by default, or every rule of a group with none on by default (security, documentation). Hardcoded values runs when named ("maintainability/hardcoded-values" = "consider", --rule hardcoded-values) or with all. Applies to [rules] levels, rules = [...] and --rule; --skip-rule maintainability still skips all of it. jevgate init writes hardcoded values on its own line. This changes which rules run for configurations that set a level on maintainability — the one public-behavior change here.
  • Status line on stderr on a terminal: JevGate · first pass · 312/768 answered · 23s, per stage (test-file purposes, planning, first pass, each follow-up round, composing). note!/say! erase it first; it is gone before the findings. Off in CI, with --watch, --format jsonl, when stderr is not a terminal, and for checks under 0.4 s.
  • Budget: before the first request it cannot cover, stderr says this check needs at least N requests, more than max_requests = 1000 in jevgate.toml allows… and how to finish; each unsent file says Request budget reached (max_requests = 1000 in jevgate.toml); rerun to continue from the cached answers, or raise the budget (or --max-requests N when the flag set it).
  • No key: jevgate: No API key configured. …, exit 2, nothing else. The report keeps each file's error (MCP and JSON consumers unchanged), Git hooks still let the change through saying why, and a watcher looks for a key again on its next snapshot.
  • macOS: the Keychain key is read without a terminal. It required stdin and stderr to be terminals, so pre-push hooks (refs on stdin) and agent hooks (event on stdin) never found the key auth login saved and let changes through unchecked. Without a terminal the Keychain dialog is turned off (SecKeychainSetUserInteractionAllowed), so a read macOS would prompt for fails at once with a message saying to run jevgate auth status once in a terminal. Verified with a throwaway item: the creating binary reads it with the dialog off and no terminal; a binary the item does not trust fails in 0.6 s with no dialog.
  • (would fail the gate) instead of (fails the gate) when the gate was not evaluated.
  • Docs and init suggest max_cost rather than a pull-request-sized max_requests.

Measured

nodehaven-online, whole check, empty cache requests first pass time result
0.31.0, project config (max_requests = 1000, concurrency = 4) 1,000 (1,846 needed) 1,126 ~70 s, silent incomplete, 377 files unchecked
this branch, same config 930 807 73 s, status line complete, gate evaluated

No question, state or composition changed: cached answers stay valid, and runs of --rule all (the corpus harness) ask exactly what they asked. Tests: 898 unit + 79 CLI + 3 pass; cargo +1.90.0 check --locked passes; jevgate check --base main passes the default gate (notes only).

Not in this PR

Function packs never cross a file boundary, so 373 of nodehaven's 611 function requests hold one function. Packing across files with today's edit-stable runs would ask 318 (first pass -36%; axios -23%, chi -31%, flask -11%, starlette -16% in dry runs). It changes what Jev sees — the 0.28 pack merge flipped about a fifth of function-simplification reviews, the one rule that blocks by default — so it needs the corpus measurement AGENTS.md asks for, and is left as a proposal.

…a status line, budget and key errors said once

A level for maintainability turned on hardcoded values, which left the
default rules in 0.26: on a 950-file project it doubled a whole check,
1,846 requests where the default rules need 943, and a jevgate.toml
ceiling of 1,000 stopped it halfway. A group now turns on the rules it
runs by default, or every rule of an opt-in group.

A check draws a status line on a terminal, says before its first request
when its request budget cannot cover it, names where the budget is set
(the old message advised a flag that cannot raise jevgate.toml's
ceiling), ends in one line without a key, and reads the macOS Keychain
without a terminal, as Git and agent hooks run.
@tauanbinato
tauanbinato merged commit ba18ced into main Sep 29, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant