List every finding in one pull request comment - #2
Merged
Merged
Conversation
check.sh exited 2 before writing its outputs when the base revision was not in the checkout, so the documented exit-code output was empty exactly when the run was incomplete. Every exit now goes through one function that writes exit-code and report first.
On Windows runners the bash shell is Git Bash, whose $PWD is /d/a/...; the report output built from it names a path Node resolves to D:\d\a\..., so upload-artifact and other JavaScript actions could not open it. cygpath -m turns it into D:/a/..., which both bash and Node open; elsewhere the path is unchanged. A new job checks it on all four runners: a repository created in the job, whose one change no cache answers, runs with --cache-only, which writes a report and ends incomplete with no key and nothing sent. It checks exit code 2 and that Node can open the report output.
JevGate 0.26.0 accepts keys from the two gateways that serve Jev. The new input api-key-kind (typesafe, openrouter or vercel; typesafe by default) says which service issued api-key, and the check gets the key as TYPESAFE_API_KEY, OPENROUTER_API_KEY or AI_GATEWAY_API_KEY, and no other kind's key: a job that keeps an OpenRouter key for its own tests must not have JevGate spend it because it also reads that variable. With a gateway kind and a JevGate older than 0.26.0, the check stops with an error naming the version, instead of JevGate reporting that no key is configured. The key is set only when api-key is given, so a key a workflow puts in the job's env for that kind is no longer replaced by an empty input. test/key-kinds.sh runs check.sh against a stand-in jevgate that records the key variables it receives, on Linux, macOS and Windows.
GitHub shows at most 10 error and 10 warning annotations per step, so a pull request with more findings showed only some of them on the diff. On pull request events the action now posts one comment and updates it on each run: the gate's result and reasons, the run's files, API requests, input tokens and cost, then every finding nobody accepted, by level (reviews open, considers collapsed past 10, notes collapsed) and by file, each linked to its line at the checked commit. A run that could not finish (exit 2: no key, a provider error such as HTTP 402, the request budget) opens with a caution alert and each distinct reason with the files it stopped. - render.cjs writes it from the JSON report alone, so it works with any JevGate version, and comment.cjs posts it, run by actions/github-script pinned by SHA on any runner that runs JavaScript actions. check.sh saves this run's report before the SARIF replay replaces it (the replay sends no request, so its report shows no cost), and only when the check wrote a new one (checksum before and after), never a report left from an earlier check. - A hidden first line keys the comment by job and working directory, and names the run that wrote it: a run for an older push leaves a newer run's comment alone, and copies left by two first runs racing are deleted. Only bot comments starting with the marker are edited. - The body stays under GitHub's 65,536-character limit, measured in UTF-8 bytes: notes are cut first, then the lowest-ranked considers, with a line saying how many of each are left out. On the largest corpus reports (1,500 to 1,750 findings of whole-repository runs) it keeps every review and fills 64,645 to 65,091 bytes in under 30 ms. - Text from the report quotes the change's code, so outside code spans it is escaped: no HTML, links, mentions or comment markers, and an unclosed backtick cannot pair with the next field's code span. - Without pull-requests: write, as on pull requests from forks, it says so in the log and the job summary and the step passes; any other failure is a warning too. comment: false turns it off. Tested with node --test (renders JevGate's own --base report and 0.25.0 reports of a run without a key and after a 402, a fake issues API for create, update, duplicates, older runs and refusals) and in CI: on four runners a read-only token's refused comment must not fail a passing check, and on pull requests from this repository two checks in one job must leave one comment holding the second's incomplete banner.
JevGate: run incompleteCaution JevGate could not finish this run (exit code 2). The gate was not applied, and findings may be missing.
1 file · 0 API requests · 0 input tokens jevgate 0.30.0 · commit 0783863 · workflow run · updated on each run |
From 0.26.0 JevGate blocks by default only on rules and levels measured right at least 80% of the time on unseen projects; the others are reported without failing the gate, and each finding records how the gate counted it (`gate`: fails, measuring or advisory). Without that, the comment would list five reviews under "gate passed" with nothing to say why. A finding that fails the gate gets "(fails the gate)" after its rule, as in JevGate's agent text, and one line counts the findings reported without failing it because their rules and levels are still being measured. Reports from earlier versions have no such field, and nothing is marked.
JevGate 0.28 removes the probability from each finding's message and records instead how often findings of its rule and level were right on projects it was never tuned on (`precision`: `right` of `labeled`). The comment listed only the message and the next step, so with 0.28 it would show neither. Each finding now ends as JevGate's own output ends it: "Right 87% of the time (23 labels)." or, below 20 labels, "Not yet measured."; a report before 0.28 shows nothing more.
JevGate 0.26 judges only what a change touches when given --base: the changed lines of changed files, with --whole-files for the old behavior. The input's description still said it reviewed changed files.
…ommitted cache
From JevGate 0.30 a finding of its own rules in a preview language's file
carries `preview` with the language, and its `precision` holds that
language's counts. The comment rebuilt the precision sentence without it
("Right 12% of the time (34 labels)." for a Bash shared-logic review,
where JevGate says "in Bash") and explained every `measuring` finding as
a rule still being measured, wrong for a C function-simplification
review, whose rule and level fail by default outside preview languages.
It now says "in <language>", and gives preview findings their own reason:
the language is in preview, and JevGate's own rules never fail the
default gate there. A law finding keeps JevGate's caveat that it was
labeled only on Bend 2 projects.
A pull request could also commit answers under .jevgate/cache that clear
its own code; JevGate 0.28 and later ignores cache files Git tracks, and
the action now removes a checked-out cache before restoring its own, for
earlier versions too.
The first example pinned 0.25.0 and the gateway one 0.26.0. 0.30.0 is the release this version of the action words its comment for (preview languages, per-language precision), and the gateways need 0.26.0 or later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
comment, on by default). GitHub shows at most 10 error and 10 warning annotations per step, so larger runs lost findings from the diff. The comment is updated in place on each run and has:fail_on,[[scope]]globs and, from 0.26, the maturity default);jev-1.13.0; "cost unknown" when requests were answered with no counted tokens, as a gateway withoutusagewould, never $0);gate, as JevGate's agent text shows it); older reports carry no such field and nothing is marked;[!CAUTION]alert first: "JevGate could not finish this run (exit code 2). The gate was not applied, and findings may be missing.", then each distinct reason with the number of files it stopped (TypeSafe HTTP 402; request was not retried (1 file)). A run that stopped before writing a report still gets the alert.render.cjswrites the comment from.jevgate/latest.jsonalone;comment.cjsposts it throughactions/github-script(v9.0.0, pinned by SHA, node24 likeactions/cachev6.1.0). Chosen over bash with jq or Python: GitHub's hosted images have both (jq 1.7 to 1.8; Python 3.12 to 3.14, documented aspythononly on Windows), but self-hosted runners need not, while every runner has the Node that runs JavaScript actions; and grouping, escaping and a byte cap are far easier to write and test in JavaScript.ghis also missing on some self-hosted runners.check.sh: saves this run's report for the comment before the SARIF replay replaces it (the replay sends no request, so its report shows no cost), and only when the check wrote a new one (cksumbefore and after; a usage error exits 2 and leaves the previous report, with agenerated_atin the same second). It also records the checked commit, the working directory's prefix in the repository (for links in monorepos) andjevgate --version.<!-- jevgate-action comment key=<job>[ <dir>] run=<run id> -->) finds the comment again. Only bot comments starting with it are edited. A run for an older push leaves a newer run's comment alone; copies left by two first runs racing are deleted, keeping the oldest. Text from the report quotes the change's code, so outside code spans HTML, links, mentions and markers are escaped, and an unclosed backtick cannot pair with the next field's code span (checked with GitHub's own Markdown renderer).pull-requests: write, and on pull requests from forks, the step logs a warning and a job-summary paragraph saying why and carries on. Any other failure is a warning too: the comment never fails the job.api-key-kind(typesafe,openrouter,vercel; defaulttypesafe): the key goes to JevGate in that kind's variable and no other kind's, so a key the job keeps for its own tests (OPENROUTER_API_KEYis common) is never spent, and aTYPESAFE_API_KEYin the job's env cannot win over the kind chosen here (JevGate 0.26 reads it first). With a gateway kind and JevGate older than 0.26.0, the check stops with an error naming the version. An emptyapi-keyno longer blanks a key set in the job'senv.e9352c3, from 0.28's tested patch): each finding ends as JevGate's own output ends it, with how often findings of its rule and level were right on projects JevGate was never tuned on ("Right 87% of the time (23 labels).", or "Not yet measured." below 20 labels), read from the report's per-findingprecision; 0.28 no longer puts the probability in the message. Older reports show nothing more.baseinput wording (4417811): from JevGate 0.26.0 a base means only the changed lines of changed files (--whole-filesinargsfor whole files); the description said changed files.exit-codeis written when the base revision is missing (it was empty exactly when the run was incomplete), and thereportoutput is a Windows path on Windows (Git Bash's$PWDis/d/a/..., which Node, and so upload-artifact, resolves toD:\d\a\...).Decisions to review
commentdefaults totrue. Existing@v1workflows with onlycontents: readget one warning per run until they addpull-requests: writeor setcomment: false. The README examples now grant it.api-key-kind, next toapi-key, and the variable names it sets. JevGate 0.26'sauth loginnames the same choice--provider(same three values);providerwould also fit here.render.cjs), as in JevGate'soutput.rs. If JevGate adds a cost field to the report, the action should read it.gate(0.26.0 on), never recomputed: that would need JevGate's scope globs and the maturity default. With older versions the comment shows the gate's reasons and no per-finding mark; the annotations still say error or warning.Measured (no Jev requests)
Rendering the largest corpus reports (whole-repository runs, far larger than a pull request's):
--baserunTests
node --test test/render.test.cjs test/comment.test.cjs(30 tests, Node's built-in runner, no dependencies): renders JevGate's own--basereport and 0.25.0 reports of a run without a key and after a 402; the cap on a synthetic 3,060-finding report; escaping; and posting against an in-memory issues API (create, update, unchanged, duplicates, older run, 403, fork, 500, unreadable report, not a pull request, dry run). Thirteen mutations of the risky lines (escaping, the run guard, the author check, the cap, refusals, the gate marks) each fail a test.bash test/key-kinds.sh:check.shagainst a stand-injevgatethat records the key variables it gets, for every kind, old and new versions, and keys already in the job's env.unit(the above on Linux, macOS, Windows);fixture(four runners: a repository created in the job whose one change no cache answers, so--cache-onlywrites a report and exits 2 with no key; then a passing check whose comment the read-only token cannot post must not fail; thereportoutput must open from Node);comment(pull requests from this repository, withpull-requests: write: two checks in one job must leave exactly one comment, holding the second's incomplete banner, so this pull request shows a live one).Known and left alone
args: --cache-onlytogether withsarif-filemakes the replay pass--cache-onlytwice, which fails with a warning that wrongly says 0.18.0 is needed. It predates this change.After merging
v1.2.0with the notes below, and movev1.site/src/ci.mdshowspermissions: contents: readonly; addpull-requests: writeand a sentence on the comment. JevGate's own.github/workflows/jevgate.ymlpins v1.0.0 by SHA.version:in the README example when 0.26.0 ships.Release notes for v1.2.0