Skip to content

fix: scan the base with the CLI the project installed - #16

Merged
HugoRCD merged 1 commit into
mainfrom
fix/baseline-project-cli
Nov 17, 2026
Merged

HugoRCD merged 1 commit into
mainfrom
fix/baseline-project-cli

Conversation

@HugoRCD

@HugoRCD HugoRCD commented Nov 17, 2026

Copy link
Copy Markdown
Member

Written by an AI agent on behalf of @HugoRCD. @HugoRCD has not reviewed it yet.

Since @evlog/cli 0.10, the CLI's launcher hands off to an @evlog/cli the project installed. The action runs npx -p @evlog/cli@<pin> evlog map in the package, so once the workflow has installed dependencies, the pull request scan runs the project's CLI. The base scan runs in a fresh worktree with no node_modules, so it always ran the pin. The comparison then used two CLI versions. A rule-set change between them makes --baseline exit 2 (MAP_BASELINE_VERSION_MISMATCH). Without one, score changes between versions show up as regressions or fixes the pull request didn't make.

With the default pin (0.8.0, no hand-off) this only affects users who set version to 0.10 or later. After #15 pins the 1.0 line, it would affect every project that installs @evlog/cli.

Fix

projectCli (in cli.mjs) asks the pinned spec which version it runs in the package (evlog --version, same npx -p, same cwd). If that's the version installed in the package, the base is scanned by running that installed binary with node. Otherwise nothing changes: no CLI installed, or an old pin that doesn't hand off. Running the installed binary instead of @evlog/cli@<version> from npm also covers a CLI that isn't on npm, such as a pkg.pr.new build or a workspace link.

Before and after

A git repo built from the hono-app fixture with @evlog/cli@0.11.2 installed, and a pull request from feature into main. version: 1.0.0-beta.2, baseline: auto. An npx shim on PATH logs which CLI each npx call resolves to.

Before (main):

exit=0
[npx] cwd=<base-worktree> spec=@evlog/cli@1.0.0-beta.2 runs=1.0.0-beta.2
[npx] cwd=<workspace> spec=@evlog/cli@1.0.0-beta.2 runs=0.11.2

After (the first line is the --version probe, and the base scan no longer goes through npx):

exit=0
[npx] cwd=<workspace> spec=@evlog/cli@1.0.0-beta.2 runs=0.11.2
[npx] cwd=<workspace> spec=@evlog/cli@1.0.0-beta.2 runs=0.11.2
baseline: scanning main (b737f3d)
baseline: . runs its own @evlog/cli 0.11.2, so the base is scanned with it too

cliVersion of the base map from baselineFor, same repo:

before: installed=undefined base map cliVersion=1.0.0-beta.2
after:  installed=0.11.2 base map cliVersion=0.11.2

Without the local install, the run is unchanged: both scans resolve to 1.0.0-beta.2.

Tests

  • test/cli.test.mjs: projectCli with a stub npx. It returns the install when the spec hands off, undefined when the spec runs itself (a pin without the hand-off), and undefined with nothing installed.

  • test/project-cli-integration.mjs, new project-cli job in test.yml: the scenario above, end to end through src/main.mjs, asserting the base runs the project's 0.11.2 and delta=0. It fails on main's code:

    AssertionError [ERR_ASSERTION]: The input did not match the regular expression /runs its own @evlog\/cli 0\.11\.2, so the base is scanned with it too/
    
  • npm test: 92 pass. npm run check: clean.

The README's "Which CLI runs" section now says a project's own install takes over.

ELI5

The action compares a pull request with the branch it targets. When a project installs its own copy of the evlog CLI, the pull request was checked with that copy and the target branch with a different one, which could fail the comparison or report changes nobody made. Both sides now use the same copy.

@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 50 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 496ef94d-dcab-4706-9d12-f8ef43ef5dc7
Commits

Reviewing files that changed from the base of the PR and between 12a0df5 and 1c21416.

Files selected for processing: 7

  • .github/workflows/test.yml
  • README.md
  • src/lib/baseline.mjs
  • src/lib/cli.mjs
  • src/main.mjs
  • test/cli.test.mjs
  • test/project-cli-integration.mjs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Observability score · 64

needs work

2 instrumented · 0 partial · 2 dark

Fix first (2)

evlog map v0.8.0 · how the score works · what each check expects

@HugoRCD
HugoRCD merged commit 7fc1faa into main Nov 17, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant