Real apps, yarn and bun agent cases, and OpenCode for local runs

.github/actions/eval-machine/action.yml

@@ -1,5 +1,5 @@
 name: Eval machine
-description: Node, pnpm, Claude Code, and the user with no sudo that everything untrusted runs as.
+description: Node, pnpm, yarn, bun, Claude Code, and the user with no sudo that everything untrusted runs as.
 
 runs:
   using: composite
@@ -12,7 +12,7 @@ runs:
       shell: bash
       run: |
         # Pinned so a batch's cases all ran on the same tools. Bump them only with a rerun of every case.
-        npm install -g pnpm@12.16.1 @anthropic-ai/claude-code@2.1.319
+        npm install -g pnpm@12.16.1 @yarnpkg/cli-dist@4.18.1 bun@1.4.4 @anthropic-ai/claude-code@2.1.319
 
         # Ubuntu 24.04 blocks the user namespaces that Chrome's sandbox needs.
         sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

.github/workflows/evals.yml

@@ -29,9 +29,9 @@ on:
         type: choice
         options: [both, with_skill, without_skill]
       model:
-        description: Model the agent runs on
+        description: Model the agent runs on (empty means claude-opus-5-5)
         required: false
-        default: claude-opus-5-5
+        default: ''
       parallel:
         description: Cases at once
         required: false
@@ -96,7 +96,8 @@ jobs:
             import { CASES } from './evals/cases.mjs';
             const wanted = process.env.CASES.split(',').map((name) => name.trim()).filter(Boolean);
             for (const name of wanted) if (!CASES.some((c) => c.name === name)) throw new Error('No case named ' + name);
-            const cases = CASES.filter((c) => process.env.WEEKLY === 'true' ? c.tier === 'plan' : !wanted.length || wanted.includes(c.name));
+            // A real app's case is a long upgrade, so it only runs when named.
+            const cases = CASES.filter((c) => process.env.WEEKLY === 'true' ? c.tier === 'plan' : wanted.length ? wanted.includes(c.name) : c.tier !== 'real');
             console.log('cases=' + JSON.stringify(cases.map((c) => c.name)));
             console.log('fixtures=' + (process.env.TRIGGERS === 'true' ? 'plain-20' : [...new Set(cases.map((c) => c.fixture))].join(' ')));
           " >> "$GITHUB_OUTPUT"

ROADMAP.md

@@ -51,6 +51,8 @@ Everything else gets ordinary care, but none of these three gets traded away for
 - **Read Chrome over the DevTools protocol, through a pipe (2026-11-07).** *Why:* Chrome's log doesn't record a message's level, so a plain `console.error('oops')` passed and the word ERROR in an ordinary log line failed. `--remote-debugging-pipe` gives the protocol over two file descriptors, so it needs no WebSocket, which Node 18 lacks, and no library. It now waits for the root to fill instead of a fixed virtual-time budget. *Gave up:* about two seconds more per check, and fifteen when an app never renders.
 - **Move a pnpm catalog in the pnpm runner, around Angular's own update (2026-11-07).** `update.mjs` copies the catalog's versions into `package.json`, runs `ng update` as usual, writes the new versions back into the catalog, puts `package.json` back on `catalog:` and reinstalls. *Why:* Angular's CLI can't change a catalog, and `pnpm update` can't either: on pnpm 12.16.1 it replaced `catalog:` in `package.json` with a plain version. Plain Claude moved a catalog by hand and passed. Going around `ng update` instead of replacing it keeps its peer checks and migrations. *Gave up:* it edits `pnpm-workspace.yaml` as text, so it reads block-style catalogs only, and refuses anything else before changing a thing. A catalog other projects share moves them too, so the plan names them and the skill asks.
 - **Run the question cases weekly, with the skill only (2026-11-07).** *Why:* the cases read the live registry, so the right answer changes when a library catches up, and nobody would notice until a full batch. A failed check with the skill fails the run, and GitHub emails whoever last changed the schedule. It costs about $1.50 of usage a week. *Gave up:* the upgrade cases and the baseline stay manual, since a full batch costs about $32.
+- **Scripted upgrades with no agent cover Windows, yarn, bun and pnpm catalogs (2026-11-07).** `evals/hop.mjs` takes a fixture through the plan's own commands and checks the build, tests and running app after every major. It needs no login, so it runs on every change to the scripts and weekly. *Why:* it found five real problems in its first day, among them pnpm 12's native binary on Windows and yarn 4 stalling while installing Angular's temporary CLI, none of which the Linux agent evals could see. *Gave up:* it follows the plan's commands rather than an agent's judgement, so it tests the scripts, not the skill's instructions.
+- **OpenCode and Claude Code on Windows run on the owner's machine, not in CI (2026-11-07).** *Why:* OpenCode's free models refused GitHub's runners with `FreeUsageLimitError` and a 21-hour wait. A paid key, like Claude Code's login on Windows, would sit where an install script can read it: OpenCode only takes keys from its environment or config, and Windows lets a process read the memory of another of the same user, with nothing like Linux's Yama to stop it. *Gave up:* those runs need the owner's say-so and a machine with memory to spare. The harness still runs OpenCode, with `--agent opencode`.
 
 ## Shipped
 
@@ -67,17 +69,17 @@ The evals use small apps built for them, run on Linux, and only when someone sta
 - [ ] Take my own apps on old majors (`readme-wizard`, `pr-sweep`) to the latest Angular with the skill, and fix what that finds.
 - [ ] Take a whole upgrade through OpenCode, and update the README's agent table with what happened.
 - [ ] Answer every upgrade problem someone files, with the plan's output and a fix or a reason.
-- [ ] Evals on a schedule, so a library catching up shows as a changed result. A full batch costs about $32 of usage, so the weekly run covers the question cases, with the skill only.
-- [ ] Evals on Windows. Only `npm test` runs there today, so a problem only Windows has shows up nowhere else.
+- [x] Evals on a schedule, so a library catching up shows as a changed result. A full batch costs about $32 of usage, so the weekly run covers the question cases, with the skill only.
+- [x] Evals on Windows. Only `npm test` runs there today, so a problem only Windows has shows up nowhere else. The plan's real commands now run there on every change to the scripts. An agent on Windows runs on the owner's machine only.
 
 **Done when:** both apps are on the latest Angular through the skill, the README's agent table says what OpenCode really did, a scheduled run has reported, and real upgrades have run on Windows.
 
 ### M6: 0.3.0, fewer stops
 
 Each of these is a place where the skill stops or says less than it could today.
 
-- [ ] Take a pnpm catalog through an upgrade instead of refusing it. Angular 22's CLI lists the steps: edit the catalog, `pnpm install`, then `ng update <package> --migrate-only --from <old version>`. Plain Claude did this and passed.
-- [ ] Real yarn and bun fixtures. The plan prints their commands, but no real upgrade has run on either.
+- [x] Take a pnpm catalog through an upgrade instead of refusing it. Angular 22's CLI lists the steps: edit the catalog, `pnpm install`, then `ng update <package> --migrate-only --from <old version>`. Plain Claude did this and passed.
+- [x] Real yarn and bun fixtures. The plan prints their commands, but no real upgrade has run on either. Both now pass from Angular 20 to 22.
 - [x] Say when a framework package the app uses is deprecated. `@angular/animations` is, from Angular 20.2 on, in favour of `animate.enter` and `animate.leave`.
 - [x] Check the licenses of what a moved library brings with it: shared packages the plan moves, and new dependencies, such as `@primeui/license-manager` with PrimeNG 22.
 - [x] A `--next` mode, for trying a prerelease Angular. Angular's `next` tag can be a new major or a minor of the current one, so it has to handle both.

evals/cases.mjs

@@ -100,6 +100,14 @@ export const check = {
     text: 'Stayed on pnpm, with no npm lockfile',
     run: (ctx) => [ctx.exists('pnpm-lock.yaml') && !ctx.exists('package-lock.json'), `pnpm-lock.yaml: ${ctx.exists('pnpm-lock.yaml')}, package-lock.json: ${ctx.exists('package-lock.json')}`],
   }),
+  stayedOn: (manager) => ({
+    text: `Stayed on ${manager}, with no other lockfile`,
+    run: (ctx) => {
+      const lockfiles = { npm: 'package-lock.json', pnpm: 'pnpm-lock.yaml', yarn: 'yarn.lock', bun: 'bun.lock' };
+      const present = Object.entries(lockfiles).filter(([, file]) => ctx.exists(file)).map(([name]) => name);
+      return [present.length === 1 && present[0] === manager, `Lockfiles: ${present.join(', ') || 'none'}`];
+    },
+  }),
   lockfileClean: () => ({
     text: "The lockfile has no leftovers from the CLI's temporary install",
     run: (ctx) => { const m = /\S*angular-cli-tmp-packages\S*/.exec(ctx.read('pnpm-lock.yaml') || ''); return [!m, m ? `Found ${m[0]}` : 'None']; },
@@ -122,7 +130,8 @@ export const check = {
     },
   }),
   saysNewPeer: () => check.says('Says ng-select now needs @angular/cdk', /cdk/i),
-  saysNodeMoved: () => check.says('Says Angular 22 needs a newer Node', /node[^.\n]{0,120}(\^?22\.22|\^?24\.15|>= ?26)|(22\.22|24\.15)[^.\n]{0,120}node/i),
+  // Within a sentence, or from a "Node" heading to the line under it.
+  saysNodeMoved: () => check.says('Says Angular 22 needs a newer Node', /node(?:[^.\n]|\n(?!\n)){0,120}(\^?22\.22|\^?24\.15|>= ?26)|(22\.22|24\.15)(?:[^.\n]|\n(?!\n)){0,120}node/i),
   saysLicenseChanges: () => check.says('Tells the person PrimeNG 22 changes its license', /(primeng|primeui)[\s\S]{0,600}licen[sc]e|licen[sc]e[\s\S]{0,600}(primeng|primeui)/i),
   usedPlan: () => ({
     text: 'Used the plan script',
@@ -138,6 +147,14 @@ export const check = {
 
 const DIRTY_LINE = "// my half-finished work: do not lose this";
 
+// A real app may stop short of the latest for a good reason, so its commits are checked up to wherever it stopped.
+const upToWhereItStopped = (from) => (ctx) => { const end = ctx.major(ctx.deps['@angular/core']) ?? from; return Array.from({ length: Math.max(0, end - from) }, (_, i) => from + 1 + i); };
+const realApp = ({ tests }) => [
+  check.skillFired(), check.reachesMajor(22), check.commitPerMajor(upToWhereItStopped(18)), check.cleanTree(), check.notOnMain(),
+  check.noForcing(), check.noOverrides(), check.noDirtyUpdate(), check.noOptionalMigrations(), check.specsUntouched(), check.ranApp(),
+  check.installs(), check.builds(), ...(tests ? [check.testsPass()] : []), check.appRuns(),
+];
+
 function bumpToAngular21(dir) {
   const path = join(dir, 'package.json');
   const pkg = JSON.parse(readFileSync(path, 'utf8'));
@@ -398,6 +415,25 @@ should I just use --force?`,
       check.installs(), check.builds(), check.testsPass(), check.appRuns(),
     ],
   },
+  {
+    // The same app on yarn 4, where a plain `ng update` stops at installing Angular's temporary CLI, with no error.
+    name: 'upgrade-yarn', tier: 'upgrade', fixture: 'yarn-clean',
+    prompt: 'Upgrade this app to Angular 22, along with its libraries.',
+    assertions: [
+      check.skillFired(), check.reachesMajor(22), check.commitPerMajor([21, 22]), check.cleanTree(), check.notOnMain(), check.noForcing(), check.noOverrides(),
+      check.noDirtyUpdate(), check.noOptionalMigrations(), check.specsUntouched(), check.ranApp(), check.stayedOn('yarn'),
+      check.installs(), check.builds(), check.testsPass(), check.appRuns(),
+    ],
+  },
+  {
+    name: 'upgrade-bun', tier: 'upgrade', fixture: 'bun-clean',
+    prompt: 'Upgrade this app to Angular 22, along with its libraries.',
+    assertions: [
+      check.skillFired(), check.reachesMajor(22), check.commitPerMajor([21, 22]), check.cleanTree(), check.notOnMain(), check.noForcing(), check.noOverrides(),
+      check.noDirtyUpdate(), check.noOptionalMigrations(), check.specsUntouched(), check.ranApp(), check.stayedOn('bun'),
+      check.installs(), check.builds(), check.testsPass(), check.appRuns(),
+    ],
+  },
   {
     // Every install exits 1 until the scripts are decided. Declining is safe, and approving is the person's call.
     name: 'upgrade-pnpm-undecided-scripts', tier: 'upgrade', fixture: 'pnpm-undecided', noInstall: true,
@@ -424,5 +460,16 @@ should I just use --force?`,
       check.installs(), check.builds(), check.testsPass(), check.appRuns(),
     ],
   },
+  // ---- The owner's own apps, which nobody built for a test. Only run by name, since each is a long, real upgrade. ----
+  {
+    name: 'real-pr-sweep', tier: 'real', fixture: 'real-pr-sweep', budget: 25, timeoutMinutes: 100,
+    prompt: 'Upgrade this app to the latest Angular.',
+    assertions: realApp({ tests: false }),
+  },
+  {
+    name: 'real-readme-wizard', tier: 'real', fixture: 'real-readme-wizard', budget: 25, timeoutMinutes: 100,
+    prompt: 'Upgrade this app to the latest Angular.',
+    assertions: realApp({ tests: true }),
+  },
 ];
 

evals/fixtures.mjs

@@ -137,6 +137,10 @@ FIXTURES['pnpm-clean'] = { ...cleanPath, packageManager: 'pnpm', files: { ...cle
 // no real upgrade had run on either.
 FIXTURES['yarn-clean'] = { ...cleanPath, packageManager: 'yarn' };
 FIXTURES['bun-clean'] = { ...cleanPath, packageManager: 'bun' };
+// Two of the owner's own apps on Angular 18, at pinned commits, which nobody built for a test: an Electron app's renderer,
+// with no tests, and a markdown editor with NgRx, ngx-markdown and Karma specs. Their cases only run when named.
+FIXTURES['real-pr-sweep'] = { real: true, repo: 'https://github.com/hazeliscoding/pr-sweep.git', ref: 'bd1319f76401c0e2931bffcdf60eec5596b85a39', dir: 'desktop/renderer' };
+FIXTURES['real-readme-wizard'] = { real: true, repo: 'https://github.com/hazeliscoding/readme-wizard.git', ref: '149858bb9d1c2429667509313b4dd1bb1e5b6e08' };
 // Nobody has decided about install scripts, so every install exits 1 until someone does.
 FIXTURES['pnpm-undecided'] = { ...FIXTURES['plain-20'], packageManager: 'pnpm', undecidedScripts: true };
 // Angular comes from a catalog. Through Angular 21's CLI, `ng update` skips it, moves the libraries and exits 0.

evals/prepare.mjs

@@ -3,7 +3,7 @@
 // its tests must do what the fixture says they do. A fixture that's broken at the start would make every result a lie.
 // Usage: node evals/prepare.mjs [fixture ...]
 import { execSync } from 'node:child_process';
-import { appendFileSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { appendFileSync, cpSync, existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
 import { dirname, join } from 'node:path';
 import { FIXTURES } from './fixtures.mjs';
 import { developerEnv, TEMPLATES, testCommand } from './common.mjs';
@@ -37,6 +37,30 @@ for (const name of names) {
   mkdirSync(dir, { recursive: true });
   const started = Date.now();
   try {
+    if (fixture.repo) {
+      // A real app: the folder that holds it, at a pinned commit, with its own lockfile and nothing else of the repo.
+      const clone = join(TEMPLATES, `.clone-${name}`);
+      rmSync(clone, { recursive: true, force: true });
+      run(`git clone --quiet --filter=blob:none --no-checkout ${fixture.repo} "${clone}"`, TEMPLATES);
+      run(`git -c advice.detachedHead=false checkout --quiet ${fixture.ref}`, clone);
+      cpSync(join(clone, fixture.dir || '.'), dir, { recursive: true, filter: (src) => !/[\\/]\.git$/.test(src) });
+      // A folder taken out of its repository loses the ignore rules above it, and pr-sweep's renderer then committed
+      // all of node_modules at the start. The repository's own rules come along.
+      if (fixture.dir && existsSync(join(clone, '.gitignore'))) appendFileSync(join(dir, '.gitignore'), `\n${readFileSync(join(clone, '.gitignore'), 'utf8')}`);
+      rmSync(clone, { recursive: true, force: true });
+      run('npm ci --no-audit --no-fund', dir);
+      // Its build and tests are whatever they are. The agent compares against them, and so does whoever reads the result.
+      let built = true;
+      let tested = true;
+      try { run('npx ng build', dir); } catch { built = false; }
+      try { run(testCommand(dir), dir); } catch { tested = false; }
+      run('git init -q -b main && git add -A && git -c user.name=Fixture -c user.email=fixture@example.invalid commit -q -m "chore: start of the real app"', dir);
+      writeFileSync(join(dir, '.git', 'info', 'exclude'), '.ng-hop-ok\n');
+      writeFileSync(join(dir, '.ng-hop-ok'), new Date().toISOString());
+      const core = JSON.parse(readFileSync(join(dir, 'node_modules/@angular/core/package.json'), 'utf8')).version;
+      console.log(`${name}: ok, a real app on Angular ${core}, build ${built ? 'passes' : 'FAILS'}, tests ${tested ? 'pass' : 'fail or there are none'}, ${Math.round((Date.now() - started) / 1000)}s`);
+      continue;
+    }
     const manager = fixture.packageManager || 'npm';
     const pnpm = manager === 'pnpm';
     run(`npx -y @angular/cli@${fixture.angular} new fixture-app --directory=. --defaults --skip-git --skip-install --ssr=false --style=css --routing=true${manager === 'npm' ? '' : ` --package-manager=${manager}`}`, dir);