Repository navigation
Real apps, yarn and bun agent cases, and OpenCode for local runs - #9
Merged
Merged
Conversation
pr-sweep's Electron renderer and readme-wizard, both on Angular 18, at pinned commits: nobody built them for a test. A fixture clones the folder that holds the app, installs from its own lockfile and records whatever its build and tests do. Each case is a long, real upgrade, so it only runs when named, and keeps a bundle of the agent's commits, uncompressed, so the upgrade can be replayed onto the real repository and the check for the login before upload can still read every byte.
… Windows OpenCode on a free model needs no login, so its job has no secret near it and runs where Claude Code's login couldn't be kept from an install script: on Windows, as well as Linux. Each run gets a config home of its own, with sharing off and the same commands denied as for Claude Code, and the skill in OpenCode's global skills folder. Transcripts are read from OpenCode's documented JSON events, and results leave out .git and links through a collector that works on both platforms.
GitHub refused the workflow: a job's own env can't read the runner context.
The same app as upgrade-pnpm, on yarn 4 and on bun, graded the same way, plus a check that it stayed on its package manager with no second lockfile. The evals machine installs pinned yarn and bun beside pnpm.
Its condition implied that every job before it had succeeded, and the canary is skipped on purpose for OpenCode.
pr-sweep's renderer, copied out of its repository, lost the root .gitignore, so the fixture committed all of node_modules at the start and every install dirtied the tree. The agent rightly stopped and asked instead of committing millions of lines.
OpenCode's free models refused GitHub's runners with FreeUsageLimitError and a 21-hour wait, and a paid key would sit in OpenCode's environment, where an install script can read it. The job, its agent input and its results collector go. The harness still runs OpenCode locally with --agent opencode.
Records why scripted upgrades cover Windows, yarn, bun and catalogs, and why OpenCode and Claude Code on Windows run on the owner's machine.
…fig names opencode run starts its session in $PWD when that's set, not in its working folder, so the last local run began in the folder the harness was launched from and had to move itself to the project. The config now uses OpenCode 2's permissions list and update setting instead of the deprecated 1.x names, and the harness stops on any other OpenCode version, where the denied commands might not apply.
OpenCode 2 names its shell tool shell and loads a skill by id, so the grader saw no commands and no skill: the forcing checks passed on nothing and the app check failed. Code Mode can call the other tools, so its code is checked like a command.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Real apps nobody built for a test, agent cases on yarn and bun, and OpenCode in the harness for local runs.
node_modules, and the agent rightly stopped to ask.upgrade-pnpm, graded the same way, plus a check that it stayed on its package manager.run.mjs --agent opencoderuns a case through OpenCode with a config home of its own, sharing off and the same commands denied, and reads its transcript from OpenCode's documented JSON events. It runs on the owner's machine only: its free models refused GitHub's runners, and a paid key would sit where an install script can read it. The first local run showed three harness bugs, now fixed:opencode runstarts its session in$PWD, so the run began in the folder the harness was launched from; OpenCode 2 calls its shell toolshelland loads a skill byid, so the grader saw no commands; and the config used OpenCode 1's names, which 2.2 still translates but has deprecated. The harness now stops on any OpenCode other than 2.What the real apps showed
Both went from Angular 18 to 22 with the skill, one commit per major.
The skill handled NgRx's migration crash,
marked18 droppingmarked.min.js, ngx-markdown 22's blank page and CRLF line endings, each from its troubleshooting notes.OpenCode
With those fixes, OpenCode 2.2 on its free model took the clean-path app from Angular 20 to 22 on Windows, one commit per major, and passed all 17 checks Claude Code is graded on, in 9 minutes. The first run, before the fixes, regrades to 17/17 too.