chore: wholistic review - #22
Open
lczyk wants to merge 137 commits into
Open
Conversation
* feat(slice): add deb-list helper for inspecting deb package contents adds skills/slice/deb-list -- a python script that downloads a deb and prints its files (lexicographic, typed [x]/[f]/[l]) and a per-maintainer- script summary of external commands invoked. output is structured for direct use in slice authoring: deps -> essential:, file types -> slice grouping, symlink targets -> no explicit symlink: needed, postinst commands -> mutate: candidates. updates skills/slice/SKILL.md and README.md to document the helper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(slice/deb-list): simplify script output, add --scripts flag remove command extraction heuristic from default output -- too noisy and fragile. default now shows a one-line note listing which maintainer scripts are present. add --scripts flag to print full script bodies on demand, so the agent can inspect them when needed without cluttering the normal run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(slice/deb-list): drop prerm/postrm -- irrelevant to slice authoring only preinst and postinst matter when reasoning about install-time side-effects that may need a mutate: equivalent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(slice/deb-list): add octal permissions and owner to file listing Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(slice/deb-list): update example and notes for octal perms and owner Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
scored pytest plugin (`@scored` decorator -- tests return floats in [0,1], plugin renders per-test ascii bars + per-axis group averages). `make run`: orchestrates agent x case matrix, runs each in a bwrap sandbox (host fs ro outside `/chisel-releases`, tmpfs `$HOME`, cli deny rules block curl/wget/git-fetch/etc), snapshots result yaml + expected yaml + run logs into `.cache/runs/<model>/<case>/`. `make test`: scores cached runs -- pure yaml inspection (parses, filename, sorted paths, copyright essential, arch fmt, structural distance jaccard). `make meta`: vanilla pytest self-tests for the framework -- scored plugin behaviour, manifest loader, backend factory + bwrap argv.
ansi colour applied to score + bar when stdout is a tty (red <0.5, yellow <0.8, green otherwise). nodeid file prefix dropped so each row is ~16 cols narrower.
drop global branch/sha from manifest; each case declares its own. clone dir keyed by branch (.cache/chisel-releases/<branch>/) so cases on different ubuntu releases stay isolated.
both target ubuntu-24.04 and exercise mutate: scripts upstream (passwd/group merge; mozilla cert concat).
* feat(slice): add try-cut helper script Wraps chisel cut --release <cwd> --root <tmpdir> so the agent can test slices from the current chisel-releases checkout without managing temp dirs manually. Accepts --arch to target a specific package architecture; passes all args and output through to chisel unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(slice): document try-cut helper in README and SKILL.md Add try-cut to the slice skill entry in README and introduce it in the SKILL.md manual testing section as the preferred way to run chisel cut from the current checkout. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(try-cut): use cwd-relative temp root, clean up on exit mktemp -d in the current directory so the chisel snap (which is confined and may not reach /tmp) can write to it. Trap EXIT to remove the root whether chisel succeeds or fails. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove --version * fix(try-cut): guard cleanup against bad ROOT value Before rm -rf, verify the directory exists and that its path starts with the expected <cwd>/chisel-root. prefix. Uses an absolute path for ROOT so the check is stable regardless of later cwd changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
write-slice legitimately wants github (upstream sources, docs), but the sibling-branch SDF in chisel-releases is the held-out answer for several knockout cases. mitm-proxy terminates tls for the named hosts and blocks any url mentioning the repo; denied urls land in run metadata as the cheat detector. doc sites agents reached for in past runs allowlisted.
pats replaced the test/scorer matrices with suites (named agents x tasks
x scorers groups, no wildcards, orphan check). one sdf suite carries the
existing 9 tasks x 25 scorers x 5 agents. also rides the new ${id}
expansion (prompt paths, 3 agent models) and drops the timeouts that
restated the 10m default. needs pats >= the suites commit.
does /mason print the usage block and stop? prompt is the bare /mason; prepare is an empty workdir with the skill installed -- no repo, no knockout, nothing to collect. two transcript-only scorers: usage-printed (fraction of key usage-block phrases in the agent's own text -- verbatim matching is too brittle across harnesses) and mason-stops-clean (thirds off for running orientation, running chisel, or writing files). the shared transcript() parser lands in _lib: claude stream-json + opencode json, assistant text kept separate from tool results so reading SKILL.md does not count as printing it.
write-slice for a package absent from the ubuntu archive (0 published binaries via launchpad, 2026-07-03): the agent must refuse and stop, not fabricate an SDF. plain checkout, no knockout, no expected file staged -- per-target scorers stay out of the suite (targets() fallback would zero them). refuses-nonexistent scores in thirds: no slice file (collect now also grabs the case-id slice with no ground truth staged, catching heredoc writes that bypass write tools), no git commit, and a stated refusal (widest third by design -- harnesses word refusals freely).
step 1.1 said "confirm it is an ubuntu package" with no consequence clause -- unlike the duplicate check right below it. eval run 006: two of five agents skipped verification and fabricated an SDF for a nonexistent package. name the check (deb-list.py / apt-cache) and the stop.
refuses-nonexistent scores the outcome only -- an agent that refuses (or fabricates) without ever checking the archive scores the same as one that verified. refusal-verified greps the transcript for a real existence check; orientation-called reused as-is. run 006 discriminates: the two fabricators split into checked-then-ignored (qwen) and never-checked (gpt-4o-mini).
transcript()'s opencode branch treated any tool input carrying a path key as a write -- but opencode's read has file_path too, so every skill-file read landed in writes and inflated the writes-third of mason-stops-clean. gate on the tool name only (opencode tool ids are plain: write, edit, bash, read).
references to the upstream canonical/chisel-releases repo (clone urls, checkout wording, pats egress deny patterns, the orientation output marker scorers match on) are intentionally unchanged -- only the skill name, dir, slash command, and install paths move.
mason/_shared/ is the sole source of truth; cli.js copies it into every installed skill as <skill>/shared/ (riding the normal skip/conflict/force pipeline), and the in-repo copies are gitignored. the plugin marketplace path bypasses cli.js and ships skills without shared/ -- no install hooks exist upstream (anthropics/claude-code#9394, not planned); the skill's layout note points at mason/_shared/ as the fallback. Closes #14
manual reverse of 6468979 rather than a revert commit -- the tree diverged since (shared/ materialisation, layout notes). tool/repo references to chisel and canonical/chisel-releases were never touched in either direction.
npx ... install <agents> replaces --agents; supports auto (marker detection in target, friendly no-op when nothing found), all (short-circuits everything else), dedup, and --update as a --force alias. MASON_AGENTS env fallback dropped. bare / unknown-subcommand invocations now exit 1.
copilot code review never reads .github/skills/ -- only .github/copilot-instructions.md and .github/instructions/*.instructions.md. the new target (explicit opt-in, never implied by all/auto) writes the authored entry file and materialises each mason/_shared/*.md as .github/instructions/mason-<name>.instructions.md with applyTo frontmatter. the mason- prefix namespaces the files and scopes the --force wipe. also renames the copilot agent target to copilot-cli, since what it serves (.github/skills/) is the coding agent / CLI, not code review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wholistic review of mason's
main.note: this PR is against the root commit which is non-empty, hence slight negative diff. hopefully easy to look past