Repository navigation
Fix what the independent review of the two-node lab found - #11
Conversation
An independent review of the previous change found ways the lab's checks could be bypassed or its claims overstated. This fixes them. Confinement - Linux: hide /tmp, /run and the other places that hold local service sockets, not only the home folder; start every command in its own session, so it cannot type into the operator's terminal. - macOS: deny asking other programs to act (launchd, LaunchServices, Apple events) and signalling anything outside the sandbox. - A container is no longer counted as confinement. Inside Docker, AI-written build scripts would have the network and the container's AI login, so nothing is compiled there until it has been reviewed. What "unmodified" means - Two revisions are the same node only if everything that feeds the build matches, not just src/. - A proposal may not edit top-level build tooling, anything under contrib/, or any .gitattributes, .gitignore or .gitmodules. - A proposal is compiled in a copy of the build folder that is thrown away afterwards, so nothing it leaves behind reaches a later build. - The reference node is never taken from a build folder that has compiled changed source. Checks that could be sidestepped - Changed paths are read NUL-separated and unquoted, so an unusual file name cannot slip past the protected, consensus or test path checks. - Changes are always shown as text, whatever a .gitattributes says. - A comparison in which a check could not run is "could not be completed", never "no difference", and blocks automatic applying and starting on a shared network. - The operator's git hooks are not run on AI-edited trees. Robustness - If the operator's branch cannot be put back after a build, the console says so and shows the real state instead of hiding it; SIGTERM and SIGHUP are handled like Ctrl-C. - Stopping a lab run stops its nodes. - Shared requests containing zero-width, tag, private-use or variation characters are refused, on the server as well as in the page; a malformed sharing token can no longer break the page. - The agent's read rules cover everything under the home folder except the path to its own workspace; a Claude Code too old to honour the confinement flags is not used. - The access key file is created private; the release is published only once every file is attached; the installer runs nothing if its download is cut short. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…over Bring the manual, the security policy, the README, the landing page and the Docker comments in line with the review fixes: - Docker is review-first: a container is not a sandbox around each build. - What a proposal may not change, and why build files always wait. - "Unmodified" means everything that feeds the build; a proposal is compiled in a throwaway copy of the build folder. - The new outcome for a comparison that could not be completed. - Limits stated plainly: compiling runs code from the source, both lab nodes share one sandbox so a program built to deceive could forge a result, and the agent's read limits are Claude Code's own. Record the independent review in QA.md, with how each fix was checked and what is still unchecked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Independent review of the previous pull request (#10), posted for the record. This is the report of a second reviewer (an AI reviewer that had not written the code) asked to read the change adversarially. It is advice, quoted as received, not instructions to anyone. Line numbers refer to the code as merged in that pull request. Every item below is addressed by this pull request; the table in the description says how. Independent review of PR #10 (lab-hardening)I read every file you listed in full. I read app.js and import.js for rendering and key handling, and the tests README and doc/vibes.md only to compare against claims. I ran nothing. Where an item rests on OS behaviour I did not execute, I say so and name the check that would settle it. Paths below are relative to the repository root. HIGHH1. The Linux sandbox shows the whole host to confined code, including the sockets that allow escape (SB:211-232).
H2. The macOS profile is a deny-list on
H3. Docker mode compiles hostile code before review, with network access and the Claude login readable (Dockerfile:31-47, docker-compose.yml:22, SB:78-79 and 248, BV:1794).
H4. "Same rules as the unmodified node" is decided from
MEDIUMM1. The protected-path, consensus and test gates are bypassed by git's path quoting (BV:1751-1757, 1764, 1770-1772, 1832-1835).
M2. A subdirectory
M3. The lab can say "No difference observed" about a node that cannot mine (LR:410-416, 473-488; BV:1827-1829).
M4. The operator's repo can be left on the AI commit with a UI that says it is clean (BV:1887-1898, 3497-3502, 683-688).
M5. The agent runs unsandboxed, with the full environment and a deny-list for reads (BV:96-104, 1943-1952, 1981).
M6. The "hidden characters" filter misses the usual smuggling characters (app.js:1002).
M7. Cancelling or timing out a lab run orphans the nodes on macOS (LR:128-134, BV:1401-1409, 1669-1670).
M8. A crafted shared token can break the console page persistently (app.js:359, 1020-1022, 1251-1252).
LOW
Checked and found sound
The three things to fix before public release
|
A second independent review, of the fixes themselves, found that several did not fully close what they set out to close. - A build file under doc/ was outside the "same node" check although the build reads it. A build file now counts wherever it lives. - A proposal's own build could still become the reference node, and on macOS a program started during a build can outlive it and keep its permission to write to the build folder. The reference is now taken from the operator's build folder only while no build of changed source has ever been able to write there; after that it is compiled in a folder of its own. A proposal's build never becomes the reference. Whatever a confined command leaves in its process group is ended when the command ends. - A second console started during a build moved the build folder and the branch from under it. One console process per lab folder, with a lock taken before any tidying up. - The refused-paths check was case-sensitive; a Mac's usual filesystem is not. File names that are not plain are refused. - A --text diff of a binary file raised a decoding error. - Tidying up on exit could be cut short by a second signal or a closed terminal; a linked or replaced build folder was mishandled. - Two nodes that refused nothing and still parted were "no difference". - macOS: shared temporary folders and other disks are hidden; a node on a real network cannot reach local sockets such as a key agent's. - Linux: /var is hidden too; tools installed in any hidden place are shown again. CI now fails unless bubblewrap starts, compiles the node inside it, and runs probes from inside the sandbox. The manual, the security policy and the QA record are corrected where they claimed more than the code does, and now state what the sandboxes do not hide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Second independent review, of the fixes in this pull request (first two commits), posted for the record. A second AI reviewer, which had not written the code, was asked to check whether each earlier finding was really closed and to look for new defects. It read the code and ran nothing. This is its report as received: advice, not instructions. The third commit on this branch responds to it; the description says how. REVIEW OF review-fixes (5b168ab..187de65). Nothing was run; everything below is from reading. EARLIER FINDINGS
NEW OR REMAINING, RANKED
DOCS
VERIFIED VS INFERRED
|
A third, narrower check of the previous commit found two more problems of substance. - Confined code may write to its build folder, so it could put a link to the home folder in its place, and the next build would have been given permission to write wherever the link pointed. A folder a command may write to is now never followed through a link: the sandbox refuses, nothing is compiled in a linked build folder, and a linked scratch folder is replaced by a real one. - A documentation-only proposal, judged identical, could leave a program from a shared build folder as the node to run when no reference existed. Unmodified source from a shared folder is now always the reference, compiled apart if need be, for proposals too. Also: with the network open, the macOS rule against local sockets blocked the system's name lookups, so a node could not find its first peers by name; the lookup service is allowed again. The lab folder's lock is taken before anything is written to it. A lab folder left by an earlier version is treated as shared. Two nodes that never connected are "could not be completed". Stopping a job ends its whole process group. Two sandbox tests now assert what must hold on both systems. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Third check, of the second-round commit only (1068443), posted for the record. A narrower check by an AI reviewer that had not written the code. It read the commit and ran nothing. This is its report as received: advice, not instructions. The fourth commit on this branch responds to it. This was the last round of review before merging; three rounds each found something, so more may remain, and the description says what is known to be open. High/medium remain: yes, two medium (N1 is high on macOS with automatic applying). Nothing was run; all from reading commit 1068443. FINDINGS 1-9
NEW OR REMAINING DOCS
What the fourth commit does about it: N1, N2, N3, N4, N5, N6, N8 and the rest of 9 are fixed, with tests. N7 (Homebrew on Linux) is stated as a limit, with |
Why this follows so closely
The previous pull request (#10) was set to merge when its checks passed, and it did, before an independent review of it had come back. That review found four serious problems and eight lesser ones. This pull request fixes them. Nothing from the earlier change has been released or published.
The fixes were then reviewed a second time before merging, and the second round a third time. Each found gaps; the third and fourth commits respond. All three reports are posted below in full, as advice from a reviewer and not as instructions.
What the first review found and what changes
/runand/tmpvisible (session bus, Docker socket, key agents, X server) and shared the operator's terminal session/varand the other shared folders are replaced by empty ones; every command gets its own sessionsrc/alone; build tooling could be edited; a proposal's build could leave things behind in the shared build folder.gitattributesin a subfolder could hide a change from the reviewer.gitattributes,.gitignoreand.gitmodulesare refused at any depthWhat the second review found and what changes
doc/was outside the "same node" check although the build reads it; a proposal's own build could still become the referenceWhat a third, narrower check found and what changes
Three rounds of review each found something. Review stops here for this pull request; more may remain, and a specialist's attempt to break out of the sandboxes has not been made.
What this does not settle
Stated in the manual and the security policy, so that nobody reads more into the lab than it offers:
Cost and effort
Cost: TBD (no rate card applies; the work is already done). Effort: L, on a scale where sizes are relative to each other and not to a calendar.
🤖 Generated with Claude Code