runner hygiene: local is the default; no simulated cells; cloud/unknown runners fail closed - #48
Open
maui314159 wants to merge 1 commit into
Open
maui314159 wants to merge 1 commit into
maui314159 wants to merge 1 commit into
Conversation
…nknown fail closed Three small changes to how `retort run` picks a runner: - `playpen.runner` defaults to `local` (schema and the `retort init` template). It is the only runner that produces measurements; `docker` was the default only by accident of enum order. - DockerRunner no longer SIMULATES a cell when `docker` is absent. The simulation returned random token counts and a 10% random failure rate in a RunArtifacts that looked exactly like a real one — a grid of those is indistinguishable from data. It now fails the cell with a clear stderr, and `retort run` refuses `runner: docker` up front when docker is not on PATH. - The reserved `cloud` name and any unrecognised runner name used to fall through to DockerRunner (i.e. to the simulation). They now raise before any cell runs, listing the implemented runners. None of the 81 committed experiments used the docker path, so nothing published changes. Tests: docker-without-binary fails closed (runner and cli), `cloud` refused, schema/init default is `local`; the integration test that relied on simulated cells uses a canned stub runner instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X4DHjawxy52QZkAJkXeDTP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three small changes to how
retort runpicks a runner. Code-only; no experiment data.What changes
playpen.runnerdefaults tolocal— inPlaypenConfigand in theretort inittemplate.localis the only runner that produces measurements;dockerwas the default only by enum order, so a freshretort init→retort runlanded on the skeleton.DockerRunnerno longer simulates a cell whendockeris absent._simulate_runreturned random token counts and a 10% random failure rate inside aRunArtifactsthat looked exactly like a real one. A grid of those is indistinguishable from data. It now fails the cell with a clear stderr, andretort runrefusesrunner: dockerup front when docker is not on PATH.cloudand unrecognised runner names fail closed. Both used to fall through toDockerRunner(i.e. to the simulation). They now raise before any cell runs, listing the implemented runners.cloudstays in the schema as a reserved name.None of the committed experiments used the docker path, so nothing published changes. The README's "Not yet" line now says
cloudrefuses to run rather than "falls through to Docker".Tests
DockerRunnerwithout docker fails closed (runner-level andretort run-level).runner: cloudis refused; schema andretort initdefault tolocal.tests/integration/test_design_to_run.pyused the simulation to drive its design → score → store pipeline; it now uses a canned stub runner.Full suite run against this branch: the only failures are the three
test_provenance.pycases that also fail on currentmainunchanged (TypeError: unhashable type: 'dict'), unrelated to this change.Why now
This is groundwork for a later sandbox-lane PR (a real container runner with in-container scoring); establishing "no runner ever invents a result" first keeps that PR free of runner-selection changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01X4DHjawxy52QZkAJkXeDTP