fix(contained): verify/setup UX — progress, fail-fast, flag order, guided credentials, and the CI publish break

.github/workflows/runtime-image.yml

@@ -93,9 +93,25 @@ jobs:
           cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
 
       - name: Export the digest
+        # The `sha256:` prefix is stripped because the digest becomes a **filename**, and
+        # `upload-artifact@v4` rejects a path containing a colon. Left on, the build pushes both
+        # architectures successfully and then fails on the upload, which skips the `manifest` job
+        # that creates the tag — so the registry ends up holding the layers with `"tags": null`
+        # and every `factory contained` command dies on ImagePullBackOff for an image that was
+        # in fact built. `Assemble the manifest list` puts the prefix back.
+        #
+        # Through `env:` rather than interpolated into the script, which is this workflow's rule
+        # for every value that reaches a shell.
+        env:
+          DIGEST: ${{ steps.build.outputs.digest }}
         run: |
+          set -eu
+          case "$DIGEST" in
+            sha256:[0-9a-f]*) ;;
+            *) echo "::error::unexpected digest format: $DIGEST"; exit 1 ;;
+          esac
           mkdir -p /tmp/digests
-          touch "/tmp/digests/${{ steps.build.outputs.digest }}"
+          touch "/tmp/digests/${DIGEST#sha256:}"
 
       - uses: actions/upload-artifact@v4
         with:

CLAUDE.md

@@ -320,10 +320,13 @@ Six things are load-bearing and fail quietly if broken:
 - **Provenance.** A run always starts from the files on this machine, uncommitted changes included — never `HEAD`, never a fresh clone. The workspace is a git worktree with the working tree rsynced over the top, because a HEAD checkout silently drops the gitignored `.factory/` the whole experiment history lives in. Five assertions then run between provisioning and the first agent call (`factory/contained/provenance.py`); a failure aborts naming the file and the likely cause, and leaves the runtime up for inspection.
 - **Identity.** A bind mount carries ownership through unchanged, so a container whose UID does not own the tree gets a *silently read-only* workspace. The rule differs between rootless, rootful and macOS, so `factory/contained/identity.py` **probes** rather than deciding: a throwaway container reports the mount's owner as the kernel inside sees it, and the run matches. The runtime image is built for arbitrary UIDs (group 0, `chmod g=u`), which is also what OpenShift's restricted SCC needs.
 - **PID 1.** The factory spawns agent subprocesses and is not a well-behaved init, so the container runs `--init` around `sleep infinity` and the run itself lives in tmux. The runtime persists after the run — a failed run is exactly when its state is worth reading.
-- **Credentials cross the boundary, by design.** There is no gateway. The policy is `FACTORY_` by default, plus exactly what `--forward` names, plus the backend variables the resolved shape requires (`factory/contained/credentials.py`) — nothing implicit. `verify` reports credential *shape*, never material, and secret-looking values are redacted anywhere a command is printed. On k8s the credentials come from a namespace Secret the user creates; the factory references it by name and never handles the material.
+- **Credentials cross the boundary, by design.** There is no gateway. The policy is `FACTORY_` by default, plus exactly what `--forward` names, plus the backend variables the resolved shape requires (`factory/contained/credentials.py`) — nothing implicit. `verify` reports credential *shape*, never material, and secret-looking values are redacted anywhere a command is printed. On k8s the credentials live in a namespace Secret. `factory/contained/k8s_credentials.py` will *create* one for you as step 3 of the cluster wizard — backend picker, then per value: typed (masked), an environment variable you name, or a file (whose required fields are printed before the question and validated after). Four rules there are load-bearing: the material never enters an **argv** (`--from-literal` is visible in `ps` and in shell history — the manifest goes to `oc apply -f -` on **stdin**), the manifest is **JSON** not YAML (a key containing `:` or a newline is ordinary here), nothing is echoed (shape-only confirmation, redacted command, scrubbed stderr), and nothing is logged but key names and value lengths. With nobody at the keyboard the step is **skipped**, not defaulted — `--yes` means "do not stop to ask me", not "choose a credential for me".
 - **Both divisions reach outward, and that is the point.** Builds cannot happen inside either boundary, so `--division` is opt-in and separately named. Locally it starts an **unauthenticated** `podman-mcp-server` on `0.0.0.0:8430` — every interface, because the tool has no bind flag and the container reaches the host through a gateway address rather than loopback — detached into its own process group, because the run outlives the launch, and stopped by `factory contained rm`. On the cluster it goes through OpenShift `Build` objects behind a sidecar container that is the only holder of `oc` and the ServiceAccount token; that separation is a boundary only while the Role excludes `pods/exec`, which `verify` asserts via a **SubjectAccessReview API object** — `oc auth can-i --as` collapses `pods/exec` onto `pods` and answers "yes" where RBAC says no. The sidecar runs a **different image** (`FACTORY_CONTAINED_SIDECAR_IMAGE`, an `oc` image) from the agent's; one image for both silently collapses the boundary.
 - **Interactive prompts stall an unattended run.** A fresh `~/.claude` makes Claude Code ask about folder trust, project MCP servers, and Bypass Permissions mode — all interactive-only, so headless agents never hit them and the interactive CEO does, and the run then sits at a menu nobody is watching. `factory/contained/claude_state.py` pre-records those answers, which the invocation already implies.
 - **All podman knowledge lives in `factory/podman.py` and all cluster knowledge in `factory/contained/k8s.py`.** Both **compose** commands and do not execute them, which is what makes `FACTORY_CONTAINED_DRY_RUN=1` print the same argv the real path runs rather than a separate rendering that drifts.
+- **Never wait on a pod with a flag.** `oc wait --for=... --timeout=Ns` can only report that a condition did not hold, so an unpullable image cost the full timeout and was then reported as a mystery ("the probe produced no output"). `k8s.classify_pod` reads the pod document and `k8s.poll_pod` acts on it: states the kubelet has already given up on (`ImagePullBackOff` — *BackOff* is the word it uses **after** retrying — `CreateContainerConfigError`, `Unschedulable`, …) return at once carrying the kubelet's own message; genuinely retryable ones (`ErrImagePull`) get 30s. `ContainerCreating` is deliberately **not** capped: a first pull legitimately runs for minutes, and capping it trades a hang for a false failure on every cold node. Both the inference probe and `wait_for_container` on the real run path go through it.
+- **A step that takes minutes must say what it is waiting for.** `style.activity()` is a transient status line: silent for 5s, then one line rewritten in place (`\r\x1b[2K`) carrying the pod's state and a clock, erased on exit so the caller's result lands in its place. Gated on `style.can_rewrite()`, **not** `enabled()` — colour and motion are different questions, and honouring `FORCE_COLOR` as permission to emit carriage returns fills a CI log with fragments. Off a TTY it degrades to one plain line per changed text; `FACTORY_NO_PROGRESS=1` disables it.
+- **Runtime flags parse on either side of a subcommand.** `contained verify --target k8s` and `contained --target k8s verify` are the same command. The flags are declared once in `contained_args._RUNTIME_FLAGS` and fed to both the real parser and a tail parser built with `default=argparse.SUPPRESS`; suppressed defaults are the whole mechanism, since a tail parser that applied its own defaults would report `--target local` for a command line that never said `--target` and silently overwrite the left-hand side. Repeatable flags merge across the subcommand; the same flag on both sides with different values is an **error**, because choosing one silently is how a bundle reaches the wrong namespace. Nothing after `--` is touched.
 
 The runtime image (`containers/factory/Containerfile`) is UBI9 + the factory wheel + the agent CLIs + tmux, published multi-arch by CI (`.github/workflows/runtime-image.yml`) — amd64 for cluster nodes, arm64 for a Mac laptop. It publishes on pushes to `main` (`:latest`), on **published releases** (`:<tag>`, plus `:latest` unless the release is a prerelease — nightlies are, so they never move `:latest`), and on dispatch. The release trigger is load-bearing: `factory contained setup` pulls and does not build, so a release whose image was never built breaks every new user's first command. Release and dispatch tag names reach the shell through `env:` and are validated against the legal image-tag character set before use.
 

docs/contained/index.md

@@ -109,8 +109,19 @@ factory contained {ls|attach|rm|sync|setup|verify|bundle|help} [name]
 | `--storage-class SC` | cluster default | Workspace PVC |
 
 A flag used against the wrong target fails at parse time naming the target it belongs to — never
-silently ignored. Runtime flags go **before** the subcommand; anything flag-shaped after it is an
-error rather than a name.
+silently ignored.
+
+Runtime flags go on **either side** of a subcommand. These are the same command:
+
+```bash
+factory contained --target k8s verify
+factory contained verify --target k8s
+```
+
+Both orders are what people type, so both work. Give the same flag on both sides with two different
+values and it stops and says so rather than picking one — `--namespace a verify --namespace b` has
+no obviously right reading, and choosing wrong applies RBAC to somebody else's namespace. Nothing
+after `--` is interpreted at all: that belongs to the factory inside the runtime.
 
 ---
 
@@ -411,7 +422,7 @@ credentials**:
 ```console
 $ factory contained --target k8s setup
 
-━━ 1/3  Cluster and namespace ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+━━ 1/4  Cluster and namespace ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 
    Clusters in your kubeconfig:
 
@@ -438,7 +449,7 @@ Create namespace 'factory-contained' now?  [y]es  [n]o  (y/N): y
    Created factory-contained.
    `oc new-project` also made it your current project.
 
-━━ 2/3  Review and apply ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+━━ 2/4  Review and apply ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    Comparing 5 object(s) against namespace 'factory-contained' on
    'https://api.my-cluster.example.com:443':
 
@@ -477,12 +488,36 @@ Apply this? (2 of 4)  [y]es  [n]o  [a]ll remaining  [q]uit  (Enter or Esc = skip
    rolebinding.rbac.authorization.k8s.io/factory-scc created
    persistentvolumeclaim/factory-workspace created
 
-The credentials Secret is yours to create — the factory never handles the material:
-  oc create secret generic factory-credentials -n factory-contained \
-      --from-literal=ANTHROPIC_API_KEY=...
+━━ 3/4  Credentials ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+
+   secret/factory-credentials is missing from factory-contained
+   The pod reads this Secret as its environment. It stays in the namespace; the
+   factory sends the material once, here, and never reads it back.
+
+   [1]  Anthropic API key
+   [2]  Vertex AI (Google Cloud)
+   [3]  copy what this shell is configured for (anthropic)
+   [s]  skip — print the command and let me do it
+
+Which inference backend should the pod use? [1/2/3/s]: 1
+
+   [t]  type it now (hidden as you type)
+   [e]  read it from an environment variable in this shell
+   [f]  read it from a file on this machine
+   [q]  cancel
+
+Where does the Anthropic API key come from? [t/e/f/q]: t
+Anthropic API key: ***********************************************
+   ANTHROPIC_API_KEY: 108 characters, starts 'sk-ant-a', ends '9f2c'  (typed)
+Use this?  [y]es  [n]o  (Y/n): y
+
+   About to create 'secret/factory-credentials' in 'factory-contained' with:
+   ANTHROPIC_API_KEY:   108 characters, starts 'sk-ant-a', ends '9f2c'
+Create it now?  [y]es  [n]o  (Y/n): y
+   secret/factory-credentials created
 ```
 
-Six details that are deliberate.
+Seven details that are deliberate.
 
 The **cluster** is asked too, not just the namespace. A kubeconfig usually holds several, and `oc
 config use-context` is the only way most people know to move between them — so picking the wrong
@@ -555,6 +590,34 @@ There is no second, blanket "are you sure?": every object was confirmed a moment
 prompt on top of that is the friction that teaches people to hit `y` without reading. `--yes`
 applies everything pending without walking, for automation.
 
+The **credentials Secret is offered as a step**, not left as a closing reminder. It used to be
+printed as an `oc create secret` line and nothing more, which meant every freshly prepared namespace
+ended one check short of an answer — the inference probe needs that Secret to authenticate, so it
+could only be skipped. Pick a backend, pick where each value comes from, and it is created.
+
+Four rules govern the material, and each one exists because the obvious implementation breaks it:
+
+- **It never reaches an argv.** `oc create secret --from-literal=KEY=value` puts the value in the
+  process table for every user on the machine, and in the shell history of anyone who copies the
+  line. The Secret is composed as a manifest and fed to `oc apply -f -` on **stdin**.
+- **The manifest is JSON, not YAML.** A key containing `:`, a newline or a leading `%` is ordinary
+  here and is a quoting bug waiting to happen in hand-built YAML.
+- **Nothing is echoed.** Typing is masked; a value is confirmed by *shape only*
+  (`108 characters, starts 'sk-ant-a', ends '9f2c'`), which is enough to catch a paste that grabbed
+  the surrounding quotes and not enough to reuse. The command printed afterwards is redacted, and
+  anything that turns up in `oc`'s stderr is scrubbed before you see it.
+- **Nothing is logged.** Key names and value lengths; never a value.
+
+A value can be **typed**, read from an **environment variable** you name, or read from a **file** —
+and for the Google credential the required fields are printed *before* you are asked, then the file
+is parsed and any missing field named. Checking it here is the point: an unusable credential is
+accepted into a Secret without complaint and would otherwise surface as an authentication failure
+inside an agent call, minutes later, looking like a model outage.
+
+If a usable Secret is already there, the step says which backend it carries and asks nothing. With
+nobody at the keyboard it is **skipped** and the manual command printed — `--yes` means "do not stop
+to ask me", not "choose a credential for me", and there is no safe default for that question.
+
 Then `verify` checks every object, every verb the ServiceAccount needs, the Secret's **keys** (never
 its values), and that inference is reachable from a pod *inside* the namespace. Results print **as
 each one lands**, not at the end — several are a cluster round trip and the in-cluster inference
@@ -582,8 +645,35 @@ The inference check is the slow one: it creates a short-lived pod, with the same
 a real run uses, and asks it to make one request — because a host-side check proves nothing about
 the *pod's* egress. It is announced before it starts, and **skipped entirely when the credentials
 Secret is missing**, since the probe pod mounts that Secret and could only spend its 180-second
-timeout rediscovering what the check above already said. That is the state a freshly prepared
-namespace is in, because creating the Secret is deliberately left to you.
+timeout rediscovering what the check above already said.
+
+While it waits it **says what it is waiting for**. Anything that takes more than five seconds grows
+a status line that rewrites itself in place, carrying the pod's own state and a clock:
+
+```console
+⠹ inference_from_cluster — probe pod: ContainerCreating (0:47)
+```
+
+Nothing is drawn below five seconds, so a fast check looks exactly as it always did, and the line
+erases itself when the result lands. In a pipe or a CI log it degrades to one plain line per change
+rather than thousands of redraws; `FACTORY_NO_PROGRESS=1` turns it off entirely.
+
+It also **stops early when the pod cannot start**. The wait used to be `oc wait --timeout=180s`,
+which is blind by construction — it can only report that a condition did not hold, so an image the
+cluster could not pull cost the full three minutes and was then described as `the probe produced no
+output`, naming neither the cause nor where to look. Now each poll reads the pod, and a state the
+kubelet has already given up on (`ImagePullBackOff`, `CreateContainerConfigError`, a pod nothing
+will schedule) returns immediately with the kubelet's own words:
+
+```console
+[FAIL] inference_from_cluster: a pod in this namespace could NOT reach inference: the probe pod
+       could not run: ImagePullBackOff — Back-off pulling image "ghcr.io/…/factory-runtime:latest"
+```
+
+A first-ever image pull is *not* treated that way — a cold `ContainerCreating` legitimately runs for
+minutes, and capping it would trade a hang for a false failure on every new node. Only states that
+are already an error get the 30-second ceiling. The same reading now guards the workspace and
+factory containers on the real run path, which had the identical problem with a 300-second timeout.
 
 Before setup, the same command lists what is missing with the command that restores each — e.g.
 `factory contained --namespace factory-contained bundle | oc apply -f -`.