Use these to sanity-check a new run. Large departures usually mean a classifier needs a new model ID or a new host-shell rule, not that the behavior changed.
| Measure | Value |
|---|---|
| Shell commands | 38,176 across 874 sessions, 48 model IDs |
| Sources | Codex 20,072 · OpenCode 13,104 · Claude Code 4,421 · Cursor 579 |
| Failed (harness-flagged) | 10.7% |
| Hidden failures (OK + hard error string) | 876 (2.3%) |
| Calls carrying an embedded program | 24.9% |
| Dialect mismatch fail rate vs matched | 29.8% vs 11.2% |
| POSIX on Git Bash / on Windows PowerShell 5.1 / on cmd.exe | 2.7% / 40.8% / 26.6% |
| python -c / python heredoc / node -e / node stdin / PS here-string / jq / awk-perl | 27.8% / 10.3% / 15.0% / 5.9% / 9.4% / 8.5% / 0% |
| Top "not found" name | head typed into PowerShell, 96 times |
| Codex exit code visible (text(r) vs text(r.output)) | GPT-6 Astra 75% · GPT-5.6 Sol 8% · GPT-5.6 Luna 8% |
| Shell share of all tool calls | Codex models 56–71% (structural) · Claude Opus 5 81% (uninstructed) · Claude Fable 5.1 73% (auto-mode instructed) · Claude Sonnet 5 37% |
| Outputs over 10k chars | 6.2% of measured (Codex omitted, token-counted) |
- OpenCode 1.18.25 resolves its shell as
$SHELL, else the first of:pwsh,powershell, Git Bash (OPENCODE_GIT_BASH_PATHor<git>/../../bin/bash.exe),%COMSPEC%. It has per-shell prompt notes (tells the model&&is unsupported on Windows PowerShell 5.1). Config keyshell("Default shell to use for terminal and bash tool") overrides all of it. On that machine it spawnedpowershell.exe5.1 even though pwsh 7.6 and Git Bash were installed. - Codex CLI 0.153 defaults to PowerShell on Windows;
exec_commandtakes ashellparameter that models passed ascmd.exeor a PowerShell path. An absolute Git Bash path is accepted too but regressed to cmd.exe in 0.149 (openai/codex #40328). The exit-code-lost-unless-printed behavior is openai/codex #32411; the pwsh app-execution-alias problem is #18937 there and anomalyco/opencode #41321 / #17372. Features includepowershell_shell_version,unified_exec,code_mode,js_repl. The exec result object carriesexit_code; the cell prints only what the model prints. - Claude Code's Bash tool on Windows is Git Bash; it was the lowest-failure host in the corpus.