Skip to content

Rename Command Model and separate Shell Gatherer collection #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Noisemaker111 merged 2 commits into from
Sep 10, 2026
Merged

Rename Command Model and separate Shell Gatherer collection #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d39e42f
Rename project to Command Model and separate Shell Gatherer
Noisemaker111 Sep 10, 2026
f6a210f
Record explicit standing agents merge authorization in root instructions
Noisemaker111 Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/agent-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ jobs:
with:
python-version: '3.12'
- name: Validate syntax
run: python -m compileall -q scripts experiments/command_specialist
run: python -m compileall -q scripts skills experiments/command_model
- name: Path binding and saved evidence invariants
run: python -m unittest discover -s experiments/command_specialist -p test_bindings.py -v
run: python -m unittest discover -s experiments/command_model -p test_bindings.py -v
- name: Native and PowerShell contract invariants
run: python -m unittest discover -s experiments/command_specialist -p test_contract.py -v
run: python -m unittest discover -s experiments/command_model -p test_contract.py -v
- name: English delegation evidence and authorization invariants
run: python -m unittest discover -s experiments/command_specialist -p test_delegate.py -v
run: python -m unittest discover -s experiments/command_model -p test_delegate.py -v
- name: Frozen main candidate and patch notes
run: python scripts/check_release.py
- name: Codex comparison evidence invariants
run: python -m unittest discover -s experiments/command_specialist/codex -p test_compare.py -v
run: python -m unittest discover -s experiments/command_model/codex -p test_compare.py -v
2 changes: 1 addition & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ work/
*.jsonl
.venv/
.worktrees/
experiments/command_specialist/js_parser/node_modules/
experiments/command_model/js_parser/node_modules/
*.log
30 changes: 25 additions & 5 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
# Command specialist
# Command Model

"Our model", "the model we are building", and "Command Model" refer to this
project. Shell Gatherer is only the collection component in skills/shell-gatherer.
Mining, labeling, execution verification, datasets and training are downstream
steps. Do not describe gathered observations as verified training examples.

Build a small, fast local model and execution loop that does command/cell work for
frontier agents. Accept English intent and known context, perform the mechanical
work, inspect real outcomes, repair ordinary failures, and return compact verified
evidence with retrievable raw output. Each delegation gets a fresh, bounded worker
lifecycle. A generated program or claimed success is not completion.

Read [project purpose and evidence](experiments/command_specialist/PURPOSE.md)
Read [project purpose and evidence](experiments/command_model/PURPOSE.md)
before choosing the next experiment. The frontier owns reasoning and delegation;
the worker owns bounded mechanical implementation and verified evidence. The
486-example inspection/evidence adapter is not an English code-execution-trained
Expand All @@ -32,6 +37,21 @@ Measure whole handoffs, including repair and fallback; small synthetic probes do
not establish general reliability. Preserve held-out data and the current pilot
model/adapter; keep private corpora and weights out of Git.

Follow [DEVELOPMENT.md](DEVELOPMENT.md) for checks, isolation, maintainer permissions
and automatic agents integration. Keep release details out of routine reports.
Jon initiates release work. Report observed improvements, failures and limitations.
Follow the agents-and-main workflow in [DEVELOPMENT.md](DEVELOPMENT.md).
Jon explicitly authorizes agents acting for eligible maintainers to prepare,
verify, push, open ready PRs and merge them into `agents` after the exact reviewed
head passes local verification and required CI. Jon reaffirmed this standing
authorization on September 10, 2026: do not ask for per-PR merge permission for
`agents`. This project-specific authorization replaces generic instructions to
ask Jon before every merge into this development branch. Use the coordinator and
its maintainer, expected-head and CI gates; never bypass a failed gate.

Use `sb agents` / `sb main` only in a checkout you own. Do implementation work in
isolated worktrees and preserve other sessions. Only Jon personally merges
`main`; prepare a release only when he explicitly requests one. Agents integration
does not authorize production deployment, publication or unrelated changes.
If an external approval control rejects an authorized action, report its actual
reason and provide the existing user authorization; do not bypass that control
or describe the project policy itself as requiring new per-PR permission.
Keep release details out of routine reports. Report observed improvements,
failures and limitations.
9 changes: 6 additions & 3 deletions DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
branch. Jon authorized agents-branch automation on September 10, 2026: agents may
prepare owned changes, verify them, open ready PRs against `agents`, and merge them
when the gates below pass. This replaces per-PR human merge requests for `agents`.
Jon reaffirmed that authorization in chat on September 10, 2026 after an external
approval control rejected a merge. The explicit authorization is also recorded
directly in root AGENTS.md so readers need not infer it from this policy link.
Only Jon personally merges release PRs into `main`. Agents must never merge main,
enable its auto-merge, or push directly to it, even after chat approval. Publication,
deployments, model/weight changes, and production data changes need separate approval.
Expand All @@ -15,9 +18,9 @@ existing PRs for the same work. Run commands from the repository root with Pytho
3.12. Required checks are:

```
python -m compileall -q scripts experiments/command_specialist
python -m unittest discover -s experiments/command_specialist -p test_bindings.py -v
python -m unittest discover -s experiments/command_specialist -p test_contract.py -v
python -m compileall -q scripts skills experiments/command_model
python -m unittest discover -s experiments/command_model -p test_bindings.py -v
python -m unittest discover -s experiments/command_model -p test_contract.py -v
```

Also verify the affected user operation through the public CLI and reopen its saved
Expand Down
118 changes: 59 additions & 59 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,81 +1,81 @@
# Shell Forensics
# Command Model

A Claude Code skill that reads the transcripts your coding agents already keep on disk (Codex, Claude Code, OpenCode, Cursor), pulls out every shell command they ran, and builds a report: which model does everything through bash or PowerShell or a JavaScript cell, what the shell is used for, how it fails, and what actually fixed it. The report is a deck of screenshot-sized cards, each with a ready-to-post summary.
A small local model and execution loop for command/cell work delegated by a
frontier agent. The frontier supplies English intent and known context; one fresh
local worker performs bounded mechanical work, verifies the result and returns
compact evidence. "Our model" and "the model we are building" refer to this project.

Example output from the first machine it ran on: 38,176 commands, 874 sessions, 48 model IDs. Findings included a 30% failure rate when the model typed POSIX into a PowerShell host versus 11% when the dialect matched, `| head` typed into PowerShell as the single most common concrete failure, `python -c` one-liners failing at 28% versus 10% for the same Python fed through a heredoc, and GPT-5.6 models printing only `r.output` in Codex cells 92% of the time so that non-zero exit codes never reached the transcript. See `reference.md` for the baseline numbers.
Start with [purpose and evidence](experiments/command_model/PURPOSE.md),
[agent instructions](AGENTS.md), and the [Codex handoff prototype](experiments/command_model/codex/README.md).
The original inspection adapter and later execution experiments have different
scopes; the purpose document preserves what each actually demonstrated.

## Install
## Components

Claude Code:
| Component | Responsibility |
| --- | --- |
| Shell Gatherer | Collect recorded shell commands, results and provenance |
| Mining and labeling pipeline | Normalize observations, recover useful candidates and assign supported labels |
| Execution verification and datasets | Check candidate behavior and save examples with frozen held-out partitions |
| Model training | Train for the actual delegation protocol, preserving original adapters |
| Local execution loop and frontier evaluation | Execute grounded English jobs and measure verified whole-operation accuracy, time and tokens |

```
git clone https://github.com/Noisemaker111/shell-forensics ~/.claude/skills/shell-forensics
```
The gatherer does not certify training examples. See the
[data pipeline](experiments/command_model/DATA_PIPELINE.md) and
[measured data results](experiments/command_model/DATA_RESULTS.md).

Then in any Claude Code session:
## Repository and local work

```powershell
git clone https://github.com/Noisemaker111/command-model
cd command-model
```
/shell-forensics
```

Codex reads skills from `~/.codex/skills`, so the same clone works there:

```
git clone https://github.com/Noisemaker111/shell-forensics ~/.codex/skills/shell-forensics
```
Follow [DEVELOPMENT.md](DEVELOPMENT.md) for owned worktrees, checks and integration.
The model source is in experiments/command_model. Existing shell-specialist model
aliases, frozen dataset seeds and old work/command-specialist data paths retain
their identities for reproducibility; this rename does not retrain weights.

You can also run the scripts by hand without any agent. Python 3.10+, standard library only:
## Shell Gatherer

```
python scripts/extract.py work # records.jsonl, toolcounts.json
python scripts/analyze.py work # records_annotated.jsonl, summary.json, prints the headline tables
python scripts/scratch.py work # scratch.json: scripts written to disk per model
python scripts/build.py work # shell-forensics.html, page-data.json
```
The self-contained skill is [skills/shell-gatherer](skills/shell-gatherer/SKILL.md).
Copy that directory to ~/.claude/skills/shell-gatherer or
~/.codex/skills/shell-gatherer to install it. In a fresh session request
"Use Shell Gatherer to collect my local shell-command observations."

Open `work/shell-forensics.html` in a browser. Add `--no-gallery` to `build.py` for a page you can share: the gallery embeds real commands with real paths from your machine.
To run directly with Python 3.10+ and no dependencies:

## What it reads

| Harness | Path | Parsed |
|---|---|---|
| Codex CLI and desktop | `~/.codex/sessions/**/*.jsonl` | `exec` JS cells; each `tools.exec_command` / `shell_command` inside is one record |
| Claude Code | `~/.claude/projects/**/*.jsonl` | `Bash` and `PowerShell` tool calls joined to their results |
| OpenCode | `~/.local/share/opencode/opencode.db` | `bash`, `shell`, `oc_bash` and `execute` tool parts, v1 and v2 stores |
| Cursor CLI | `~/.cursor/chats/*/*/store.db` | `Shell` tool calls joined to results |

Read-only. Nothing leaves the machine unless you publish the page.
```powershell
New-Item -ItemType Directory -Force work
python skills/shell-gatherer/scripts/gather.py work
```

## What the classifiers do
This writes records.jsonl and toolcounts.json. The old scripts/extract.py command
forwards to the same implementation. Recorded output is capped at 700 characters;
full-transcript ingestion is a separate pipeline. No collected commands execute.

Each command is tagged with the dialect the model wrote (POSIX, PowerShell, cmd, neutral, mixed), the host shell that ran it, its job (git, read, search, build/test, and so on), any embedded program (`python -c`, heredoc, `node -e`, here-string, jq, nested pwsh), the failure cause parsed from the output, and whether the harness reported success while the output contained a hard error. For every failure it also looks at the next shell command in the session to see what the model tried and whether that worked.
## Downstream shell analysis

These are regex heuristics checked by hand on samples. Expect a few percent noise per cell. The page says so.
These scripts consume observations; they are not part of gathering:

## Caveats worth repeating
```powershell
python scripts/analyze.py work
python scripts/scratch.py work
python scripts/build.py work --no-gallery
```

- Harness design dominates. Codex has no read or edit tools, so its shell share is structural. Compare models within one harness.
- Instructions leak in. Claude Code's auto mode tells the model to work through Bash; the page marks those families.
- Host shells differ per harness and per OS, so failure rates across harnesses are not comparable.
- One user, one machine, one task mix per run. Post your numbers and the picture gets better.
The report is work/shell-analysis.html with page-data.json. Analysis uses heuristic
classifications, not verified task-success labels. The optional gallery exposes
recorded commands and paths; keep private data local unless publication is authorized.
[Historical observations](reference.md) preserve the original analysis context.

## Files
## Naming and compatibility

- `SKILL.md` — instructions the agent follows
- `reference.md` — baseline numbers and verified harness facts from the first run
- `scripts/extract.py`, `analyze.py`, `scratch.py`, `build.py`, `template.html`
The repository was formerly shell-forensics and the model experiment was called
command specialist. Current names are **Command Model** (project) and **Shell
Gatherer** (collection). Historical artifacts and installed model aliases may keep
old names. On the development machine, the former repository folder is a
compatibility junction to command-model so existing worktrees, environments and
saved absolute paths remain accessible. New work should use the canonical name.

MIT license.

## Local command model experiment

The current [command-specialist direction](experiments/command_specialist/PURPOSE.md)
is English delegation of bounded command/cell work from a frontier agent to a
fresh local worker, returning verified evidence. It distinguishes the narrow
trained inspection pilot from the broader runtime and its measured host results.

The optional [command specialist pilot](experiments/command_specialist/README.md)
audits training-data quality, benchmarks small local models on bounded PowerShell
inspection and evidence-selection tasks, and trains a local LoRA adapter. Its
synthetic capability scores are separate from this repository's observed transcript
statistics and do not establish superiority to frontier models.
Loading
Loading