Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion experiments/command_specialist/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ The product goal is an agent-to-specialist handoff: the frontier agent supplies
intent and known context, the local specialist handles supported mechanical work,
and the agent receives compact faithful evidence with a retrievable raw result.
These five inspection operations are a capability pilot, not the complete command
runner product or a new chat interface. Automatic host integration remains pending.
runner product or a new chat interface. The [OpenCode2 adapter](opencode/README.md) now provides
file-page evidence handoffs through native read authorization. Routing the five
command-planning operations through the host remains pending.

Exact paths can now be supplied separately from task wording. The runner binds them
to request-local references for prediction and restores the original paths before
Expand Down
89 changes: 89 additions & 0 deletions experiments/command_specialist/opencode/HOST_RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Configured-host results — September 10, 2026

The OpenCode2 `inspect_file` adapter was exercised through the installed public
`opencode2 run --standalone` command, using the existing configured
`cliproxyapi/gpt-5.6-sol#xhigh` frontier route and local
`shell-specialist-pilot`. The installed host reported `0.0.0-beta-19398`.
The dev configuration was loaded with an isolated verification agent, database,
Quest state and telemetry. Shell, edits and delegation were denied for that agent.
The plugin was added only to the isolated launch configuration; no global release
was changed. The trained model, corpus and previous worktree were untouched.

## Observed operation

The frontier discovered the typed tool through Code Mode, supplied a known exact
path and evidence intent, received selected text and file line numbers, and used
native read for saved raw-page access and continuation/fallback. The local model
received numbered authorized text, never a path to regenerate.

| Case | Observed result |
| --- | --- |
| Unicode/apostrophe filename, whole five-line page | Correct ERROR and SUMMARY at source lines 3 and 5; eight generated selection tokens |
| Same file, offset 2 and limit 4 | Same correct source lines 3 and 5; raw four-line page reopened successfully through native read |
| 105-line file, first 100 lines | ERROR retained, truncation true and next offset 101; model also selected two irrelevant INFO lines |
| Continuation | Frontier used native read at offset 101 and returned the actual final SUMMARY from line 105 |
| One-line no-match file, twice | Selection failed; raw page retained with explicit native-read fallback; frontier followed it and distinguished no matches from selection failure |
| Existing file denied by native per-path read policy | Tool failed; no specialist result artifact for the denied target; no alternate executor used |
| Page exceeding 8,000-character context budget | Explicit fallback before model invocation; frontier reopened saved raw text with native read |

Every saved selected line was also reopened and compared with its original fixture
line; all were verbatim. This establishes text integrity, not selection relevance
or recall. All seven saved result records and their raw artifacts were reopened.
The no-match errors did not retain model token/timing telemetry, so their generation
counts are unknown, not zero.

## Timing and limits

Windows, sequential local calls, unchanged trained 1.5B baseline, 4K model context,
thinking off, temperature zero, 160-token generation cap. Cache state was not reset.
These are acceptance observations, not a statistically powered benchmark.

| Observation | Adapter wall | Awaited host tool call |
| --- | ---: | ---: |
| Initial correct small-page selection | 263 ms | Not separately recorded |
| Offset-page selection with model reload | 3,266 ms | 3,276 ms |
| First 100-line selection | 320 ms | 327 ms |
| Later 100-line selection | 621 ms | 632 ms |
| No-match selection failures | 1,008 / 1,027 ms | 1,016 / 1,035 ms |
| Context-budget fallback, no model call | 6 ms | 13 ms |
| Denied source read | No adapter packet | 6 ms |

The 3,266 ms call included 2,937 ms of Ollama-reported model load, rather than a
uniform warm-call assumption. The adapter timing includes native read, Python
startup/transport, local inference, validation, raw/record persistence and record
reopening, excluding the final timing-stamp write and host serialization.

Two timed full CLI sessions took **61.4 seconds** and **129.9 seconds**, respectively,
from standalone process launch to final frontier response. They included discovery,
multiple inspections, frontier generation, native fallback/reopening, and, in the
second session, incorrect tool-name attempts and recovery. These are different
multi-case workloads, not comparable speedup arms. No frontier token/cost saving,
p50/p95, cost per correct task, or whole-workflow speedup is claimed.

## Failures discovered during verification

The first host run could not discover the plugin: the initial configuration named
a module file. This host requires `plugins` entries naming directories with an
index entrypoint. The final implementation and instructions use that format.

A later frontier run incorrectly used `tools[item.path]` when discovery returned
`tools.inspect_file`, then tried another incorrect name. It eventually recovered
to `tools.inspect_file(...)` and completed the requested continuation and fallback.
Those retries remain in the full-operation time. They are evidence that successful
adapter calls alone do not establish a reliable or faster frontier workflow.

The local model selected irrelevant INFO lines in both long-page trials and failed
both no-match trials. It must remain an opt-in evidence aid with visible coverage
and raw access. Improving negative-case behavior and evidence precision belongs in
a fresh development evaluation before expanding or retraining the specialist.

An auxiliary `debug config` command timed out waiting for a background service;
verification used task-owned standalone sessions instead. The isolated worktree
initially lacked Acorn; `npm ci` restored its declared dependency. Final checks:
26 Python tests, six parser tests, three adapter core checks, and `git diff --check`.

Private reproduction evidence is under the task worktree's `work/host-smoke/` and
the OS temporary `command-specialist-host/` directory. Raw logs are not published.
The next integration work is command planning/execution through appropriate native
host authorization and a larger real-intent development benchmark; this change
only integrates bounded file-page evidence extraction.
108 changes: 108 additions & 0 deletions experiments/command_specialist/opencode/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# OpenCode2 file-evidence handoff

`inspect_file` is the first installed-host integration for the local specialist.
The frontier agent passes an exact known path and a short evidence request. The
adapter calls the host's registered `read` implementation, passes its text page to
`shell-specialist-pilot` on local Ollama, and returns only selected **verbatim**
lines, absolute file line numbers, coverage status, and a saved raw-page path.
Paths never pass through the local model. Existing request-local path bindings in
`run.py` remain the command-planning interface for the separate five-operation CLI.

This integration deliberately starts with evidence extraction from file pages.
It does not yet route arbitrary shell work or the CLI's five planning operations.
The caller already knows the path and read bounds, so there is no planning-model
call. Native `read` remains the right tool when the caller needs all lines.

## Configure

Keep the existing host configuration and add this **directory** to `plugins`.
OpenCode2 requires a directory containing `index.mjs`, not the entrypoint filename:

```json
{
"plugins": [{
"package": "C:/path/to/shell-forensics/experiments/command_specialist/opencode",
"options": {
"python": "C:/path/to/existing/.venv/Scripts/python.exe",
"model": "shell-specialist-pilot",
"baseURL": "http://127.0.0.1:11434",
"artifacts": "C:/path/to/private/specialist-results",
"timeoutMs": 15000
}
}]
}
```

Only Python's standard library is required at inference time. Reuse the existing
Ollama model; installation does not train, download or replace it. The Python
executable defaults to `COMMAND_SPECIALIST_PYTHON`, then `python`; the other defaults
are shown above except artifacts, which default to the OS temporary directory's
`command-specialist-host` subdirectory. The endpoint must be loopback HTTP. Config
options are operator-owned and cannot be supplied in model tool arguments.

Do not activate an unreviewed plugin in a production release. This repository's PR
prepares the adapter; global setup/release activation is separate.

## Agent use

Discover `inspect_file` with the host's Code Mode `search`, then call the returned
concrete tool (normally `tools.inspect_file`):

```javascript
const packet = await tools.inspect_file({
path: "logs/build [draft]'s.log",
evidence_request: "Return every ERROR and the final SUMMARY.",
offset: 1,
limit: 100
});
return packet;
```

Use `read` on `packet.raw_result` to reopen the original captured page. The text
artifact has page-relative lines; `packet.source_lines` contains original file
line numbers. A sibling JSON record preserves the full native tool result, host
and specialist identities, timing, requested bounds, and the returned packet.
Neither artifact is a complete-file claim when the native read was truncated.
Artifacts are private local files and are not automatically deleted.

- `status: selected` means indices were validated and evidence copied from the
native page. It does **not** certify recall, task success, or absence of errors.
- `truncated: true` and `next_offset` mean more source lines exist. Follow the
supplied native-read fallback, or request another specialist page explicitly.
- `status: fallback` retains raw output and supplies native `read` arguments.
Unsupported native output, shortened lines, excessive context, invalid indices,
unavailable Python/Ollama, or timeout do not become successful empty selections.
- Permission denials and native read failures propagate as tool errors. They
trigger no model invocation or new result artifact and are never retried through
an independent filesystem or shell executor.

## Boundaries and authorization

The adapter exposes the native `read` permission category and calls the captured
registered native read with the original session, agent, message and call identity.
That implementation retains path resolution, per-resource permission prompts and
denials, external-directory checks, and nearby instruction loading. The adapter
never reads the source file itself. Its subprocess only receives already-authorized
text and communicates with local Ollama. No shell commands are generated or run.

The page is at most 100 lines, and text plus request must fit the pilot's 8,000
character budget. The model selects ordered line indices; the adapter independently
validates them and reconstructs all returned text. Native shortened-line markers
cause fallback rather than a false exact-text claim. Empty pages need no model call.
No-match results remain model selections, not deterministic absence proofs.

`wall_ms` measures the native read (including permission wait), Python startup and
transport, local inference, validation, raw/record writing and record reopening;
it excludes the final timing-stamp write, host serialization, frontier generation,
and discovery. Measure the awaited host call and entire frontier operation
separately before making a whole-workflow performance claim.

## Checks

```powershell
node --test experiments/command_specialist/opencode/core.test.mjs
python -m unittest discover -s experiments/command_specialist -p 'test_*.py'
```

The core checks supplement real configured-host use. See `HOST_RESULTS.md` for
observed host runs, failures, exact scope, and remaining work.
22 changes: 22 additions & 0 deletions experiments/command_specialist/opencode/core.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import test from 'node:test'
import assert from 'node:assert/strict'
import { selectedEvidence, textPage } from './index.mjs'

test('selection preserves bytes and translates page indices to file lines', () => {
const lines = ['routine', "ERROR: café [draft]'s file_1 {{build}}", '', 'SUMMARY: failed\u2028verbatim']
const result = selectedEvidence(lines, [2, 3, 4], 21)
assert.deepEqual(result, { evidence: lines.slice(1), source_lines: [22, 23, 24] })
})

test('invalid selections cannot fabricate or duplicate evidence', () => {
for (const indices of [[0], [3], [1, 1], [2, 1], [true], [1.5], ['1'], null]) {
assert.throws(() => selectedEvidence(['one', 'two'], indices, 1))
}
})

test('unsupported or shortened native output cannot become exact evidence', () => {
assert.throws(() => textPage({ output: { type: 'list-page', entries: [] } }))
assert.throws(() => textPage({ output: { type: 'text-page', content: 'value... (line truncated to 2000 chars)', offset: 1, truncated: false } }))
assert.deepEqual(textPage({ output: { type: 'text-page', content: 'one\n\nthree\u2028same line', offset: 9, truncated: true, next: 12 } }),
{ lines: ['one', '', 'three\u2028same line'], offset: 9, truncated: true, next: 12 })
})
Loading