Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/agent-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ jobs:
run: python -m unittest discover -s experiments/command_specialist -p test_bindings.py -v
- name: Native and PowerShell contract invariants
run: python -m unittest discover -s experiments/command_specialist -p test_contract.py -v
- name: English delegation evidence and authorization invariants
run: python -m unittest discover -s experiments/command_specialist -p test_delegate.py -v
- name: Frozen main candidate and patch notes
run: python scripts/check_release.py
- name: Session workspace isolation
Expand Down
114 changes: 114 additions & 0 deletions experiments/command_specialist/ENGLISH_HANDOFF.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# English handoff contract (prototype v1)

A frontier supplies one English task, exact target bindings, relevant runtime/API
context, constraints, and an observable completion condition. The frontier does
not supply generated source, escaped shell strings, or a command sequence.
Missing target/API context is a context failure, not evidence of model incapacity.

Each invocation creates a fresh worker message history and unique evidence directory.
It retains internal action/result history only for this delegation. It returns one
packet, never recursively delegates, and never resumes a previous local chat.
Ollama may keep model weights warm; that is not conversation persistence.

The worker chooses compact typed actions. Premature finish is rejected with observed failure feedback. A write action requests raw file content
in a separate inference rather than putting a program inside a JSON string. Source generation receives an English view of the same task, saved source and actual last execution result; JSON action history stays in the selector. Native
write/read/process operations preserve exact target bindings. Actual execution
feedback goes back to the same worker for ordinary repair. This split is an
experimental baseline, not a measured superiority claim.

The result distinguishes worker explanation from observed actions, saved files,
SHA-256 identities, exit codes, verbatim stdout/stderr and raw model responses.
Only a runtime-checked completion condition can set `verified`. A generated file,
`finish`, or exit zero alone cannot. A run must execute the same bytes that were
written and remain saved, exit zero without a limit violation, and match the
required stdout exactly, including platform newlines (Windows print uses `\r\n`). The worker must also finish within its action budget.
This verifies the stated stdout condition, not arbitrary semantic correctness.

## Minimal local CLI

From the repository root, invoke Python 3.12:

```
python experiments/command_specialist/delegate.py --task-file task.json --root scratch --allow-execute
```

Example handoff (scratch must already exist):

```json
{
"task": "Write a Python program that calculates the sum of integers 1 through 10. Run it, inspect the actual output, then finish.",
"targets": {"program": "sum.py"},
"context": "Python 3.12; standard library only.",
"constraints": ["Modify only the bound program target"],
"completion": {"target": "program", "stdout": "55\r\n"}
}
```

This CLI supports trusted Python write/read/run tasks only. `--allow-execute`
explicitly authorizes native generated code with the caller's account privileges;
path binding constrains direct adapter operations, not Python's capabilities.
Natural-language constraints are model instructions, not a security sandbox.
Use disposable data. The CLI is not registered as a new host tool. A future host
adapter must use its native capabilities, session identity, hooks, instructions,
permissions and cancellation. CLI authorization does not substitute for those.
A missing authorization terminates as denied before model or native actions; no
alternate executor or model is attempted.

Default bounds: 12 actions, 300 seconds for model/operation work, 20 seconds per
execution, 8192 output tokens per inference, 32768 context tokens, and 64 KiB
execution output. Execution output is spooled to per-action files and monitored;
limit detection can overshoot between polls, and raw files preserve that evidence.
Model HTTP timeout bounds the client's wait; it does not prove server inference
was cancelled. Process cleanup and final persistence can exceed the work deadline.
Ctrl-C terminates the running process tree and saves a cancelled result. Generated
programs that detach children are outside this trusted prototype's lifecycle
coverage. No concurrent benchmark or shared model setting change is implied.

Raw artifacts are saved before execution and after each observation, with atomic
replacement of `result.json`. Abrupt host termination may leave status `running`;
never interpret that as success. The packet provides the artifact path. Reopen it
and the actual target to check persistence. Failure, exhausted budgets, unavailable
worker and denial preserve distinct statuses and do not silently fall back.

## Representative PTY handoff (next stage)

The full contract must accommodate this without the frontier writing code:

> Write and run the bound checker using node-pty. Launch Bun on the exact recorder
> path with inherited environment and working directory, a 100x24 xterm-256color
> PTY, and a normalized absolute script path. Observe the diagnostic banner, send
> Escape once after startup, and report actual cancellation and child exit status.
> Scope: Startup/cancel only; no physical-key evidence generated.

The frontier supplies checker and recorder bindings, the actual banner/cancel
markers and concise node-pty API context. Completion requires observations from
the real recorder and reopened saved report, not a replacement printing markers.
This Python-only prototype cannot yet perform that workload, host-native
permissions, or configured frontier return integration. The earlier file adapter
remains separate pending a verified replacement; its result is not proof of
English delegation.

## Non-quantized runtime and capacity

The English loop requires `shell-specialist-f16`: the original local
Qwen2.5-Coder-1.5B-Instruct weights exported as FP16 with the unchanged pilot LoRA
adapter. It checks Ollama model metadata before inference and rejects quantized
or unknown precision. FP16 is non-quantized 16-bit floating point, not FP32.
The old Q4 pilot remains installed for historical reproducibility but is not used
by this loop. No retraining or change to the trained adapter is involved.

`--num-ctx 32768 --num-predict 8192 --seconds 300 --max-actions 12` are explicit
CLI controls and the defaults. The context window includes history and generated
output. A conservative UTF-8 byte upper bound plus per-message overhead reserves
the output allowance; oversized history fails instead of intentionally truncating
instructions. This can reject text before the model's token capacity is actually
full. Exact tokenizer budgeting is future work. Increasing output is permission
to generate more, not a minimum response length. Long-context reliability has not
been established by short task smokes.

For a local runtime setup, use the existing llama.cpp `convert_hf_to_gguf.py` on
the original local base snapshot with `--outtype f16`. Import that GGUF with the
existing pilot's exact `ADAPTER` and chat template under `shell-specialist-f16`
using `ollama create`; do not pass a quantization option. Verify `ollama show`
reports F16 and compare the adapter identity. The tested machine's BF16 GGUF import
failed validation, while FP16 succeeded. Weights and setup logs stay private.
57 changes: 57 additions & 0 deletions experiments/command_specialist/ENGLISH_RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# English loop observations — September 10, 2026

The standalone public CLI now accepts one English handoff and returns a saved,
runtime-verified result from a fresh local worker. This is step 1/2 of the handoff
plan, not configured OpenCode2 delegation or the representative PTY recorder test.

The active runtime is non-quantized FP16 Qwen2.5-Coder 1.5B with the original pilot
adapter. Original BF16 training weights were available; BF16 GGUF import failed
Ollama validation, while FP16 import succeeded. `ollama show` reported F16 and
`ollama ps` reported 32768 context, 4.2 GB and 100% GPU on the RTX 3070. These are
runtime observations, not peak memory measurements. The original Q4 model is not
used by the new loop. No retraining occurred.

## Actual public operations

All model calls used local Ollama, 32768 context and an 8192 output-token cap,
12 actions and a 300-second worker budget. Private task/result files and raw model
responses are retained under the owned worktree's ignored `work/` directory.

- Fresh sum program: actual missing-file exit 2, local source generation/write,
rerun exit 0 with `55`, then finish. Saved source computes the sum. 2.235 seconds.
- Fresh sum-of-squares program with Unicode and an apostrophe in its exact path:
missing-file exit 2, write, rerun exit 0 with `140`, finish. 2.343 seconds.
- Existing broken program: actual `NameError` for `valuez`, local correction to
`values`, rerun exit 0 with `16`, finish. 1.594 seconds.
- Final-revision fresh squares target: exit 2, write, exit 0 with `140`, finish.
2.843 seconds. This includes the corrected source newline handling.
- Without execution authorization: denied before any model/action call.
- One-action budget: exhausted and unverified even though the existing program
exited zero with the expected stdout.

Times are individual whole CLI-worker measurements including inference, native
operations and persistence, not p50/p95, a speedup claim, or paid charges. They
exclude frontier conversation overhead. Every successful result and saved source
was reopened; recorded precision, action exits and source identity were checked.

## Failures that informed the implementation

The initial Q4 probe repeatedly ran a missing file. A first FP16 prompt generated
correct source but repeatedly copied a write example. Removing that example let
actions advance, but JSON action history contaminated source generation: one
output was a task-shaped dictionary that executed with empty stdout. The runtime
correctly refused success. English-only source-generation context containing the
same task, saved source and actual execution feedback yielded the successful runs.
Multiple variables changed; these observations do not isolate quantization effects.

Action selection still wastes a run on absent files, and final worker explanations
are weak (often just the filename). Deterministic evidence verification is needed.
The model has not demonstrated broad English task reliability or long-context
reasoning. An 8K output cap is configured; these short samples did not use 8K tokens.

Required compileall and both existing five-test suites passed. Three new concise
invariant tests passed for denial/fresh histories, exact target boundaries, and
verification against actual executed/saved bytes. They supplement the real runs.
Host hooks/permissions, PTY startup/cancel, detached child cleanup, and external
frontier delivery remain unverified. No host registration or global installation
was changed by this prototype.
6 changes: 6 additions & 0 deletions experiments/command_specialist/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -261,3 +261,9 @@ frontier endpoint. Report failures and fallback rate, not only answered cases.
Integrate only once measured gains survive the whole host operation. Expensive or
destructive actions retain the host's permission checks; model output never grants
authorization. Keep full raw results retrievable when a compact packet is insufficient.

## English delegation prototype

See [the English handoff contract](ENGLISH_HANDOFF.md) for the fresh-worker local
CLI, non-quantized runtime, configurable budgets and verification boundaries.
This is separate from the earlier read-only inspection adapter.
Loading
Loading