|
| 1 | +# Command specialist: purpose and evidence |
| 2 | + |
| 3 | +This is the current project direction. Historical reports describe their own |
| 4 | +experiments; they do not redefine the product. The goal is to reduce frontier |
| 5 | +agent time and token use on command/cell work while preserving task correctness. |
| 6 | +Training throughput, model size and isolated inference speed are supporting |
| 7 | +measurements, not the product outcome. |
| 8 | + |
| 9 | +## Division of work |
| 10 | + |
| 11 | +The frontier owns the user goal, higher-level reasoning, task decomposition and |
| 12 | +whether a result is sufficient to continue. It delegates a bounded mechanical |
| 13 | +job in English, with exact known paths, constraints, relevant environment/API |
| 14 | +facts and a checkable result. It should not have to write the command or program |
| 15 | +that the local worker is supposed to produce. |
| 16 | + |
| 17 | +The local worker owns the delegated implementation: inspect actual inputs, |
| 18 | +construct commands or code within the supported executor, execute, inspect the |
| 19 | +result and repair ordinary failures. One delegation creates one fresh worker |
| 20 | +lifecycle; it can contain several internal actions. It returns compact observed |
| 21 | +results and a retrievable raw artifact to the same frontier turn, then ends. |
| 22 | +Warm model weights are compatible with fresh task context. A later delegation |
| 23 | +receives explicit inputs and prior artifacts, not hidden conversation memory. |
| 24 | + |
| 25 | +The runtime owns exact path binding, input inspection, execution limits, process |
| 26 | +cleanup, saved evidence and completion checks. Unknown contents, schemas and API |
| 27 | +behavior must be inspected or reported as missing context. They are not details |
| 28 | +for the model to guess. Verification must check the requested result: exit zero |
| 29 | +or a printed success marker does not establish that saved artifacts are correct. |
| 30 | +The frontier owns recovery after an unsuccessful handoff; measure that cost too. |
| 31 | + |
| 32 | +Python is the current trusted local execution prototype, not the definition of |
| 33 | +the product. The original PTY startup/cancel job remains a representative future |
| 34 | +boundary. The project is neither restricted forever to five reads nor committed |
| 35 | +to turning the local worker into an independent general-purpose coding agent. |
| 36 | + |
| 37 | +## What was trained versus what was tested |
| 38 | + |
| 39 | +| Layer | Actual scope | What it does not establish | |
| 40 | +| --- | --- | --- | |
| 41 | +| Pretrained Qwen2.5-Coder model | Existing coding/instruction abilities before this project | Reliable performance on every local task | |
| 42 | +| Preserved pilot LoRA | 486 examples: 366 typed inspection plans and 120 evidence-line selections | Post-training for arbitrary Python writing, repair or multi-step English execution | |
| 43 | +| Original inspection runner | Five read/search/list/JSON-field operations and faithful evidence extraction | A complete command/cell delegation product | |
| 44 | +| Codex English prototype | A real configured MCP handoff to a fresh local Python worker and return to the originating frontier | Transparent shell replacement, all native per-command permission semantics, or desktop UI verification | |
| 45 | +| Expanded ten-stage experiment | Chained/grouped mechanical work with independent artifact assessment | Ten independently arising frontier decisions or ten required native tool calls | |
| 46 | + |
| 47 | +The 486 rows comprised 420 synthetic fixture examples and 66 examples with |
| 48 | +synthetic instructions derived from historical read commands. The data pipeline |
| 49 | +later recovered 70,611 command observations, not 70,611 gold training examples. |
| 50 | +Recorded exit zero does not prove task correctness. See [data results](DATA_RESULTS.md). |
| 51 | +The adapter's training objective and the English code-generation objective are |
| 52 | +different. A failed broader task cannot by itself establish a model-size ceiling. |
| 53 | + |
| 54 | +## Why the successful frontier handoff was possible |
| 55 | + |
| 56 | +The [single-handoff CSV smoke](codex/RESULTS.md) used the real Codex host and |
| 57 | +preserved FP16 specialist adapter. Both arms passed original and changed-input |
| 58 | +checks. Normal shell work took 71.075 seconds; English delegation took 43.118 |
| 59 | +seconds, including 2.813 seconds inside the local worker. Frontier native calls |
| 60 | +fell from six to one, alongside one English handoff. |
| 61 | + |
| 62 | +That success demonstrates a working handoff/return path and one successful task. |
| 63 | +The pretrained model already had coding ability; the harness could use that even |
| 64 | +though the project adapter was trained on a narrower objective. We did not isolate |
| 65 | +the adapter's causal contribution. Fewer frontier interactions are a plausible |
| 66 | +source of savings, but this single pair also includes different startup, approval |
| 67 | +and recovery costs. It does not prove typical savings or general training success. |
| 68 | + |
| 69 | +The [ten-stage pilot](codex/TEN-STAGE-PILOT.md) changed the workload. Normal Codex |
| 70 | +batched it into five native calls and passed all original/alternate checks in |
| 71 | +93.998 seconds. Ten handoffs took 200.381 seconds and two grouped handoffs took |
| 72 | +156.575 seconds; both failed all stage assessments. One local worker satisfied a |
| 73 | +stdout marker without creating the required artifact. That exposed a completion |
| 74 | +contract defect, not successful execution. Dependent stage failures are not ten |
| 75 | +independent estimates of model accuracy. |
| 76 | + |
| 77 | +These observations coexist: one handoff worked and was faster in its smoke; |
| 78 | +the expanded task did not. Neither observation should be erased or generalized. |
| 79 | +We drifted by treating broader code generation as if it had already been trained, |
| 80 | +then pursuing prompts, model capacity and training throughput before closing the |
| 81 | +handoff's grounding and correctness gaps. |
| 82 | + |
| 83 | +## Current implementation versus ongoing experiments |
| 84 | + |
| 85 | +The merged prototype and its recorded smokes are documented in the |
| 86 | +[English contract](ENGLISH_HANDOFF.md) and [Codex hookup](codex/README.md). |
| 87 | +Local work has explored explicit input inspection, artifact acceptance checks, |
| 88 | +raw prompt changes, adapter removal, a separate 3B FP16 model and training-speed |
| 89 | +profiling. These are diagnostics, not a new verified end-to-end result. Some |
| 90 | +runtime changes remain unmerged; a new description is not evidence they are |
| 91 | +available through the configured host. Preserve their raw failures and source |
| 92 | +identities. Do not transfer old measurements to a changed harness. |
| 93 | + |
| 94 | +The 3B diagnostic runs but partly offloads to CPU at 32K context on the test GPU; |
| 95 | +it is not an established replacement. A short-example checkpoint experiment sped |
| 96 | +up training on 32 old training rows with identical saved adapter bytes. It neither |
| 97 | +trains the English execution objective nor establishes product latency savings. |
| 98 | +No new 5,000-example English-execution training run has been completed. |
| 99 | + |
| 100 | +## Next work and decision gates |
| 101 | + |
| 102 | +1. Re-establish one grounded English handoff through the real frontier host. |
| 103 | + Supply observable inputs, preserve actual execution errors, check saved task |
| 104 | + artifacts and replay on changed inputs. Reopen the result received by the |
| 105 | + originating frontier. Start with the previous successful task shape before |
| 106 | + expanding it. Do not count a local-only probe as this verification. |
| 107 | +2. Compare that frozen setup with the same frontier doing normal shell work. |
| 108 | + Let the baseline batch naturally. Repeat in alternating order, retain failures, |
| 109 | + and include repair/fallback in total time through frontier completion. Report |
| 110 | + frontier cached/uncached input and output separately from local model tokens; |
| 111 | + token counts are not measured charges. Report independent evaluator time |
| 112 | + separately from user-operation time. |
| 113 | +3. Extend to several independently arising jobs and dependent jobs. Distinguish |
| 114 | + mechanical stages, frontier decisions, handoff events and internal actions. |
| 115 | + More handoffs are not inherently better. Choose boundaries for useful work, |
| 116 | + not to inflate the baseline or projected savings. |
| 117 | +4. If failure analysis requires training, build execution-verified examples for |
| 118 | + the actual runtime protocol: grounded English jobs, valid implementations, |
| 119 | + actual failures and repairs, and faithful return evidence. Keep frozen held-out |
| 120 | + sessions/families separate; never turn the benchmark answers into training |
| 121 | + targets. Counts alone do not establish coverage. Evaluate accuracy and whole |
| 122 | + handoff cost before promoting a new adapter or model. |
| 123 | + |
| 124 | +Only investigate a lower-level optimization when an observed handoff failure or |
| 125 | +bottleneck motivates it. State the connection and the success criterion first. |
| 126 | +Do not automatically launch more model comparisons or training-speed work because |
| 127 | +an earlier diagnostic made those measurements available. User steering can change |
| 128 | +priority; update this direction explicitly instead of silently replacing the goal. |
0 commit comments