Measure ten-stage English delegation against native Codex shell

experiments/command_specialist/codex/README.md

@@ -86,3 +86,51 @@ its internal Python actions do not inherit all native shell resource checks.
 It is not general arbitrary-shell delegation or the actual PTY recorder workload.
 The old OpenCode2 inspection adapter is unrelated. The local 1.5B specialist can
 fail; keep those results and frontier recovery cost visible.
+
+
+## Ten-stage continuous session
+
+Run `ten_step.py run --arm baseline`, then `--arm chained`, then `--arm grouped`
+from the repository root with the MCP-enabled Python environment. Never run the
+arms concurrently. `prepare` creates a fixture and prompt for manual use in a new
+Codex chat instead. The same ten-stage synthetic incident packet covers inventory,
+logs, configuration differences, tests, changed paths, command durations, failure
+details, a combined summary and a SHA-256 manifest. Later stages consume earlier
+outputs. No private transcript text is used in fixture data.
+
+Baseline may batch all work in one script and choose its own native call count.
+Chained requests ten serial English handoffs, each with a new local process and
+history. Grouped requests two handoffs of five stages. This tests ten mechanical
+steps known at the start of one session; it does not force ten frontier decisions
+or simulate discoveries that were unavailable in the original prompt.
+
+All arms get the installed interpreter path. The runner preserves ordinary host
+startup, instructions, tool discovery and approval costs. The current loop uses
+the nonquantized FP16 specialist, not a new model or trained adapter. Each local
+worker has its existing action/time limits. Failures are retained without silent
+native fallback. A success marker alone is insufficient: the external evaluator
+reopens every saved JSON and checks the content and manifest, then reruns the
+saved scripts on changed inputs in a separate directory. All twenty checks, input
+integrity, expected handoff count, and local completion must pass. External
+assessment is outside the timed chat. These scripts are trusted local code, not
+sandboxed by the harness.
+
+Compare saved runs with:
+
+```
+python experiments/command_specialist/codex/compare_ten.py BASELINE/summary.json CHAINED/summary.json GROUPED/summary.json --out work/ten-stage-comparison.json
+```
+
+Keep each summary beside its original run.json and fixture.json. Savings require
+successful runs with matching source hashes, input hashes, model and effort.
+Token usage separates cached and uncached frontier input, frontier output, and
+local model input/output. It does not estimate actual charges. event-times.jsonl
+records arrival timestamps for raw events; the union of started/completed tool
+spans avoids double counting overlap. Time outside those spans includes startup
+and frontier work and must not be labeled entirely model inference.
+
+Subtracting observed serial local worker time gives a conditional instant-worker
+floor with every other cost unchanged. It is not an achievable optimum or a
+ceiling on redesigned workflows. Failed arms receive no savings claim. Repeat
+successful matched AB/BA series before claiming typical savings; one run per arm
+is a pilot that can reveal failure modes and batching effects.

experiments/command_specialist/codex/TEN-STAGE-PILOT.md

@@ -0,0 +1,79 @@
+# Ten-stage incident packet pilot
+
+September 10, 2026. Three sequential fresh Codex CLI sessions used gpt-6-astra,
+low effort, existing login/configuration and normal approval review. Local workers
+used shell-specialist-f16 with the unchanged pilot adapter, 32768 context, 8192
+output allowance and one fresh bounded process/history per handoff. No quantized
+model or retraining was introduced. This is one observation per arm, not a
+statistical benchmark or a claim about general command work.
+
+Eight recent local chat transcripts contained 78 shell-tool events. Aggregate
+patterns included reading files, inventory, log search, JSON processing, Git
+state, validation, saved evidence, path handling and table data. Those patterns
+informed ten synthetic stages; no transcript text or real project data appears
+in the fixture. Private source hashes/provenance and raw events remain local.
+
+| Strategy | Handoffs | Native calls | Whole chat | Correct original stages | Correct alternate stages |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Normal Codex shell | 0 | 5 | 93.998 s | 10/10 | 10/10 |
+| One handoff per stage | 10 | 0 | 200.381 s | 0/10 | 0/10 |
+| Two groups of five | 2 | 0 | 156.575 s | 0/10 | 0/10 |
+
+The baseline could batch efficiently and implemented one script. All ten stages
+were known at session start, with later stages consuming earlier outputs. This
+is not ten independently arising requests, nor ten forced native commands.
+Whole-chat timing includes startup, instruction/tool discovery, approval, model
+work, execution and frontier completion. Independent original/alternate artifact
+assessment runs afterward and is excluded from that timing. Input bytes stayed
+unchanged in all three arms. Source and fixture hashes matched across the series.
+
+| Strategy | Frontier input | Cached input (included) | Uncached input | Frontier output | Local input / output |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Normal shell | 136130 | 123776 | 12354 | 2134 | 0 / 0 |
+| Ten handoffs | 330385 | 310272 | 20113 | 2425 | 215216 / 4117 |
+| Two handoffs | 254899 | 239360 | 15539 | 1588 | 92908 / 7417 |
+
+Input totals accumulate across model requests; they are not a single context
+window size. Cached input is already included in input. These are observed token
+counts, not actual dollar charges. Two handoffs used 25.6% fewer frontier output
+tokens than baseline, but more input tokens and failed the task; that is not a
+successful-task efficiency improvement.
+
+Ten handoffs made 151 local model calls and recorded ten failed executions.
+Nine workers exhausted their limits. The one execution-verified worker only
+printed its expected marker and created no requested JSON. The external oracle
+rejected it. The first stage wrote a non-JSON inventory; later outputs were
+missing. Two grouped workers made 32 model calls and seven failed executions;
+one exhausted its limits and one failed. Detailed actions and stderr remain in
+local result artifacts. Dependent failures are not independent model trials.
+
+| Strategy | Observed tool span union | Local worker time within spans | Outside observed tool spans |
+| --- | ---: | ---: | ---: |
+| Normal shell | 8.841 s | 0 s | 85.157 s |
+| Ten handoffs | 98.306 s | 64.983 s | 102.075 s |
+| Two handoffs | 100.688 s | 87.937 s | 55.887 s |
+
+Tool spans use event arrival timestamps. Outside-span time includes startup and
+frontier work, not exclusively model inference. Removing all observed local
+worker time, while holding every other cost fixed, leaves 135.398 s for ten
+handoffs and 68.638 s for two. The latter is arithmetically 25.360 s below the
+baseline, but both underlying runs failed. Neither number estimates an accurate
+implementation or a global best possible saving. Correct repair, different
+handoff boundaries and frontier behavior would change the timings.
+
+The experiment establishes that ten fresh lifecycles can be chained through the
+real host, with saved results returned to the frontier. It does not establish
+reliable task completion or time savings. Correctness and useful artifact-level
+completion checks are the next bottlenecks. Grouping appears worth investigating,
+but faster inference alone does not fix incorrect programs or redundant handoffs.
+
+An earlier 134.725 s baseline preflight also passed all twenty artifact checks.
+Its original summary falsely flagged an input change because the collector used
+Windows default decoding for a Unicode filename. Explicit UTF-8 decoding fixed
+the collector; a separate correction artifact preserves the original evidence.
+The preflight lacked the supplied interpreter context and is excluded above.
+
+Reproduce with the three ten_step.py arms and compare_ten.py documented in
+README.md. Repeat successful matched runs in alternating order before estimating
+typical latency or savings. No raw chat logs, generated worker programs, private
+machine paths or model weights belong in this report's public source.

experiments/command_specialist/codex/bench.py

@@ -8,6 +8,7 @@
 import subprocess
 import sys
 import time
+import threading
 import uuid
 
 HERE = Path(__file__).resolve().parent
@@ -77,10 +78,10 @@ def codex_command():
     return [path]
 
 
-def run(directory, model, effort):
+def run(directory, model, effort, *, collector=None, timeout=600, extra_sources=()):
     if (directory / 'events.jsonl').exists():
         raise ValueError('Run evidence already exists; prepare a fresh workspace')
-    sources = [HERE / 'server.py', HERE / 'bench.py', HERE.parent / 'delegate.py']
+    sources = [HERE / 'server.py', HERE / 'bench.py', HERE.parent / 'delegate.py', *extra_sources]
     source_hashes = {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in sources}
     root = directory / 'workspace'
     command = codex_command() + ['exec', '--json', '--approve-for-me', '-C', str(root),
@@ -90,18 +91,30 @@ def run(directory, model, effort):
     start = time.perf_counter()
     timed_out = False
     with (directory / 'events.jsonl').open('wb') as output, (directory / 'stderr.txt').open('wb') as errors, (directory / 'prompt.txt').open('rb') as prompt:
-        process = subprocess.Popen(command, stdin=prompt, stdout=output, stderr=errors)
+        process = subprocess.Popen(command, stdin=prompt, stdout=subprocess.PIPE, stderr=errors)
+        def capture():
+            with (directory / 'event-times.jsonl').open('w', encoding='utf-8') as times:
+                for index, line in enumerate(iter(process.stdout.readline, b'')):
+                    output.write(line)
+                    output.flush()
+                    times.write(json.dumps({'line': index, 'elapsed_ms': round((time.perf_counter()-start)*1000)})+'\n')
+                    times.flush()
+        reader = threading.Thread(target=capture, daemon=True)
+        reader.start()
         try:
-            process.wait(timeout=600)
+            process.wait(timeout=timeout)
         except (subprocess.TimeoutExpired, KeyboardInterrupt):
             timed_out = True
             subprocess.run(['taskkill', '/PID', str(process.pid), '/T', '/F'],
                            stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=15)
             process.wait(timeout=15)
+        reader.join(timeout=15)
+        if reader.is_alive():
+            raise RuntimeError('Event capture did not close after Codex exit')
     write_json(directory / 'run.json', {'source_sha256': source_hashes, 'source_changed_during_run': any(hashlib.sha256(p.read_bytes()).hexdigest() != source_hashes[p.name] for p in sources), 'model': model, 'effort': effort, 'exit_code': process.returncode, 'timed_out_or_cancelled': timed_out,
         'wall_ms': round((time.perf_counter()-start)*1000), 'command': command,
         'billing': 'Existing Codex login; tokens are usage, not a dollar charge estimate'})
-    return collect(directory)
+    return (collector or collect)(directory)
 
 
 def collect(directory):

experiments/command_specialist/codex/compare_ten.py

@@ -0,0 +1,57 @@
+"""Compare whole-session ten-stage evidence without rewarding failed work."""
+import argparse
+import json
+from pathlib import Path
+
+
+def compare(rows):
+    baseline = next(r for r in rows if r['arm'] == 'baseline')
+    results = []
+    for row in rows:
+        usage = row['frontier_usage']
+        result = {k: row[k] for k in ('arm', 'success', 'wall_ms', 'delegations', 'native_commands',
+            'completed_stages', 'alternate_stages', 'local_worker_ms', 'outside_tool_span_ms')}
+        result['frontier_input_tokens'] = usage['input_tokens']
+        result['frontier_cached_input_tokens'] = usage['cached_input_tokens']
+        result['frontier_uncached_input_tokens'] = usage['input_tokens'] - usage['cached_input_tokens']
+        result['frontier_output_tokens'] = usage['output_tokens']
+        result['local_input_tokens'] = sum(r['input_tokens'] for r in row['local'])
+        result['local_output_tokens'] = sum(r['output_tokens'] for r in row['local'])
+        result['local_model_calls'] = sum(r['model_calls'] for r in row['local'])
+        result['failed_local_executions'] = sum(r['failed_executions'] for r in row['local'])
+        matched = all(row[k] == baseline[k] for k in ('model', 'effort', 'benchmark_identity'))
+        eligible = row['success'] and baseline['success'] and matched
+        result['savings_eligible'] = eligible
+        if eligible:
+            result['saved_ms'] = baseline['wall_ms'] - row['wall_ms']
+            result['saved_percent'] = 100 * result['saved_ms'] / baseline['wall_ms']
+            result['instant_worker_floor_ms'] = row['wall_ms'] - row['local_worker_ms']
+            result['instant_worker_saved_ms'] = baseline['wall_ms'] - result['instant_worker_floor_ms']
+        results.append(result)
+    return {'results': results, 'limits': [
+        'Each row is one whole Codex CLI session, not a latency distribution or billing estimate.',
+        'Only successful matching-model runs receive savings. Verify matching source/config and fixtures before pooling.',
+        'Instant-worker floor subtracts observed serial local worker wall time only; all other costs are held fixed.',
+        'Outside-tool span is observed event timing, including startup and frontier work, not pure model inference.',
+        'Independent changed-input evaluation happens after the timed chat.']}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('summaries', nargs='+', type=Path)
+    parser.add_argument('--out', required=True, type=Path)
+    args = parser.parse_args()
+    rows = []
+    for path in args.summaries:
+        row = json.loads(path.read_text(encoding='utf-8'))
+        run = json.loads((path.parent/'run.json').read_text(encoding='utf-8'))
+        fixture = json.loads((path.parent/'fixture.json').read_text(encoding='utf-8'))
+        row['benchmark_identity'] = {'source': run['source_sha256'], 'inputs': fixture['input_hashes'], 'case': fixture['case']}
+        rows.append(row)
+    report = compare(rows)
+    args.out.parent.mkdir(parents=True, exist_ok=True)
+    args.out.write_text(json.dumps(report, indent=2)+'\n', encoding='utf-8')
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == '__main__': main()