#220: preserve qualified edit/recovery evaluation on current shared repair

evals/edit-recovery/FINDINGS.md

@@ -0,0 +1,59 @@
+# Controlled trial findings
+
+These findings are for a substitute fixture, not full #220 acceptance. Source is
+`44530155261e62b49065c8c240bc9a8715acf89d`. The observed pre-M2 snapshot and full
+historical prompt are absent. This distinction is described in README.md.
+
+One actual headless Astra trial was run per cell, using the current-source Rust
+binary, explicit approved config home, approved profile resolver, 40-call cap,
+five-minute turn budget, and no shell. Exact commands, binary SHA256, elapsed time,
+raw outputs, process RSS, validated session trace, preservation hashes and external
+Cargo outcomes are retained under `evalwork/results/branch4-wave2/issue220/`.
+The current arm's directory is `control-2`; `control` is an interrupted preparation,
+not a model trial. The first smoke attempt failed authentication; a later attempt
+succeeded without any credential-store modification.
+
+| Mission / guidance | Calls | Before first edit attempt | Successful edits | Peak RSS bytes |
+|---|---:|---:|---:|---:|
+| ordinary / current | 37 | 14 | 0 | 24,608,768 |
+| ordinary / explicit | 39 | 16 | 0 | 24,559,616 |
+| recovery / current | 16 | none | 0 | 24,018,944 |
+| recovery / explicit | 9 | none | 0 | 23,101,440 |
+
+Both ordinary runs selected `replace` but failed its SHA256 precondition. The
+current run first supplied an empty hash, then an abbreviated hash copied from the
+refusal, neither of which is a complete independently computed SHA256. Both left
+all 34 original tests intact and made no target change. Independent Cargo compile
+and phase2 runs passed in both ordinary workspaces (34 tests each); the exact-target
+grader still rejected both. Both recovery runs refused
+to rebuild from opaque CTXDIGEST reads, leaving the supplied damaged fixture with
+two test declarations. Both recovery compile and phase2 commands exited 101 with
+`unexpected closing delimiter: ')'` at `tests/phase2.rs:1:5`. All four grades rejected
+acceptance. An exit-zero wrap-up is not task success.
+
+The explicit instruction did not improve completion in these four runs. Fewer
+recovery calls cannot be called a performance win because neither recovered the
+file. The ordinary explicit arm used more calls. There is no statistical estimate,
+provider-general competence claim, Terminal-Bench score, or token saving claim.
+RSS is sampled by the actual Rust process; request-size bytes are estimates, not
+tokens. Token usage is not present in the terminal envelope and is reported unknown.
+
+No whole-file block write occurred in these trials, so they do not show that an
+instruction prevented the historical behavior. They also do not isolate instruction
+placement in the system prefix. The shell-disabled recovery roster is narrower
+than the original workload; the observed opaque-read barrier is a material
+confounder. No guidance or tool-description production change is justified here.
+No change to #79's prefix, #190's shell schema, #134's unknown-tool handling, or
+`write_file` semantics is made.
+
+## Required continuation
+
+The driver must provide the authorized preserved pre-M2 snapshot/diff and complete
+launch prompt/tool-visible trace, with the hashes recorded in the issue, within
+this worktree. Do not import a dirty sibling file. Replay the exact output-byte
+exhaustion mission rather than substituting this tool-count edit. Resolve or
+explicitly study the opaque-read limitation with the original shell-enabled roster;
+do not add a hidden fallback reader or automatic restoration. Repeat the matched
+ordinary and damaged-file guidance comparison with authenticated traces and exact
+integrity/compile/test grading. Until then the investigation remains BLOCKED, not a
+completed negative result.

evals/edit-recovery/HISTORICAL-FINDINGS.md

@@ -0,0 +1,126 @@
+# Historical shell-enabled findings (#220)
+
+## Population and qualifications
+
+Four fresh Astra trials, one per mode/guidance cell, used the same disclosed unmerged
+#66 owner candidate: source `2566994ab3ebab82edd2f52db55afba2a595fc8b`, executable
+SHA256 `7d9b7a1ca01c5c6145357ce9aa354558842a273a8b1eec12b5273cb074357320`.
+Profile SHA256 was `891795efb06ec8fab463b17e7e5a4a5451de5a93e53cf0baaf2e71666de59102`,
+provider `codex`, model `gpt-6-astra`, reasoning high, maxOutputTokens 40000.
+All used the supplied config home, shell enabled, unlimited calls, no turn deadline,
+32 KiB shell, 16 MiB tool and 16 MiB live aggregate output limits. No settings file
+or settings environment override was present. `--print-config` confirms resolution.
+
+These are **not** exact historical DeepSeek runs, stock-main live acceptance, or a
+#66 source merge. Historical old-binary prefix/request bytes remain unavailable.
+Recovered calls are diagnostic, not authenticated store/vault replay. Current
+production validation of new session records does not upgrade old evidence.
+
+The full historical dirty fixture matched all 17,047 launcher file hashes in every
+prepared case. The baseline reproduced the exact invalid-round-index panic after
+compilation succeeded: 32 tests passed, the targeted exhaustion test failed, and
+the shared-budget test passed. Actual original inventory: 33 tests and 1,528 lines.
+The damaged starting state retains its exact reported hash and 166 lines.
+
+## Observed worker behavior
+
+| Case | Calls | Calls before first file edit | Target block method | Child seconds | Peak RSS bytes |
+| --- | ---: | ---: | --- | ---: | ---: |
+| ordinary/current | 3 | 2 | shell Python byte splice | 72.09 | 16,924,672 |
+| ordinary/explicit | 7 | 3 before refused replace; 5 before successful edit | registered replace | 204.18 | 17,711,104 |
+| recovery/current | 4 | 0 before restoration; 3 before block edit | shell copy, Python byte splice | 98.07 | 16,973,824 |
+| recovery/explicit | 9 | 0 before restoration; 3 before block edit | shell restoration, registered replace | 172.55 | 17,711,104 |
+
+All returned child exit 0 with actual target edits, rather than the prior no-edit
+wrap-ups. Independent outside-function checks retain all imports, all test names,
+the successful M1 shared-budget test and every other historical tracked file.
+The current arms chose shell block splices, not registered `write_file`; these
+preserved complete source bytes read locally at edit time. The explicit arms
+selected `replace`. No arm reproduced a block-as-whole-file write.
+
+Ordinary/explicit first passed an empty expected SHA256 and was correctly refused.
+It computed the full file hash with shell and retried successfully. Recovery/explicit
+also passed an empty hash for its formatting replacement, then independently hashed
+and retried successfully. Its separate failed shell call looked for nonexistent
+`src/tools/read_file.rs`; the retained result is compacted. These are distinct
+observed tool-call failures, not authorization or transport failures. All four
+candidate calls authenticated without changing Keychain policy. Earlier authorization
+and opaque-read failures remain in the original report.
+
+Recovery/current explicitly copied `recovery/phase2.rs` at call 1, after printing
+its hash, then at call 4 asserted the full authorized hash and byte equality before
+editing. Recovery/explicit asserted that hash before restoration at call 1 and
+reverified outside-function preservation at calls 5 and 9. No observed command
+searched or imported sibling source. Shell is not a sandbox; this is command-level
+observed confinement plus independent file checks, not syscall-level proof.
+
+All four target functions retain sixteen reads that fill the live-byte budget,
+a valid seventeenth `write_file` with an absent-file witness, the exact error and
+failed lifecycle, and a TEST context large enough not to mask the output cap.
+Recovery/explicit additionally executes the exact refused call as a positive
+control after asserting its absence. The independent mutant deliberately executes
+the refused call before returning the same error. Its recorded witness failures
+and the complete per-case compile/test results belong in `completion-impl2.md` and
+the immutable `impl2-*` command reports, not inferred from worker summaries.
+
+## Interpretation
+
+Explicit guidance coincided with registered `replace` selection in these observations,
+without establishing a completion or preservation improvement. Both current arms already
+produced source-preserving edits; explicit arms used more calls and time, including
+SHA precondition retries. This is one sample per cell, with fixed order and no
+randomization. Do not infer model-general rates, statistical significance, caching
+benefits, or a performance regression from these timings. Token usage is unavailable;
+request byte estimates and live output counters are retained, not called tokens.
+No Terminal-Bench score or performance win is claimed.
+
+No production guidance/tool-description change is justified by this comparison.
+In particular, editor choice alone does not justify altering the shared system prefix.
+No whole-file semantics, automatic restoration, fallback, alias, compatibility
+reader or migration is added. #134's current unknown-tool handling is not relabeled
+as missing because of the older `run_scheduler` observation.
+
+## Evidence index
+
+Under `evalwork/results/branch4-wave2/issue220/`:
+
+- `recovered-history/`: authorized reconstruction, launcher mission/diff/manifest,
+  damaged payload, and all 140 diagnostic calls; immutable original evidence.
+- `impl2-identities/`: rebase identity, source/host-prefix hashes, candidate help,
+  public profile settings, resolved settings, historical call/payload accounting.
+- `impl2-{ordinary,recovery}-{current,explicit}/`: exact prompts and launch vectors,
+  raw stdout/stderr/RSS, terminal exits, complete file manifests, final source,
+  external Cargo logs, current production-validated persisted traces and call roster.
+- `impl2-baseline/`, `impl2-damaged-baseline/`: independent negative controls.
+- `impl2-mutant-*/`: disposable deliberately defective runtime, mutation hashes,
+  commands and execution-witness test results.
+- `completion-impl2.md`: final acceptance matrix, source/gates and any limitations.
+- `impl2-dogfood.md`: continuation tool observations. `dogfood.md` retains old runs.
+
+`HISTORICAL.md` documents reproduction and grader limitations. The original
+`completion.md`, `README.md` and `FINDINGS.md` remain the earlier blocked report,
+not final acceptance. The driver owns independent final acceptance and review.
+
+## Final integration qualification (impl3)
+
+Ordinary/explicit had five preceding calls: FOUR successful and one failed replace,
+then its first successful edit at call 6. The earlier completion table's phrase
+"5 successful" is incorrect; its raw trials and reports remain unchanged.
+Editor selection is an observation, not a causal guidance effect from one
+fixed-order trial per cell. No guidance-benefit claim is established.
+
+The independent acceptance220 audit supports these four candidate/Astra cells,
+not unqualified final acceptance. The retained full-test status 0 includes the
+malformed shared parity Python fixture diagnosed in #252. External scratch alone
+would not restore its intended adversarial coverage. Known-PID absence does not
+prove complete descendant cleanup; no unsupported historical issue220 orphan event
+is inferred. Separate #252 ownership was requested in comment5577761171. Mandatory
+full-gate coverage, identity-safe cleanup evidence and later driver review remain
+outstanding. New final-source commands and identities are in `completion-impl3.md`
+and `impl3-*` evidence; they do not rerun or replace the measured trials.
+
+Retained production-checked trace linkage is not fresh authenticated replay or a
+wire capture. Old outbound wire/authentication is missing, and the shared historical
+target was reused without a separately retained executable hash per cell. Candidate,
+final-source CLI and older dirty d666 authoring binary identities remain distinct.
+The earlier four failed no-edit trials remain failures, not pooled with these cells.

evals/edit-recovery/HISTORICAL.md

@@ -0,0 +1,107 @@
+# Recovered historical workload evaluator (#220 continuation)
+
+This continuation supplements, rather than rewrites, `README.md` and `FINDINGS.md`.
+Those files describe the earlier blocked, shell-disabled substitute probes. Their
+reports and all failed runs remain retained. See `HISTORICAL-FINDINGS.md` for measured
+results of this generation and the evidence-local `completion-impl2.md` for gates.
+
+## Identity and acquisition
+
+`historical.py prepare` materializes Git `c6645998af3caba5d36b7bd04ae2093babca6a1d`
+into a **new disposable evidence-local directory**, then applies the 11 authorized
+reconstructed file generations. It verifies every one of the 17,047 launcher tracked
+file hashes, including unchanged files. No owner worktree is opened or modified.
+The recovered phase2 source is 57,400 bytes, 1,528 lines, SHA256
+`c84874023f22db98b647387e89e1adf2eebcbdc910a8049ff65d98fe7b75c156`.
+It has **33 test declarations**, not the earlier substitute's 34. Imports and the
+successful shared-budget test are part of the exact preservation boundary.
+
+Recovery starts from the actual 8,348-byte, 166-line damaged file, SHA256
+`2d69b6679c6e8baf921249939f6c743900931d51ed463c34a47132849b727ab8`.
+The authorized dirty snapshot is supplied at `recovery/phase2.rs`. Restoration is
+an explicit worker action, not a tool feature or automatic harness repair. The
+harness never repairs the live trial after launch. Clean Git is not substituted
+for dirty historical input.
+
+The original prompt is retained byte-for-byte in recovered history. Trial prompts
+change its owner-workspace path to the disposable path, append a common isolation
+and exact outside-function preservation instruction, and add the recovery task in
+recovery cells. Explicit arms alone append `GUIDANCE`. This is a user-instruction
+comparison, not a system-prefix placement experiment. `prompt.txt` and manifest
+hashes expose every adaptation.
+
+## Repeatable commands
+
+All destinations must be new `impl2-*` directories under the issue evidence root.
+Supply the recovered-history artifacts listed in `RECOVERY.md`; none are committed.
+
+```sh
+E=evalwork/results/branch4-wave2/issue220
+python3 evals/edit-recovery/historical.py prepare "$E/impl2-new-ordinary-current" \
+  --mode ordinary --guidance current
+LLXPRT_CONFIG_HOME=/path/to/approved/config-home \
+CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target/issue220-historical" \
+RUSTUP_TOOLCHAIN=1.88.0 \
+python3 evals/edit-recovery/historical.py run "$E/impl2-new-ordinary-current" \
+  --binary /path/to/disclosed/issue66-candidate --profile /path/to/astra-headless.json
+python3 evals/edit-recovery/historical.py grade "$E/impl2-new-ordinary-current"
+CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target" \
+cargo +1.88.0 build --offline --locked --example issue220_trace --example issue220_prefix
+LLXPRT_CONFIG_HOME=/path/to/approved/config-home \
+python3 evals/edit-recovery/export_historical.py "$E/impl2-new-ordinary-current"
+python3 evals/edit-recovery/mutation_historical.py "$E/impl2-new-ordinary-current" \
+  "$E/impl2-new-mutant-ordinary-current"
+python3 -m unittest discover -s evals/edit-recovery -p 'test_*.py'
+```
+
+Repeat all four mode/guidance combinations with matched binary/profile/settings.
+The runner requires the disclosed candidate hash and preserves explicit config home.
+It enables shell, unlimited tool calls, no turn-time override and records 32 KiB
+shell / 16 MiB tool / 16 MiB live aggregate output limits. Original launch enabled
+shell and unlimited calls and did not override output limits. The candidate's #66
+live-output projection differs materially from the earlier stock tool, even where
+host tool descriptions are identical. Do not mix those generations in a comparison.
+No credential contents, Keychain changes or source-encoding workarounds are needed.
+
+## Grading, including false-success controls
+
+`grade` checks exact original prefix and suffix, imports by byte preservation, all
+33 names in order, target modification, removal of the invalid partial-round access,
+all other launcher files and additions. It retains independent Cargo compilation
+and full phase2 results. Its `structural_and_test_pass` is **not semantic acceptance**:
+a vacuous passing test is still unacceptable. The fixture has deliberately flexible
+scenario requirements rather than one textual answer oracle.
+
+Inspect the final function for sixteen budget-filling reads followed by a valid
+side-effecting call, exact output-cap error, failed lifecycle, appropriate TEST
+context, no invented tool-budget notice and a direct execution witness. Then use
+`mutation_historical.py` to deliberately execute the refused call before returning
+the same error in a disposable runtime. The grader requires the witness panic in a
+running test, not compilation failure or merely exit 101. This rejects deletion of
+the old assertion without positive evidence. The issue's production tree is never
+mutated by this control. Retain an unchanged baseline and damaged baseline too.
+
+`export_historical.py` uses production `SessionStore::load_at` and `snapshot`, not an
+old-state reader. It checks call accounting and preserves every available argument
+and persisted result, including failures. Inspect shell calls as well as registered
+editors to identify the first actual edit. Explicit recovery copy/read operations,
+independent source hashes and outside-block checks establish observed provenance;
+matching final bytes alone do not. Shell commands have general user privileges.
+Observed command confinement is not syscall auditing or OS sandbox isolation.
+Compacted persisted results are not exact captures of full live provider requests.
+
+`examples/issue220_prefix.rs` exports current host system/ToolSpec values, clearly
+labeled **not a provider wire capture**. Candidate/current source hash comparisons
+permit specific host-description attribution, not an exact historical prefix claim.
+No semantic store/vault authentication was established for the recovered old trace.
+All 140 historical calls and the damaging payload remain unauthenticated diagnostic
+evidence, regardless of hash consistency. No missing historical request is invented.
+
+## Scope exclusions retained
+
+No production guidance change without measured benefit. No `write_file` semantics
+change, automatic restoration, fallback, alias, compatibility reader, credential
+handling, owner #66 source merge, sibling source import, or changes to #79/#190.
+The tightened compatibility gate and apiMode-only settings on the required main
+remain intact. A candidate-model result is not stock-main live acceptance, and an
+exit-zero summary without a target edit is always failure.