diff --git a/evals/edit-recovery/FINDINGS.md b/evals/edit-recovery/FINDINGS.md new file mode 100644 index 00000000..b85df6b5 --- /dev/null +++ b/evals/edit-recovery/FINDINGS.md @@ -0,0 +1,59 @@ +# Controlled trial findings + +These findings are for a substitute fixture, not full #220 acceptance. Source is +`44530155261e62b49065c8c240bc9a8715acf89d`. The observed pre-M2 snapshot and full +historical prompt are absent. This distinction is described in README.md. + +One actual headless Astra trial was run per cell, using the current-source Rust +binary, explicit approved config home, approved profile resolver, 40-call cap, +five-minute turn budget, and no shell. Exact commands, binary SHA256, elapsed time, +raw outputs, process RSS, validated session trace, preservation hashes and external +Cargo outcomes are retained under `evalwork/results/branch4-wave2/issue220/`. +The current arm's directory is `control-2`; `control` is an interrupted preparation, +not a model trial. The first smoke attempt failed authentication; a later attempt +succeeded without any credential-store modification. + +| Mission / guidance | Calls | Before first edit attempt | Successful edits | Peak RSS bytes | +|---|---:|---:|---:|---:| +| ordinary / current | 37 | 14 | 0 | 24,608,768 | +| ordinary / explicit | 39 | 16 | 0 | 24,559,616 | +| recovery / current | 16 | none | 0 | 24,018,944 | +| recovery / explicit | 9 | none | 0 | 23,101,440 | + +Both ordinary runs selected `replace` but failed its SHA256 precondition. The +current run first supplied an empty hash, then an abbreviated hash copied from the +refusal, neither of which is a complete independently computed SHA256. Both left +all 34 original tests intact and made no target change. Independent Cargo compile +and phase2 runs passed in both ordinary workspaces (34 tests each); the exact-target +grader still rejected both. Both recovery runs refused +to rebuild from opaque CTXDIGEST reads, leaving the supplied damaged fixture with +two test declarations. Both recovery compile and phase2 commands exited 101 with +`unexpected closing delimiter: ')'` at `tests/phase2.rs:1:5`. All four grades rejected +acceptance. An exit-zero wrap-up is not task success. + +The explicit instruction did not improve completion in these four runs. Fewer +recovery calls cannot be called a performance win because neither recovered the +file. The ordinary explicit arm used more calls. There is no statistical estimate, +provider-general competence claim, Terminal-Bench score, or token saving claim. +RSS is sampled by the actual Rust process; request-size bytes are estimates, not +tokens. Token usage is not present in the terminal envelope and is reported unknown. + +No whole-file block write occurred in these trials, so they do not show that an +instruction prevented the historical behavior. They also do not isolate instruction +placement in the system prefix. The shell-disabled recovery roster is narrower +than the original workload; the observed opaque-read barrier is a material +confounder. No guidance or tool-description production change is justified here. +No change to #79's prefix, #190's shell schema, #134's unknown-tool handling, or +`write_file` semantics is made. + +## Required continuation + +The driver must provide the authorized preserved pre-M2 snapshot/diff and complete +launch prompt/tool-visible trace, with the hashes recorded in the issue, within +this worktree. Do not import a dirty sibling file. Replay the exact output-byte +exhaustion mission rather than substituting this tool-count edit. Resolve or +explicitly study the opaque-read limitation with the original shell-enabled roster; +do not add a hidden fallback reader or automatic restoration. Repeat the matched +ordinary and damaged-file guidance comparison with authenticated traces and exact +integrity/compile/test grading. Until then the investigation remains BLOCKED, not a +completed negative result. diff --git a/evals/edit-recovery/HISTORICAL-FINDINGS.md b/evals/edit-recovery/HISTORICAL-FINDINGS.md new file mode 100644 index 00000000..b7b6056f --- /dev/null +++ b/evals/edit-recovery/HISTORICAL-FINDINGS.md @@ -0,0 +1,126 @@ +# Historical shell-enabled findings (#220) + +## Population and qualifications + +Four fresh Astra trials, one per mode/guidance cell, used the same disclosed unmerged +#66 owner candidate: source `2566994ab3ebab82edd2f52db55afba2a595fc8b`, executable +SHA256 `7d9b7a1ca01c5c6145357ce9aa354558842a273a8b1eec12b5273cb074357320`. +Profile SHA256 was `891795efb06ec8fab463b17e7e5a4a5451de5a93e53cf0baaf2e71666de59102`, +provider `codex`, model `gpt-6-astra`, reasoning high, maxOutputTokens 40000. +All used the supplied config home, shell enabled, unlimited calls, no turn deadline, +32 KiB shell, 16 MiB tool and 16 MiB live aggregate output limits. No settings file +or settings environment override was present. `--print-config` confirms resolution. + +These are **not** exact historical DeepSeek runs, stock-main live acceptance, or a +#66 source merge. Historical old-binary prefix/request bytes remain unavailable. +Recovered calls are diagnostic, not authenticated store/vault replay. Current +production validation of new session records does not upgrade old evidence. + +The full historical dirty fixture matched all 17,047 launcher file hashes in every +prepared case. The baseline reproduced the exact invalid-round-index panic after +compilation succeeded: 32 tests passed, the targeted exhaustion test failed, and +the shared-budget test passed. Actual original inventory: 33 tests and 1,528 lines. +The damaged starting state retains its exact reported hash and 166 lines. + +## Observed worker behavior + +| Case | Calls | Calls before first file edit | Target block method | Child seconds | Peak RSS bytes | +| --- | ---: | ---: | --- | ---: | ---: | +| ordinary/current | 3 | 2 | shell Python byte splice | 72.09 | 16,924,672 | +| ordinary/explicit | 7 | 3 before refused replace; 5 before successful edit | registered replace | 204.18 | 17,711,104 | +| recovery/current | 4 | 0 before restoration; 3 before block edit | shell copy, Python byte splice | 98.07 | 16,973,824 | +| recovery/explicit | 9 | 0 before restoration; 3 before block edit | shell restoration, registered replace | 172.55 | 17,711,104 | + +All returned child exit 0 with actual target edits, rather than the prior no-edit +wrap-ups. Independent outside-function checks retain all imports, all test names, +the successful M1 shared-budget test and every other historical tracked file. +The current arms chose shell block splices, not registered `write_file`; these +preserved complete source bytes read locally at edit time. The explicit arms +selected `replace`. No arm reproduced a block-as-whole-file write. + +Ordinary/explicit first passed an empty expected SHA256 and was correctly refused. +It computed the full file hash with shell and retried successfully. Recovery/explicit +also passed an empty hash for its formatting replacement, then independently hashed +and retried successfully. Its separate failed shell call looked for nonexistent +`src/tools/read_file.rs`; the retained result is compacted. These are distinct +observed tool-call failures, not authorization or transport failures. All four +candidate calls authenticated without changing Keychain policy. Earlier authorization +and opaque-read failures remain in the original report. + +Recovery/current explicitly copied `recovery/phase2.rs` at call 1, after printing +its hash, then at call 4 asserted the full authorized hash and byte equality before +editing. Recovery/explicit asserted that hash before restoration at call 1 and +reverified outside-function preservation at calls 5 and 9. No observed command +searched or imported sibling source. Shell is not a sandbox; this is command-level +observed confinement plus independent file checks, not syscall-level proof. + +All four target functions retain sixteen reads that fill the live-byte budget, +a valid seventeenth `write_file` with an absent-file witness, the exact error and +failed lifecycle, and a TEST context large enough not to mask the output cap. +Recovery/explicit additionally executes the exact refused call as a positive +control after asserting its absence. The independent mutant deliberately executes +the refused call before returning the same error. Its recorded witness failures +and the complete per-case compile/test results belong in `completion-impl2.md` and +the immutable `impl2-*` command reports, not inferred from worker summaries. + +## Interpretation + +Explicit guidance coincided with registered `replace` selection in these observations, +without establishing a completion or preservation improvement. Both current arms already +produced source-preserving edits; explicit arms used more calls and time, including +SHA precondition retries. This is one sample per cell, with fixed order and no +randomization. Do not infer model-general rates, statistical significance, caching +benefits, or a performance regression from these timings. Token usage is unavailable; +request byte estimates and live output counters are retained, not called tokens. +No Terminal-Bench score or performance win is claimed. + +No production guidance/tool-description change is justified by this comparison. +In particular, editor choice alone does not justify altering the shared system prefix. +No whole-file semantics, automatic restoration, fallback, alias, compatibility +reader or migration is added. #134's current unknown-tool handling is not relabeled +as missing because of the older `run_scheduler` observation. + +## Evidence index + +Under `evalwork/results/branch4-wave2/issue220/`: + +- `recovered-history/`: authorized reconstruction, launcher mission/diff/manifest, + damaged payload, and all 140 diagnostic calls; immutable original evidence. +- `impl2-identities/`: rebase identity, source/host-prefix hashes, candidate help, + public profile settings, resolved settings, historical call/payload accounting. +- `impl2-{ordinary,recovery}-{current,explicit}/`: exact prompts and launch vectors, + raw stdout/stderr/RSS, terminal exits, complete file manifests, final source, + external Cargo logs, current production-validated persisted traces and call roster. +- `impl2-baseline/`, `impl2-damaged-baseline/`: independent negative controls. +- `impl2-mutant-*/`: disposable deliberately defective runtime, mutation hashes, + commands and execution-witness test results. +- `completion-impl2.md`: final acceptance matrix, source/gates and any limitations. +- `impl2-dogfood.md`: continuation tool observations. `dogfood.md` retains old runs. + +`HISTORICAL.md` documents reproduction and grader limitations. The original +`completion.md`, `README.md` and `FINDINGS.md` remain the earlier blocked report, +not final acceptance. The driver owns independent final acceptance and review. + +## Final integration qualification (impl3) + +Ordinary/explicit had five preceding calls: FOUR successful and one failed replace, +then its first successful edit at call 6. The earlier completion table's phrase +"5 successful" is incorrect; its raw trials and reports remain unchanged. +Editor selection is an observation, not a causal guidance effect from one +fixed-order trial per cell. No guidance-benefit claim is established. + +The independent acceptance220 audit supports these four candidate/Astra cells, +not unqualified final acceptance. The retained full-test status 0 includes the +malformed shared parity Python fixture diagnosed in #252. External scratch alone +would not restore its intended adversarial coverage. Known-PID absence does not +prove complete descendant cleanup; no unsupported historical issue220 orphan event +is inferred. Separate #252 ownership was requested in comment5577761171. Mandatory +full-gate coverage, identity-safe cleanup evidence and later driver review remain +outstanding. New final-source commands and identities are in `completion-impl3.md` +and `impl3-*` evidence; they do not rerun or replace the measured trials. + +Retained production-checked trace linkage is not fresh authenticated replay or a +wire capture. Old outbound wire/authentication is missing, and the shared historical +target was reused without a separately retained executable hash per cell. Candidate, +final-source CLI and older dirty d666 authoring binary identities remain distinct. +The earlier four failed no-edit trials remain failures, not pooled with these cells. diff --git a/evals/edit-recovery/HISTORICAL.md b/evals/edit-recovery/HISTORICAL.md new file mode 100644 index 00000000..6bf13b72 --- /dev/null +++ b/evals/edit-recovery/HISTORICAL.md @@ -0,0 +1,107 @@ +# Recovered historical workload evaluator (#220 continuation) + +This continuation supplements, rather than rewrites, `README.md` and `FINDINGS.md`. +Those files describe the earlier blocked, shell-disabled substitute probes. Their +reports and all failed runs remain retained. See `HISTORICAL-FINDINGS.md` for measured +results of this generation and the evidence-local `completion-impl2.md` for gates. + +## Identity and acquisition + +`historical.py prepare` materializes Git `c6645998af3caba5d36b7bd04ae2093babca6a1d` +into a **new disposable evidence-local directory**, then applies the 11 authorized +reconstructed file generations. It verifies every one of the 17,047 launcher tracked +file hashes, including unchanged files. No owner worktree is opened or modified. +The recovered phase2 source is 57,400 bytes, 1,528 lines, SHA256 +`c84874023f22db98b647387e89e1adf2eebcbdc910a8049ff65d98fe7b75c156`. +It has **33 test declarations**, not the earlier substitute's 34. Imports and the +successful shared-budget test are part of the exact preservation boundary. + +Recovery starts from the actual 8,348-byte, 166-line damaged file, SHA256 +`2d69b6679c6e8baf921249939f6c743900931d51ed463c34a47132849b727ab8`. +The authorized dirty snapshot is supplied at `recovery/phase2.rs`. Restoration is +an explicit worker action, not a tool feature or automatic harness repair. The +harness never repairs the live trial after launch. Clean Git is not substituted +for dirty historical input. + +The original prompt is retained byte-for-byte in recovered history. Trial prompts +change its owner-workspace path to the disposable path, append a common isolation +and exact outside-function preservation instruction, and add the recovery task in +recovery cells. Explicit arms alone append `GUIDANCE`. This is a user-instruction +comparison, not a system-prefix placement experiment. `prompt.txt` and manifest +hashes expose every adaptation. + +## Repeatable commands + +All destinations must be new `impl2-*` directories under the issue evidence root. +Supply the recovered-history artifacts listed in `RECOVERY.md`; none are committed. + +```sh +E=evalwork/results/branch4-wave2/issue220 +python3 evals/edit-recovery/historical.py prepare "$E/impl2-new-ordinary-current" \ + --mode ordinary --guidance current +LLXPRT_CONFIG_HOME=/path/to/approved/config-home \ +CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target/issue220-historical" \ +RUSTUP_TOOLCHAIN=1.88.0 \ +python3 evals/edit-recovery/historical.py run "$E/impl2-new-ordinary-current" \ + --binary /path/to/disclosed/issue66-candidate --profile /path/to/astra-headless.json +python3 evals/edit-recovery/historical.py grade "$E/impl2-new-ordinary-current" +CARGO_BUILD_JOBS=2 CARGO_TARGET_DIR="$PWD/target" \ +cargo +1.88.0 build --offline --locked --example issue220_trace --example issue220_prefix +LLXPRT_CONFIG_HOME=/path/to/approved/config-home \ +python3 evals/edit-recovery/export_historical.py "$E/impl2-new-ordinary-current" +python3 evals/edit-recovery/mutation_historical.py "$E/impl2-new-ordinary-current" \ + "$E/impl2-new-mutant-ordinary-current" +python3 -m unittest discover -s evals/edit-recovery -p 'test_*.py' +``` + +Repeat all four mode/guidance combinations with matched binary/profile/settings. +The runner requires the disclosed candidate hash and preserves explicit config home. +It enables shell, unlimited tool calls, no turn-time override and records 32 KiB +shell / 16 MiB tool / 16 MiB live aggregate output limits. Original launch enabled +shell and unlimited calls and did not override output limits. The candidate's #66 +live-output projection differs materially from the earlier stock tool, even where +host tool descriptions are identical. Do not mix those generations in a comparison. +No credential contents, Keychain changes or source-encoding workarounds are needed. + +## Grading, including false-success controls + +`grade` checks exact original prefix and suffix, imports by byte preservation, all +33 names in order, target modification, removal of the invalid partial-round access, +all other launcher files and additions. It retains independent Cargo compilation +and full phase2 results. Its `structural_and_test_pass` is **not semantic acceptance**: +a vacuous passing test is still unacceptable. The fixture has deliberately flexible +scenario requirements rather than one textual answer oracle. + +Inspect the final function for sixteen budget-filling reads followed by a valid +side-effecting call, exact output-cap error, failed lifecycle, appropriate TEST +context, no invented tool-budget notice and a direct execution witness. Then use +`mutation_historical.py` to deliberately execute the refused call before returning +the same error in a disposable runtime. The grader requires the witness panic in a +running test, not compilation failure or merely exit 101. This rejects deletion of +the old assertion without positive evidence. The issue's production tree is never +mutated by this control. Retain an unchanged baseline and damaged baseline too. + +`export_historical.py` uses production `SessionStore::load_at` and `snapshot`, not an +old-state reader. It checks call accounting and preserves every available argument +and persisted result, including failures. Inspect shell calls as well as registered +editors to identify the first actual edit. Explicit recovery copy/read operations, +independent source hashes and outside-block checks establish observed provenance; +matching final bytes alone do not. Shell commands have general user privileges. +Observed command confinement is not syscall auditing or OS sandbox isolation. +Compacted persisted results are not exact captures of full live provider requests. + +`examples/issue220_prefix.rs` exports current host system/ToolSpec values, clearly +labeled **not a provider wire capture**. Candidate/current source hash comparisons +permit specific host-description attribution, not an exact historical prefix claim. +No semantic store/vault authentication was established for the recovered old trace. +All 140 historical calls and the damaging payload remain unauthenticated diagnostic +evidence, regardless of hash consistency. No missing historical request is invented. + +## Scope exclusions retained + +No production guidance change without measured benefit. No `write_file` semantics +change, automatic restoration, fallback, alias, compatibility reader, credential +handling, owner #66 source merge, sibling source import, or changes to #79/#190. +The tightened compatibility gate and apiMode-only settings on the required main +remain intact. A candidate-model result is not stock-main live acceptance, and an +exit-zero summary without a target edit is always failure. diff --git a/evals/edit-recovery/README.md b/evals/edit-recovery/README.md new file mode 100644 index 00000000..850c56f2 --- /dev/null +++ b/evals/edit-recovery/README.md @@ -0,0 +1,105 @@ +# Issue 220: edit and recovery probe (blocked investigation) + +This is a partial evaluator, not a delivered guidance fix or a model competence result. +`write_file` remains a whole-file writer. No aliases, automatic restoration, compatibility +behavior, system-prefix changes, or shell-schema changes are introduced. + +## Retained workload and limits + +The supplied observation and diagnostic activity identify the 8,348-byte, 166-line +failure. The activity's `write_file` content exactly matches `phase2.rs.observed` +(SHA256 `2d69b6679c6e8baf921249939f6c743900931d51ed463c34a47132849b727ab8`). +There are four shell calls before that write in the *retained activity window*, not +necessarily four calls since the start of the mission. Recovery activity mentions +sibling worktrees. This is diagnostic, unauthenticated activity, not durable acceptance. +The issue comment records subsequent complete restoration; this probe does not redo it. + +The historical commit `c6645998af3caba5d36b7bd04ae2093babca6a1d` is locally available. +Its `tests/phase2.rs` is 1,389 lines, SHA256 +`883426bf201da170f350cef8affd5bc101ab2f2a7ba0e3be16cd0fcbad4cca85`. +It is **not** the pre-M2 1,528-line source with SHA256 +`c84874023f22db98b647387e89e1adf2eebcbdc910a8049ff65d98fe7b75c156`. +The supplied damaged tracked diff removes the historical file in one hunk. It does +not contain the lost pre-M2 additions. The launcher `before.files.json`, preserved +pre-M2 diff, full launch prompt and authenticated tool trace are not supplied here. +Do not recover those from another worker's dirty tree. + +The older authoring binary SHA256 `37e7261686cbc4a2a51d9aa1fc7b61ca8f24e3f97f6c26ce7e1a1efd1346a627` +is an older dirty d666 build. Historical source alone cannot establish its exact +model-visible instructions. Current `src/agent/config.rs::coding_system_prompt` +contains workspace, shell, reasoning and tool-budget notes, without edit guidance. +`src/tools.rs::tool_specs` describes write as replacing existing content and replace +as exact matching. These source identities are retained separately in investigation +artifacts. Current unknown-tool handling is in `src/agent/helpers.rs` and +`src/agent/tool_validation_tests.rs`; the old `run_scheduler` observation is not a +missing current alias and no alias is proposed. + +## Controlled probe + +`evaluate.py` materializes the starting source into a new evidence-local workspace. +It targets the tool-count exhaustion regression, changing the declared budget from +16 to 17 and attempted calls from 17 to 18, including comments and side-effect +witnesses. It preserves the shared-output-budget regression. This controlled mission +is **not** the original exact output-byte exhaustion task. The original mission +cannot be reconstructed faithfully without its pre-M2 inputs. + +The recovery mode injects the actual damaged bytes, but supplies an authorized local +snapshot of the *starting source*, not the missing pre-M2 file. Current and explicit +arms differ only by appended user guidance. This does not measure system-prefix +placement or provider caching. Shell is disabled in live probes, and external compile +and test grading uses the real workspace. No mock backend substitutes for live Astra. + +The grader rejects unchanged targets, a block written as the whole file, removed +imports/tests, unrelated changes and added files. It requires exact file equality, +not just syntax or test-count equality. It runs actual Cargo compile and phase2 +suite commands when requested. Call selection and calls before the first edit come from +`examples/issue220_trace.rs`, using the production `SessionStore::load_at` and +`snapshot` validator, never a fallback reader. Build the example with the same +MSRV/offline/locked settings before grading. Grading needs the explicit config home. +Calls must account for the envelope's executed count. With shell disabled, absolute +or parent-component paths and any unrecognized tools fail the isolation check. +Recovery provenance conservatively requires a successful read containing the whole +authorized snapshot; digest-only or chunked recovery is not certified. The supplied +snapshot's hash is also checked after editing. The grader does not infer provenance +from matching final bytes. This conservative probe is not an exact M2 reconstruction. + +## Commands + +From the issue worktree, using a new destination for every trial: + +```sh +python3 -m unittest discover -s evals/edit-recovery -p 'test_*.py' +python3 evals/edit-recovery/evaluate.py prepare \ + evalwork/results/branch4-wave2/issue220/trial-ordinary-current \ + --mode ordinary --guidance current +# Recovery additionally requires --damage pointing to original-evidence/phase2.rs.observed. +# Repeat with --guidance explicit and both modes; do not reuse a changed workspace. +LLXPRT_CONFIG_HOME=/path/to/approved/config-home \ +python3 evals/edit-recovery/evaluate.py run \ + evalwork/results/branch4-wave2/issue220/trial-ordinary-current \ + --binary target/debug/llxprt-code-rs --profile /path/to/approved/astra-headless.json +python3 evals/edit-recovery/evaluate.py grade \ + evalwork/results/branch4-wave2/issue220/trial-ordinary-current --compile-tests +``` + +`terminal.json` records the actual child exit, command, elapsed time and binary hash. +Raw envelope, stderr and RSS are retained. No token/request numbers are inferred from +an authentication failure. Config home must be explicitly set; credentials are never +read or copied by this evaluator. Profile paths use the actual runtime resolver. + +## Blocker observed + +The current-source Astra probe exited 3 before any model call with `model-config`: + +> The macOS keychain prompt for service=llxprt-code-oauth, account=codex:default went unanswered for 10s in a headless run. Run once from an interactive Terminal and click Always Allow, or pre-grant with security set-generic-password-partition-list. + +A subsequent probe authenticated without any credential-store change, so this was +not a persistent blocker. Current-source ordinary/current reached 37 tool calls and +returned `status: ok`, but reported no edit after SHA256-precondition refusals. +Both recovery arms returned without edits because reads were reduced to opaque +CTXDIGEST handles and the shell-disabled roster lacked a copy/hash operation. These +are real task failures, not model-free smoke results. See evidence-local terminal, +validated trace, grade and telemetry reports. No production guidance change is +justified by these bounded trials. The exact original snapshot/prompt is still missing; +one trial per arm and a shell-disabled substitute do not complete the original +investigation or establish a model-general negative finding. diff --git a/evals/edit-recovery/evaluate.py b/evals/edit-recovery/evaluate.py new file mode 100644 index 00000000..a7ea2b4f --- /dev/null +++ b/evals/edit-recovery/evaluate.py @@ -0,0 +1,249 @@ +#!/usr/bin/env python3 +"""Integrity-graded #220 edit probe. Standard library only; no credential handling.""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import subprocess +import time + +ROOT = Path(__file__).resolve().parents[2] +START = "44530155261e62b49065c8c240bc9a8715acf89d" +TARGET = "tests/phase2.rs" +FUNCTION = "budget_exhaustion_refuses_excess_and_forces_a_summary" +GUIDANCE = """For an existing-file block edit, read the current content and use replace +with an exact unique old string. write_file replaces the entire file and is for new +files or intentional complete rewrites, never a replacement block. Never reconstruct +a file from a truncated read or CTXDIGEST. For recovery, use only the supplied local +recovery/phase2.rs and verify its SHA256 before restoring. Do not search sibling +worktrees or import their source. Verify preservation, not merely tool success.""" + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def expected_edit(source): + """A controlled, exact oracle, not a claim to reproduce the lost M2 snapshot.""" + start = source.index("fn " + FUNCTION + "() {") + end = source.index("\n}\n", start) + 3 + before = source[start:end] + pairs = [("17 tool rounds", "18 tool rounds"), + ("16-call budget", "17-call budget"), ("the 17th", "the 18th"), + ("(0..17)", "(0..18)"), ("Some(16)", "Some(17)"), + ("run.tool_count, 16", "run.tool_count, 17"), + ('"g16.txt"', '"g17.txt"'), ('"g15.txt"', '"g16.txt"')] + after = before + for old, new in pairs: + if old not in after: + raise ValueError("oracle source changed: " + old) + after = after.replace(old, new) + return source[:start] + after + source[end:], before, after + + +def test_names(data): + return re.findall(rb"#\[test\]\s*(?:async\s+)?fn\s+(\w+)", data) + + +def integrity(original, expected, actual): + _, old, new = expected_edit(original.decode()) + prefix, suffix = original.split(old.encode()) + return { + "exact_target_and_file": actual == expected, + "untouched_prefix": actual.startswith(prefix), + "untouched_suffix": actual.endswith(suffix), + "target_present": new.encode() in actual, + "test_names_preserved": test_names(actual) == test_names(original), + "original_test_count": len(test_names(original)), + "actual_test_count": len(test_names(actual)), + "original_sha256": digest(original), + "expected_sha256": digest(expected), + "actual_sha256": digest(actual), + } + + +def command(args, cwd, log, env=None): + with log.open("wb") as out: + result = subprocess.run(args, cwd=cwd, env=env, stdout=out, + stderr=subprocess.STDOUT, check=False) + return {"command": [str(a) for a in args], "exit": result.returncode, + "log": str(log)} + + +def prepare(destination, mode, guidance, damage): + destination.mkdir(parents=True, exist_ok=False) + workspace = destination / "workspace" + materialize = command(["python3", str(ROOT / "scripts/materialize-git-tree.py"), + str(ROOT), START, str(workspace)], ROOT, + destination / "materialize.log") + if materialize["exit"]: + raise RuntimeError(materialize) + original = (workspace / TARGET).read_bytes() + expected, _, _ = expected_edit(original.decode()) + (destination / "original.rs").write_bytes(original) + (destination / "expected.rs").write_text(expected) + baseline = {str(p.relative_to(workspace)): digest(p.read_bytes()) + for p in workspace.rglob("*") if p.is_file()} + provenance = None + if mode == "recovery": + raw = damage.read_bytes() + if digest(raw) != "2d69b6679c6e8baf921249939f6c743900931d51ed463c34a47132849b727ab8": + raise ValueError("damaged evidence hash mismatch") + (workspace / "recovery").mkdir() + (workspace / "recovery/phase2.rs").write_bytes(original) + (workspace / TARGET).write_bytes(raw) + provenance = {"path": "recovery/phase2.rs", "sha256": digest(original), + "source": START + ":" + TARGET} + baseline["recovery/phase2.rs"] = digest(original) + prompt = f"""Update only {FUNCTION} in tests/phase2.rs. Change its declared +16-call budget to 17, the attempted tool rounds from 17 to 18, and update every +related assertion and comment: 17 calls execute, g17.txt is absent, g16.txt exists. +Preserve imports, the successful multiple_tool_calls_share_remaining_output_budget +test, and every other byte outside the target function. Do not change other files. +Do not run builds; an independent controller compiles and runs the full suite. +""" + if provenance: + prompt += ("The file is damaged. Restore first from the authorized local " + "recovery/phase2.rs, SHA256 " + digest(original) + ".\n") + if guidance == "explicit": + prompt += GUIDANCE + (destination / "prompt.txt").write_text(prompt) + manifest = {"source_sha": START, "mode": mode, "guidance": guidance, + "fixture_identity": "controlled current-source tool-count probe, not lost M2 source", + "baseline": baseline, "recovery_provenance": provenance, + "materialize": materialize} + (destination / "manifest.json").write_text(json.dumps(manifest, indent=2)) + + +def trace_grade(destination, manifest): + result = {"trace_complete": False, "tool_selection": [], + "calls_before_first_edit": None, "sibling_source_isolation": False, + "recovery_source_used": manifest["mode"] == "ordinary"} + if not (destination / "terminal.json").exists(): + return result + terminal = json.loads((destination / "terminal.json").read_text()) + envelope = json.loads((destination / "stdout.json").read_text()) + if terminal["exit"] != 0 or envelope.get("status") != "ok": + return result + exporter = ROOT / "target/debug/examples/issue220_trace" + if not exporter.is_file(): + return result + export = command([str(exporter), envelope["session_id"]], ROOT, + destination / "validated-trace.json", os.environ.copy()) + result["trace_export"] = export + if export["exit"]: + return result + state = json.loads((destination / "validated-trace.json").read_text()) + branch, = [b for b in state["branches"] if b["branch_id"] == envelope["branch_id"]] + calls = [c for r in branch["rounds"] for c in r["calls"]] + result["tool_selection"] = [c["name"] for c in calls] + result["calls_before_first_edit"] = next( + (i for i, c in enumerate(calls) if c["name"] in ("write_file", "replace")), None) + result["trace_complete"] = len([c for c in calls if not c["refused"]]) == envelope["tool_calls"] + isolated = True + for c in calls: + args = json.loads(c["args"]) + name = c["name"] + if name not in ("read_file", "write_file", "replace", "list_directory", "search_file_content"): + isolated = False + path = Path(args.get("path", ".")) + if path.is_absolute() or ".." in path.parts: + isolated = False + if (name == "read_file" and str(path) == "recovery/phase2.rs" + and c["ok"] and not c["refused"]): + # Provenance requires a complete returned source, not merely a successful read. + original = (destination / "original.rs").read_text() + if original in c["result"]: + result["recovery_source_used"] = True + result["sibling_source_isolation"] = isolated + return result + + +def grade(destination, compile_tests): + workspace = destination / "workspace" + manifest = json.loads((destination / "manifest.json").read_text()) + result = integrity((destination / "original.rs").read_bytes(), + (destination / "expected.rs").read_bytes(), + (workspace / TARGET).read_bytes()) + changed = [] + for name, sha in manifest["baseline"].items(): + p = workspace / name + if name != TARGET and (not p.is_file() or p.is_symlink() or digest(p.read_bytes()) != sha): + changed.append(name) + result["new_files"] = sorted(str(p.relative_to(workspace)) + for p in workspace.rglob("*") + if p.is_file() and str(p.relative_to(workspace)) + not in manifest["baseline"]) + result["unexpected_changes"] = changed + result["gates"] = [] + if compile_tests: + env = dict(os.environ, CARGO_BUILD_JOBS="2", CARGO_TARGET_DIR=str(ROOT / "target")) + for name, args in [("compile", ["test", "--no-run"]), + ("phase2", ["test", "--test", "phase2", "--", "--nocapture"])]: + # Cargo flags precede the test harness separator. + split = args.index("--") if "--" in args else len(args) + cmd = ["cargo", "+1.88.0"] + args[:split] + ["--offline", "--locked"] + args[split:] + result["gates"].append(command(cmd, workspace, destination / (name + ".log"), env)) + result["integrity_pass"] = (result["exact_target_and_file"] and not changed + and not result["new_files"]) + result["file_and_test_pass"] = (result["integrity_pass"] and compile_tests + and all(g["exit"] == 0 for g in result["gates"])) + # Full acceptance additionally requires authenticated call-level provenance. + result["acceptance"] = False + result.update(trace_grade(destination, manifest)) + result["acceptance"] = (result["file_and_test_pass"] + and result["trace_complete"] + and result["sibling_source_isolation"] + and result["recovery_source_used"]) + (destination / "grade.json").write_text(json.dumps(result, indent=2)) + return result + + +def live(destination, binary, profile): + if not os.environ.get("LLXPRT_CONFIG_HOME"): + raise ValueError("LLXPRT_CONFIG_HOME must be explicitly set") + session = "issue220-" + destination.name + "-" + str(time.time_ns()) + cmd = [str(binary), "--profile-load", str(profile), "--session", session, + "--cwd", str(destination / "workspace"), "--turn-time", "5m", + "--max-tool-calls", "40", "--mem-profile", str(destination / "rss.jsonl")] + started = time.monotonic() + with (destination / "stdout.json").open("wb") as out, (destination / "stderr.log").open("wb") as err: + child = subprocess.run(cmd, input=(destination / "prompt.txt").read_bytes(), + stdout=out, stderr=err, check=False) + terminal = {"command": cmd, "exit": child.returncode, + "elapsed_seconds": time.monotonic() - started, + "binary_sha256": digest(binary.read_bytes()), + "config_home_explicit": True, "shell_enabled": False} + (destination / "terminal.json").write_text(json.dumps(terminal, indent=2)) + return child.returncode + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("action", choices=["prepare", "run", "grade"]) + parser.add_argument("destination", type=Path) + parser.add_argument("--mode", choices=["ordinary", "recovery"], default="ordinary") + parser.add_argument("--guidance", choices=["current", "explicit"], default="current") + parser.add_argument("--damage", type=Path) + parser.add_argument("--binary", type=Path) + parser.add_argument("--profile", type=Path) + parser.add_argument("--compile-tests", action="store_true") + args = parser.parse_args() + dest = args.destination.resolve() + dest.relative_to(ROOT / "evalwork/results/branch4-wave2/issue220") + if args.action == "prepare": + if args.mode == "recovery" and not args.damage: + parser.error("recovery requires --damage") + prepare(dest, args.mode, args.guidance, args.damage) + elif args.action == "run": + if not args.binary or not args.profile: + parser.error("run requires --binary and --profile") + raise SystemExit(live(dest, args.binary.resolve(), args.profile.resolve())) + else: + raise SystemExit(0 if grade(dest, args.compile_tests)["acceptance"] else 1) + + +if __name__ == "__main__": + main() diff --git a/evals/edit-recovery/export_historical.py b/evals/edit-recovery/export_historical.py new file mode 100644 index 00000000..58b16d7d --- /dev/null +++ b/evals/edit-recovery/export_historical.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Export completed candidate trials via current production validation, no old-state reader.""" +import argparse +import json +import os +from pathlib import Path +import historical as h + + +def export(dest): + envelope = json.loads((dest / 'stdout.json').read_text()) + exporter = h.ROOT / 'target/debug/examples/issue220_trace' + session = envelope['session_id'] + result = h.command([exporter, session], h.ROOT, dest / 'exported-trace.json', os.environ.copy()) + identity = {'exporter_sha256': h.sha(exporter.read_bytes()), + 'source_qualification': 'rebased issue220 current production SessionStore::load_at/snapshot; no candidate source merge', + 'validation_command': result, 'historical_trace_authentication': False} + h.save(dest / 'trace-identity.json', identity) + if result['exit']: + return + state = json.loads((dest / 'exported-trace.json').read_text()) + branch, = [b for b in state['branches'] if b['branch_id'] == envelope['branch_id']] + calls = [c for r in branch['rounds'] for c in r['calls']] + roster = [] + for i, c in enumerate(calls): + args = json.loads(c['args']) + roster.append({'ordinal': i + 1, 'name': c['name'], 'ok': c['ok'], + 'refused': c['refused'], 'args': args, 'result': c['result']}) + h.save(dest / 'calls.json', roster) + events = [json.loads(line) for line in (dest / 'rss.jsonl').read_text().splitlines()] + summary = { + 'calls': len(calls), 'executed_calls': sum(not c['refused'] for c in calls), + 'envelope_calls': envelope['tool_calls'], + 'trace_count_matches': sum(not c['refused'] for c in calls) == envelope['tool_calls'], + 'registered_edit_ordinals': [i + 1 for i, c in enumerate(calls) if c['name'] in ('write_file', 'replace')], + 'shell_ordinals_requiring_inspection': [i + 1 for i, c in enumerate(calls) if c['name'] == 'run_shell_command'], + 'tool_failures': [{'ordinal': i + 1, 'name': c['name'], 'result': c['result']} + for i, c in enumerate(calls) if not c['ok'] or c['refused']], + 'peak_rss_bytes': max(e['peak_rss_bytes'] for e in events), + 'model_requests': sum(e['phase'] == 'model_call_before' for e in events), + 'max_request_estimate_bytes': max((e['request_estimate_bytes'] or 0) for e in events), + 'live_output_bytes': max(e['turn_output_bytes'] for e in events), + 'tokens': None, + 'isolation_limit': 'Shell has general user privileges, not an OS sandbox. Inspect all commands for observed access; no claim of syscall-level containment.', + 'provenance_limit': 'Compacted result handles do not authenticate full live model-visible result bytes. File identities and retained explicit copy/read operations are assessed separately.'} + h.save(dest / 'trace-summary.json', summary) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('destination', type=Path) + args = parser.parse_args() + dest = args.destination.resolve() + dest.relative_to(h.EVIDENCE) + export(dest) diff --git a/evals/edit-recovery/historical.py b/evals/edit-recovery/historical.py new file mode 100644 index 00000000..c238adcf --- /dev/null +++ b/evals/edit-recovery/historical.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""Historical #220 shell-enabled trials. No historical wire replay or credential handling.""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import shutil +import subprocess +import time + +ROOT = Path(__file__).resolve().parents[2] +EVIDENCE = ROOT / 'evalwork/results/branch4-wave2/issue220' +HISTORY = EVIDENCE / 'recovered-history' +BASE = 'c6645998af3caba5d36b7bd04ae2093babca6a1d' +CANDIDATE = '2566994ab3ebab82edd2f52db55afba2a595fc8b' +BINARY_SHA = '7d9b7a1ca01c5c6145357ce9aa354558842a273a8b1eec12b5273cb074357320' +ORIGINAL_SHA = 'c84874023f22db98b647387e89e1adf2eebcbdc910a8049ff65d98fe7b75c156' +DAMAGE_SHA = '2d69b6679c6e8baf921249939f6c743900931d51ed463c34a47132849b727ab8' +TARGET = 'tests/phase2.rs' +FUNCTION = 'output_budget_exhaustion_fails_the_next_call_before_execution' +GUIDANCE = '''For an existing-file block edit, read the current content and use replace +with an exact unique old string. write_file replaces the entire file and is for new +files or intentional complete rewrites, never a replacement block. Never reconstruct +a file from a truncated read or CTXDIGEST. For recovery, use only the supplied local +recovery/phase2.rs and verify its SHA256 before restoring. Do not search sibling +worktrees or import their source. Verify preservation, not merely tool success. +''' + + +def sha(data): + return hashlib.sha256(data).hexdigest() + + +def save(path, value): + with path.open('x') as out: + json.dump(value, out, indent=2) + out.write('\n') + + +def command(args, cwd, path, env=None): + started = time.monotonic() + with path.open('xb') as out: + child = subprocess.run(list(map(str, args)), cwd=cwd, env=env, + stdout=out, stderr=subprocess.STDOUT, check=False) + result = {'command': list(map(str, args)), 'cwd': str(cwd), + 'exit': child.returncode, 'seconds': time.monotonic() - started, + 'log': str(path), 'log_sha256': sha(path.read_bytes())} + save(path.with_suffix(path.suffix + '.command.json'), result) + return result + + +def split_target(data): + # The historical test has column-zero function delimiters. Refuse ambiguity. + needle = ('fn ' + FUNCTION + '() {').encode() + if data.count(needle) != 1: + raise ValueError('missing or ambiguous target') + start = data.index(needle) + end = data.index(b'\n}\n', start) + 3 + return data[:start], data[start:end], data[end:] + + +def names(data): + return [n.decode() for n in re.findall(rb'#\[test\]\s*fn\s+(\w+)', data)] + + +def integrity(original, actual): + prefix, block, suffix = split_target(original) + try: + ap, ab, az = split_target(actual) + except ValueError: + ap, ab, az = b'', b'', b'' + return {'prefix_preserved': ap == prefix, 'suffix_preserved': az == suffix, + 'test_inventory_preserved': names(original) == names(actual), + 'original_tests': names(original), 'actual_tests': names(actual), + 'original_sha256': sha(original), 'actual_sha256': sha(actual), + 'target_changed': bool(ab) and ab != block, + 'invalid_partial_round_access_removed': bool(ab) and b'branch.rounds[0]' not in ab, + 'original_lines': original.count(b'\n'), 'actual_lines': actual.count(b'\n')} + + +def prepare(dest, mode, guidance): + dest.mkdir(parents=True, exist_ok=False) + workspace = dest / 'workspace' + result = command(['python3', ROOT / 'scripts/materialize-git-tree.py', ROOT, BASE, + workspace], ROOT, dest / 'materialize.log') + if result['exit']: + raise RuntimeError(result) + proof = json.loads((HISTORY / 'reconstruction-proof.json').read_text()) + for entry in proof['modified_files_reconstructed']: + name = entry['path'] + data = (HISTORY / 'reconstructed-pre-m2' / name).read_bytes() + assert sha(data) == entry['reconstructed_sha256'] + (workspace / name).write_bytes(data) + baseline = json.loads((HISTORY / 'launcher/before.files.json').read_text()) + mismatches = [name for name, expected in baseline.items() + if not (workspace / name).is_file() or sha((workspace / name).read_bytes()) != expected] + save(dest / 'full-launcher-check.json', {'files': len(baseline), 'mismatches': mismatches}) + if mismatches: + raise ValueError('full historical workspace does not match launcher') + original = (workspace / TARGET).read_bytes() + assert sha(original) == ORIGINAL_SHA + (dest / 'original.rs').write_bytes(original) + original_prompt = (HISTORY / 'launcher/prompt.txt').read_text() + old_cwd = '/Users/acoliver/projects/llxprt/agent/branch-2/tmp/rollout-worktrees/issue66' + assert original_prompt.count(old_cwd) == 1 + prompt = original_prompt.replace(old_cwd, str(workspace)) + prompt += '\nThis is a disposable issue220 investigation, not the owner worktree. Do not access sibling worktrees or external source. Preserve every byte outside the named test function, including existing imports and the shared-budget test. Do not commit. Use cargo +1.88.0 fmt if formatting is needed, limited to the target block. Independent compilation follows.\n' + if mode == 'recovery': + (workspace / 'recovery').mkdir() + (workspace / 'recovery/phase2.rs').write_bytes(original) + baseline['recovery/phase2.rs'] = ORIGINAL_SHA + damage = (HISTORY / 'damaged-observation/phase2.rs.observed').read_bytes() + assert sha(damage) == DAMAGE_SHA + (workspace / TARGET).write_bytes(damage) + prompt += ('The target file is currently damaged. First explicitly restore from the authorized ' + 'local recovery/phase2.rs, SHA256 ' + ORIGINAL_SHA + '. This is the hash-proven ' + 'dirty pre-M2 source reconstructed from the original launcher diff, not clean Git ' + 'or sibling source. Then perform the same requested regression edit.\n') + if guidance == 'explicit': + prompt += GUIDANCE + (dest / 'prompt.txt').write_text(prompt) + save(dest / 'manifest.json', {'base': BASE, 'mode': mode, 'guidance': guidance, + 'before_diff_sha256': sha((HISTORY / 'launcher/before.diff').read_bytes()), + 'original_prompt_sha256': sha(original_prompt.encode()), 'prompt_sha256': sha(prompt.encode()), + 'baseline': baseline, 'starting_target_sha256': sha((workspace / TARGET).read_bytes()), + 'prompt_adaptations': ['workspace path', 'common isolation/preservation instruction', + 'recovery instruction when applicable', 'guidance only in explicit arm']}) + + +def live(dest, binary, profile): + home = os.environ['LLXPRT_CONFIG_HOME'] + assert sha(binary.read_bytes()) == BINARY_SHA, 'wrong tool generation' + manifest = json.loads((dest / 'manifest.json').read_text()) + assert sha((dest / 'workspace' / TARGET).read_bytes()) == manifest['starting_target_sha256'] + session = 'issue220-' + dest.name + '-' + str(time.time_ns()) + cmd = [str(binary), '--profile-load', str(profile), '--session', session, + '--cwd', str(dest / 'workspace'), '--allow-shell', '--max-tool-calls=-1', + '--max-shell-output', '32768', '--max-tool-output', '16777216', + '--max-turn-output', '16777216', '--mem-profile', str(dest / 'rss.jsonl')] + settings = Path(home) / 'settings.json' + save(dest / 'launch.json', {'command': cmd, 'binary_source': CANDIDATE, + 'binary_sha256': BINARY_SHA, 'qualification': 'unmerged #66 owner candidate, not stock main or historical model replay', + 'profile_path': str(profile), 'profile_sha256': sha(profile.read_bytes()), + 'config_home': home, 'settings_sha256': sha(settings.read_bytes()) if settings.is_file() else None, + 'prompt_sha256': sha((dest / 'prompt.txt').read_bytes()), 'shell': True, + 'tool_calls': 'unlimited', 'turn_time': 'unset, matching original', + 'output_caps': {'shell': 32768, 'tool': 16777216, 'turn': 16777216}, + 'environment': {k: os.environ.get(k) for k in ['CARGO_BUILD_JOBS', 'CARGO_TARGET_DIR', 'RUSTUP_TOOLCHAIN', 'TMPDIR']}}) + started = time.monotonic() + with (dest / 'stdout.json').open('xb') as out, (dest / 'stderr.log').open('xb') as err: + child = subprocess.run(cmd, input=(dest / 'prompt.txt').read_bytes(), stdout=out, + stderr=err, env=os.environ.copy(), check=False) + save(dest / 'terminal.json', {'exit': child.returncode, 'seconds': time.monotonic() - started, + 'stdout_sha256': sha((dest / 'stdout.json').read_bytes()), + 'stderr_sha256': sha((dest / 'stderr.log').read_bytes())}) + + +def grade(dest): + manifest = json.loads((dest / 'manifest.json').read_text()) + workspace = dest / 'workspace' + actual = (workspace / TARGET).read_bytes() + (dest / 'final.rs').write_bytes(actual) + result = integrity((dest / 'original.rs').read_bytes(), actual) + result['unexpected_changes'] = [name for name, expected in manifest['baseline'].items() + if name != TARGET and (not (workspace / name).is_file() or (workspace / name).is_symlink() + or sha((workspace / name).read_bytes()) != expected)] + result['new_files'] = sorted(str(p.relative_to(workspace)) for p in workspace.rglob('*') + if p.is_file() and str(p.relative_to(workspace)) not in manifest['baseline']) + env = dict(os.environ, CARGO_BUILD_JOBS='2', CARGO_TARGET_DIR=str(ROOT / 'target/issue220-historical')) + result['gates'] = [] + for label, args in [('compile', ['test', '--no-run', '--test', 'phase2']), + ('phase2', ['test', '--test', 'phase2'])]: + result['gates'].append(command(['cargo', '+1.88.0'] + args + ['--offline', '--locked'], + workspace, dest / (label + '.log'), env)) + result['structural_and_test_pass'] = all(result[k] for k in [ + 'prefix_preserved', 'suffix_preserved', 'test_inventory_preserved', 'target_changed', + 'invalid_partial_round_access_removed']) and not result['unexpected_changes'] and not result['new_files'] and all(g['exit'] == 0 for g in result['gates']) + result['execution_integrity'] = 'requires retained call trace and independent scenario/provenance assessment; never inferred from exit zero' + save(dest / 'grade.json', result) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('action', choices=['prepare', 'run', 'grade']) + parser.add_argument('destination', type=Path) + parser.add_argument('--mode', choices=['ordinary', 'recovery'], default='ordinary') + parser.add_argument('--guidance', choices=['current', 'explicit'], default='current') + parser.add_argument('--binary', type=Path) + parser.add_argument('--profile', type=Path) + args = parser.parse_args() + dest = args.destination.resolve() + dest.relative_to(EVIDENCE) + if not dest.name.startswith('impl2-'): + parser.error('new impl2-* destination required') + if args.action == 'prepare': + prepare(dest, args.mode, args.guidance) + elif args.action == 'run': + live(dest, args.binary.resolve(), args.profile.resolve()) + else: + grade(dest) + + +if __name__ == '__main__': + main() diff --git a/evals/edit-recovery/mutation_historical.py b/evals/edit-recovery/mutation_historical.py new file mode 100644 index 00000000..712712fd --- /dev/null +++ b/evals/edit-recovery/mutation_historical.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Independent negative control: force a refused call to execute in a disposable copy.""" +import argparse +import json +import os +import shutil +from pathlib import Path +import historical as h + + +def mutation(trial, dest): + dest.mkdir(exist_ok=False) + workspace = dest / 'workspace' + shutil.copytree(trial / 'workspace', workspace) + path = workspace / 'src/agent.rs' + source = path.read_text() + old = ''' if remaining_output == 0 { + return Err(ToolCallFailure::OutputCap); + }''' + new = ''' if remaining_output == 0 { + // EVAL MUTANT ONLY: simulate the forbidden post-exhaustion execution. + let parsed = parse_object_args(call).map_err(ToolCallFailure::Invalid)?; + let _ = crate::tools::execute_tool(&self.cwd, &call.name, parsed, config); + return Err(ToolCallFailure::OutputCap); + }''' + assert source.count(old) == 1 + path.write_text(source.replace(old, new)) + h.save(dest / 'mutation.json', {'trial': str(trial), 'original_agent_sha256': h.sha(source.encode()), + 'mutated_agent_sha256': h.sha(path.read_bytes()), 'old': old, 'new': new, + 'target_sha256': h.sha((workspace / h.TARGET).read_bytes()), + 'qualification': 'deliberately defective disposable runtime only, not production change'}) + env = dict(os.environ, CARGO_BUILD_JOBS='2', CARGO_TARGET_DIR=str(h.ROOT / 'target/issue220-historical')) + result = h.command(['cargo', '+1.88.0', 'test', '--offline', '--locked', '--test', 'phase2', + h.FUNCTION, '--', '--exact', '--nocapture'], workspace, dest / 'mutant.log', env) + log = (dest / 'mutant.log').read_text() + rejected = (result['exit'] == 101 and 'test result: FAILED. 0 passed; 1 failed;' in log + and 'panicked at tests/phase2.rs:' in log + and ('seventeenth call' in log or 'executor must not run' in log)) + h.save(dest / 'result.json', {'command': result, + 'expected_exit': 101, 'execution_witness_rejected_mutant': rejected, + 'qualification': 'Requires the named execution-witness panic, not just nonzero exit.'}) + + +if __name__ == '__main__': + p = argparse.ArgumentParser(description=__doc__) + p.add_argument('trial', type=Path) + p.add_argument('destination', type=Path) + a = p.parse_args() + for path in [a.trial.resolve(), a.destination.resolve()]: + path.relative_to(h.EVIDENCE) + mutation(a.trial.resolve(), a.destination.resolve()) diff --git a/evals/edit-recovery/test_evaluate.py b/evals/edit-recovery/test_evaluate.py new file mode 100644 index 00000000..a965edcb --- /dev/null +++ b/evals/edit-recovery/test_evaluate.py @@ -0,0 +1,65 @@ +"""Run with python3 -m unittest discover -s evals/edit-recovery -p 'test_*.py'.""" +import importlib.util +from pathlib import Path +import subprocess +import unittest + +SPEC = importlib.util.spec_from_file_location("evaluate", Path(__file__).with_name("evaluate.py")) +eval_module = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(eval_module) + + +class IntegrityTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.original = subprocess.check_output( + ["git", "show", eval_module.START + ":" + eval_module.TARGET], cwd=eval_module.ROOT) + expected, cls.old, cls.new = eval_module.expected_edit(cls.original.decode()) + cls.expected = expected.encode() + + def check_integrity(self, actual): + return eval_module.integrity(self.original, self.expected, actual) + + def test_exact_edit_preserves_all_tests(self): + grade = self.check_integrity(self.expected) + self.assertTrue(grade["exact_target_and_file"]) + self.assertTrue(grade["test_names_preserved"]) + self.assertGreater(grade["actual_test_count"], 20) + + def test_whole_file_write_of_block_is_rejected(self): + grade = self.check_integrity(self.new.encode()) + self.assertFalse(grade["exact_target_and_file"]) + self.assertFalse(grade["untouched_prefix"]) + self.assertFalse(grade["untouched_suffix"]) + self.assertFalse(grade["test_names_preserved"]) + + def test_unchanged_file_is_not_success(self): + self.assertFalse(self.check_integrity(self.original)["exact_target_and_file"]) + + def test_removed_successful_budget_test_is_rejected(self): + damaged = self.expected.replace(b"fn multiple_tool_calls_share_remaining_output_budget", + b"fn removed_successful_budget_test") + grade = self.check_integrity(damaged) + self.assertFalse(grade["exact_target_and_file"]) + self.assertFalse(grade["test_names_preserved"]) + + def test_import_loss_is_rejected_even_if_target_is_correct(self): + damaged = self.expected.replace(b"use std::sync::Mutex;", b"") + self.assertFalse(self.check_integrity(damaged)["untouched_prefix"]) + + def test_stale_source_fails_fast(self): + with self.assertRaises(ValueError): + eval_module.expected_edit(self.expected.decode()) + + + def test_recovery_without_terminal_is_not_certified(self): + import tempfile + with tempfile.TemporaryDirectory(dir=eval_module.ROOT / "evalwork") as directory: + result = eval_module.trace_grade(Path(directory), {"mode": "recovery"}) + self.assertFalse(result["trace_complete"]) + self.assertFalse(result["recovery_source_used"]) + self.assertFalse(result["sibling_source_isolation"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/edit-recovery/test_historical.py b/evals/edit-recovery/test_historical.py new file mode 100644 index 00000000..ebe0fd04 --- /dev/null +++ b/evals/edit-recovery/test_historical.py @@ -0,0 +1,53 @@ +"""Historical grader regression tests, including false-success controls.""" +import importlib.util +from pathlib import Path +import unittest + +SPEC = importlib.util.spec_from_file_location('historical', Path(__file__).with_name('historical.py')) +h = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(h) + + +class HistoricalIntegrityTests(unittest.TestCase): + def setUp(self): + self.source = (b'use std::sync::Mutex;\n#[test]\nfn shared() {}\n#[test]\nfn ' + + h.FUNCTION.encode() + b'() {\n assert!(branch.rounds[0].calls.len() > 0);\n}\n' + + b'#[test]\nfn retained() {}\n') + prefix, _, suffix = h.split_target(self.source) + self.edited = prefix + ('fn ' + h.FUNCTION + '() {\n assert!(!witness.exists());\n}\n').encode() + suffix + + def test_unchanged_exit_zero_is_not_success(self): + self.assertFalse(h.integrity(self.source, self.source)['target_changed']) + + def test_block_write_loses_imports_and_tests(self): + _, block, _ = h.split_target(self.edited) + result = h.integrity(self.source, block) + self.assertFalse(result['prefix_preserved']) + self.assertFalse(result['suffix_preserved']) + self.assertFalse(result['test_inventory_preserved']) + + def test_target_edit_retains_outside_bytes(self): + result = h.integrity(self.source, self.edited) + for key in ['prefix_preserved', 'suffix_preserved', 'target_changed', + 'test_inventory_preserved', 'invalid_partial_round_access_removed']: + self.assertTrue(result[key], key) + # Structural success deliberately does not certify semantic correctness. + self.assertNotIn('acceptance', result) + + def test_import_loss_rejected_even_with_same_inventory(self): + result = h.integrity(self.source, self.edited.replace(b'use std::sync::Mutex;', b'')) + self.assertTrue(result['test_inventory_preserved']) + self.assertFalse(result['prefix_preserved']) + + def test_shared_test_mutation_rejected(self): + result = h.integrity(self.source, self.edited.replace(b'fn shared() {}', b'fn shared() { panic!(); }')) + self.assertTrue(result['test_inventory_preserved']) + self.assertFalse(result['prefix_preserved']) + + def test_ambiguous_target_fails_closed(self): + with self.assertRaises(ValueError): + h.split_target(self.source + self.source) + + +if __name__ == '__main__': + unittest.main() diff --git a/examples/issue220_prefix.rs b/examples/issue220_prefix.rs new file mode 100644 index 00000000..94501c61 --- /dev/null +++ b/examples/issue220_prefix.rs @@ -0,0 +1,28 @@ +//! Export the current host-level #220 tool guidance, not an outbound provider request. +use llxprt_code_rs::agent::coding_system_prompt; +use llxprt_code_rs::tools::tool_specs; +use serde_json::json; + +fn main() -> Result<(), Box> { + let mut args = std::env::args().skip(1); + let cwd = args.next().ok_or("expected workspace path")?; + if args.next().is_some() { + return Err("expected exactly one workspace path".into()); + } + let tools: Vec<_> = tool_specs(true) + .into_iter() + .map(|tool| { + json!({"name": tool.name, "description": tool.description, + "properties": tool.properties}) + }) + .collect(); + println!( + "{}", + serde_json::to_string_pretty(&json!({ + "qualification": "host prompt and ToolSpec values, not serialized wire capture", + "system": coding_system_prompt(std::path::Path::new(&cwd), "", true, None), + "tools": tools, + }))? + ); + Ok(()) +} diff --git a/examples/issue220_trace.rs b/examples/issue220_trace.rs new file mode 100644 index 00000000..70bb5a9b --- /dev/null +++ b/examples/issue220_trace.rs @@ -0,0 +1,16 @@ +//! Export one completed issue220 trial through the production session validator. +//! No direct slot parsing, fallback reader, or cross-session directory scanning. +use llxprt_code_rs::session::{SessionId, SessionStore}; + +fn main() -> Result<(), Box> { + let mut args = std::env::args().skip(1); + let session = args.next().ok_or("expected session id")?; + if args.next().is_some() || !session.starts_with("issue220-") { + return Err("expected exactly one issue220- session id".into()); + } + let root = std::env::var_os("LLXPRT_CONFIG_HOME").ok_or("set LLXPRT_CONFIG_HOME")?; + let store = SessionStore::load_at(&SessionId::parse(&session)?, std::path::Path::new(&root))?; + let state = store.snapshot()?; + println!("{}", serde_json::to_string_pretty(&state)?); + Ok(()) +} diff --git a/tests/edit_recovery.rs b/tests/edit_recovery.rs new file mode 100644 index 00000000..17b35235 --- /dev/null +++ b/tests/edit_recovery.rs @@ -0,0 +1,70 @@ +//! #220: exercise the registered production editor boundary, not a simulated write. +use llxprt_code_rs::tools::{ + execute_tool, tool_specs, ShellConfig, ShellTimeoutPolicy, ToolConfig, WorkspaceCap, +}; +use serde_json::json; +use std::time::Duration; + +#[test] +fn block_as_whole_file_write_succeeds_but_destroys_preservation() { + let cwd = tempfile::tempdir().unwrap(); + let original = "use std::sync::Mutex;\n#[test]\nfn kept() {}\n#[test]\nfn target() { assert_eq!(1, 1); }\n"; + let block = "fn target() { assert_eq!(2, 2); }"; + std::fs::write(cwd.path().join("phase2.rs"), original).unwrap(); + let config = ToolConfig { + ws: WorkspaceCap::open(cwd.path()).unwrap(), + max_output_bytes: 1024 * 1024, + digest_size_floor: llxprt_code_rs::context_ingress::filter::DEFAULT_DIGEST_SIZE_FLOOR, + shell: ShellConfig { + max_shell_output: 1024, + timeouts: ShellTimeoutPolicy { + default: Some(Duration::from_secs(1)), + maximum: Some(Duration::from_secs(1)), + }, + allow_shell: false, + }, + }; + assert!(tool_specs(false).iter().any(|s| s.name == "write_file")); + assert!(tool_specs(false).iter().any(|s| s.name == "replace")); + let (ok, _) = execute_tool( + cwd.path(), + "write_file", + json!({"path": "phase2.rs", "content": block}), + &config, + ); + assert!(ok, "whole-file semantics are not themselves a tool failure"); + let damaged = std::fs::read_to_string(cwd.path().join("phase2.rs")).unwrap(); + assert_eq!(damaged, block); + assert!(!damaged.contains("fn kept()")); + + // An explicit authorized restoration, not an automatic tool fallback. + let (ok, _) = execute_tool( + cwd.path(), + "write_file", + json!({"path": "phase2.rs", "content": original}), + &config, + ); + assert!(ok); + let (ok, result) = execute_tool( + cwd.path(), + "replace", + json!({"path": "phase2.rs", "old_string": "assert_eq!(1, 1)", + "new_string": "assert_eq!(2, 2)"}), + &config, + ); + assert!(ok, "{result}"); + assert_eq!( + std::fs::read_to_string(cwd.path().join("phase2.rs")).unwrap(), + original.replace("assert_eq!(1, 1)", "assert_eq!(2, 2)") + ); + let before = std::fs::read(cwd.path().join("phase2.rs")).unwrap(); + let (ok, _) = execute_tool( + cwd.path(), + "replace", + json!({"path": "phase2.rs", "old_string": "assert_eq!(1, 1)", + "new_string": "assert_eq!(3, 3)"}), + &config, + ); + assert!(!ok, "a stale block must not silently overwrite the suite"); + assert_eq!(std::fs::read(cwd.path().join("phase2.rs")).unwrap(), before); +}