From 9fb791b0cea9e60cd81650d39710f27b309763e5 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 12:41:26 +0800 Subject: [PATCH 01/17] feat: add contract-aware documentation policy --- skills/scientific-code-documenter/SKILL.md | 54 ++++++++++++++-- .../agents/openai.yaml | 4 +- .../references/comment-examples.md | 48 ++++++++++++++ .../references/jupyter-notebooks.md | 57 +++++++++++++++++ .../references/scientific-contracts.md | 62 +++++++++++++++++++ 5 files changed, 217 insertions(+), 8 deletions(-) create mode 100644 skills/scientific-code-documenter/references/jupyter-notebooks.md create mode 100644 skills/scientific-code-documenter/references/scientific-contracts.md diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/scientific-code-documenter/SKILL.md index a7c20e9..541c76f 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/scientific-code-documenter/SKILL.md @@ -1,6 +1,6 @@ --- name: scientific-code-documenter -description: Write, refactor, document, or review scientific and technical code for readable structure, concise human-sounding comments and docstrings, and explicit units, shapes, assumptions, conventions, and numerical choices while preserving behavior. Use when users ask to add or improve comments or docstrings, clean up cramped code, improve scientific-code readability, document units, dimensions, array or table shapes, scientific assumptions, or numerical conventions, remove excessive or AI-like comments, or review documentation quality. Do not trigger for ordinary debugging, dependency updates, test execution, or feature implementation that does not request documentation or readability work. +description: Document, prune, refactor, or review scientific code and Jupyter notebooks while preserving scientific, data, interface, execution-state, and artifact contracts. Use for evidence-backed comments, docstrings, notebook Markdown, units, shapes, schemas, ordering, missing-value conventions, assumptions, numerical choices, or readability work that must not change analysis semantics. Do not trigger for ordinary notebook execution, data analysis, debugging, performance work, methodology changes, output regeneration, dependency updates, or feature implementation. --- # Scientific Code Documenter @@ -16,11 +16,13 @@ Improve scientific and technical code without obscuring its logic or silently ch - Read nearby code to learn naming, layout, docstring, and comment conventions. - Identify generated and vendored files that should remain untouched. - Check the working tree and preserve unrelated user changes. -2. Classify the request as new code, documentation, readability refactoring, or review. Perform only the requested mode or modes. -3. Define the smallest coherent scope. Prefer focused edits over repository-wide style churn. -4. Apply repository-specific conventions first. Improve local inconsistency only when the existing pattern is unclear, cramped, or misleading. -5. Preserve public interfaces, calculations, numerical behavior, outputs, and file formats unless the user explicitly requests a behavioral change. -6. Run the most relevant available validation and report exactly what ran. +2. Classify the operating mode and treat review-only or no-edit language as a hard boundary. +3. Build a compact contract inventory for the requested scope. Classify each relevant item as `evidence-backed`, `observable-only`, or `unknown`. +4. For notebooks, classify the artifact role as `source`, `tutorial`, `analysis artifact`, `generated`, or `unknown` before deciding whether an edit is safe. +5. Grade the proposed work from Tier 0 through Tier 3 and perform only the authorized, protectable tier. +6. Define the smallest coherent scope and apply repository-specific conventions before general preferences. +7. Preserve public interfaces, calculations, numerical behavior, outputs, notebook state, and file formats unless the user explicitly authorizes and protects a change. +8. Run the narrowest relevant validation. Fail closed on unexplained contract changes and report exactly what ran. ### Treat operating mode as a hard boundary @@ -28,6 +30,27 @@ Improve scientific and technical code without obscuring its logic or silently ch - In any non-editing mode, inspect and report inaccurate, stale, unsupported, or missing documentation; explain its impact and location; and propose concrete replacement wording when useful. Do not apply the replacement, edit, rename, reformat, or restructure source files, or run formatters, fixers, or other commands that modify files. Use read-only validation when useful, and verify that the working tree remains unchanged before finishing. - Report a stale or dangerous comment as a high-priority issue when warranted, but do not treat its seriousness as authorization to edit. +## Inventory scientific and artifact contracts + +Before editing, record only the contract items relevant to the target. Consider scientific meaning, units, shapes, schemas, ordering, missing values, state dependencies, public and filesystem interfaces, expected outputs, stored outputs, notebook structure, and notebook role. + +- `evidence-backed`: supported by tests, schemas, types, authoritative code or documentation, or user instructions. Document it precisely. +- `observable-only`: visible in the implementation, but its scientific intent is not established. Describe only the observed operation. +- `unknown`: insufficient evidence exists. Preserve the behavior and request or recommend expert confirmation. + +Do not infer a scientific contract from domain familiarity or an arithmetic pattern. Consult [references/scientific-contracts.md](references/scientific-contracts.md) when units, schemas, ordering, missing values, state, or interface boundaries need detailed treatment. + +## Grade risk before acting + +- **Tier 0 — review/planning only:** make no changes when the request is non-editing, intent is too uncertain, the artifact is generated or unknown, or preservation cannot be verified. +- **Tier 1 — documentation only:** edit comments, docstrings, or Markdown; prune redundant or misleading commentary; document only evidence-backed contracts. Do not change behavior. +- **Tier 2 — local readability refactoring:** refactor only when explicitly requested, locally protectable, and free of Tier-3 effects. Require targeted behavior checks. +- **Tier 3 — protected/high risk:** report by default. This includes numerical evaluation order, public interfaces, notebook structure or identity, execution state, stored outputs, metadata, and artifact contracts. Edit only with explicit authorization and suitable protection. + +Label Tier-3 findings with one or more reasons: `numerical-semantics`, `public-interface`, `notebook-structure`, `execution-state`, `stored-output`, `metadata`, or `artifact-contract`. + +For review findings, report: location, issue, evidence, contract affected, confidence, behavior risk, proposed wording or action, and required expert confirmation. Separate established facts from inference. + ## Adapt to the task ### Write new code @@ -44,6 +67,7 @@ Improve scientific and technical code without obscuring its logic or silently ch - Explain purpose, interpretation, assumptions, and non-obvious choices. - Remove or correct stale, misleading, redundant, and obviously generated-sounding comments when confidence is high. - Leave suspicious logic unchanged and flag it separately when its intended behavior is uncertain. +- Prefer deleting or consolidating stale, obvious, duplicated, or unsupported commentary over adding another explanation. ### Refactor for readability @@ -137,6 +161,8 @@ Maintain distinctions that affect interpretation, including measured versus calc - Avoid stock phrases such as “This function is responsible for,” “This block handles,” “Here, we,” and “It is important to note that.” - Avoid vague praise such as “robust,” “seamless,” “comprehensive,” or “efficient” unless technically defined. - Avoid repeatedly stating “This ensures that” when the consequence is already evident. +- Remove comments that narrate syntax, duplicate nearby Markdown, contradict the code, or bury the important contract. +- Replace or remove an unsupported scientific claim only when repository evidence establishes the correction or the claim is clearly redundant. Otherwise report the ambiguity and require expert confirmation, even when editing is authorized. Consult [references/comment-examples.md](references/comment-examples.md) when concrete wording examples would help. Do not load it when the repository's own conventions already settle the choice. @@ -190,6 +216,19 @@ nanoseconds to seconds” when the units are not documented. ## Apply language-specific conventions +### Jupyter Notebook + +- Treat `.ipynb` as a structured artifact, not ordinary JSON. Classify its role before editing; generated, cached, result, publication, and unknown-role notebooks are normally Tier 0. +- Preserve `nbformat`, `nbformat_minor`, notebook and cell metadata, cell count and order, cell IDs and types, execution counts, outputs, attachments, and untouched source by default. +- Do not execute, clear or regenerate outputs, normalize JSON or metadata, convert to a script, add or remove cells, reorder cells, or repair hidden state unless explicitly requested and protected. +- Change only the minimum existing Markdown or code-cell source needed. Prefer one authoritative explanation instead of duplicated Markdown and comments. +- Require both structural preservation and a focused textual diff. If the editing mechanism would rewrite unrelated serialization, report the limitation and do not edit. +- Report hidden or prior-state dependencies without silently restructuring the notebook. +- When source changes but stored outputs remain, say the notebook was not executed and do not imply that displayed outputs were recomputed. +- Use a dependency-free structural comparison or repository validator when available. A deterministic fixture probe may add evidence, but it does not prove general notebook equivalence. + +Consult [references/jupyter-notebooks.md](references/jupyter-notebooks.md) before editing a notebook or reviewing notebook state, serialization, generated status, or stale outputs. + ### Python - Follow repository formatter and lint settings; otherwise use PEP 8-compatible, Black-like readability. @@ -246,6 +285,8 @@ nanoseconds to seconds” when the units are not documented. Run the narrowest relevant checks available, such as the formatter, linter, type checker, unit or integration tests, syntax compiler, TeX build, ShellCheck, or a targeted smoke test. For documentation-only edits, still check parsing, compilation, or rendered structure when practical. +For notebook edits, validate protected structure and intended source allowances without executing the original notebook. Treat validator errors, ambiguous results, unexpected source changes, and protected-field changes as failures. When the task is repeatable, check that a second identical documentation pass converges to no meaningful diff. + Never claim a check passed unless it ran successfully. If a check cannot run, name it and explain why. Finish with a concise report that states: @@ -255,3 +296,4 @@ Finish with a concise report that states: - whether behavior was intentionally preserved; - which validation commands ran and their outcomes; - any unresolved ambiguity, scientific assumption, or suspicious logic left unchanged. +- for notebooks, the apparent role, whether execution occurred, and whether stored outputs were preserved rather than regenerated. diff --git a/skills/scientific-code-documenter/agents/openai.yaml b/skills/scientific-code-documenter/agents/openai.yaml index f13a5e5..7f13d71 100644 --- a/skills/scientific-code-documenter/agents/openai.yaml +++ b/skills/scientific-code-documenter/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "Scientific Code Documenter" - short_description: "Document and clarify scientific code" - default_prompt: "Use $scientific-code-documenter to improve this scientific code documentation and readability without changing its behavior." + short_description: "Document scientific code and notebooks" + default_prompt: "Use $scientific-code-documenter to improve this scientific code or notebook documentation while preserving its contracts and behavior." diff --git a/skills/scientific-code-documenter/references/comment-examples.md b/skills/scientific-code-documenter/references/comment-examples.md index 7bfbd2e..a010c0b 100644 --- a/skills/scientific-code-documenter/references/comment-examples.md +++ b/skills/scientific-code-documenter/references/comment-examples.md @@ -29,6 +29,20 @@ def absorbance(transmitted, incident): """This function is responsible for taking two arrays and returning an array.""" ``` +## Data contracts and missing values + +Prefer evidence-backed schema and ordering information: + +```python +def export_summary(rows: list[Result]) -> str: + """Serialize `sample_id`, `estimate`, and `status` in input row order. + + Missing estimates remain empty fields; they are not imputed. + """ +``` + +Do not claim that missing values are below detection limits unless repository evidence says so. + ## Logical stages and inline decisions Prefer comments that expose intent or a constraint: @@ -59,6 +73,40 @@ Do not guess at an undocumented correction: corrected = signal - 0.037 * reference ``` +In a review finding, distinguish observation from intent: “The implementation subtracts `0.037 * reference`; the repository does not establish the factor's units or physical basis. Expert confirmation is required before assigning scientific meaning.” + +## Comment pruning and notebook duplication + +Prefer one authoritative notebook explanation: + +```markdown +The next cell preserves acquisition order because the exported table is consumed positionally. +``` + +Remove an adjacent `# Preserve acquisition order for positional export` comment if it adds no local detail. Do not add a third explanation. + +Prune narration: + +```python +# Loop over every row. +for row in rows: + # Append the row. + serialized.append(encode(row)) +``` + +The code is clearer without either comment. Add documentation only if ordering, encoding, or failure behavior is non-obvious and evidence-backed. + +## Unsupported explanations and expert boundaries + +Replace an unsupported claim with neutral wording only when repository evidence establishes that the neutral wording is accurate: + +```python +# Apply the configured scaling factor; its units and scientific interpretation are not documented. +scaled = raw * factor +``` + +If the repository does not establish a safe replacement, report the unsupported claim without changing the file. If the correct interpretation would affect methodology or results, require expert confirmation even when other edits are authorized. + Do not invent provenance or interpretation: ```python diff --git a/skills/scientific-code-documenter/references/jupyter-notebooks.md b/skills/scientific-code-documenter/references/jupyter-notebooks.md new file mode 100644 index 0000000..0648176 --- /dev/null +++ b/skills/scientific-code-documenter/references/jupyter-notebooks.md @@ -0,0 +1,57 @@ +# Jupyter Notebook handling + +Use this reference before editing or reviewing an `.ipynb` file. + +## Classify notebook role + +- `source`: maintained input to the analysis workflow. +- `tutorial`: intentionally explanatory and usually editable within its teaching contract. +- `analysis artifact`: committed state or results whose stored outputs may be evidentiary. +- `generated`: produced by a pipeline, exporter, benchmark, or cache; normally review only. +- `unknown`: insufficient repository evidence; default to review only. + +Repository documentation, generators, build rules, paths, headers, and version-control policy are stronger evidence than the mere fact that a notebook is committed. + +## Protected fields + +Preserve by default: + +- `nbformat` and `nbformat_minor`; +- notebook metadata; +- cell count, order, IDs, types, and metadata; +- execution counts and code-cell outputs; +- Markdown attachments; +- untouched cell source; +- source representation, JSON formatting, key order, Unicode escaping, and trailing-newline behavior where the editing mechanism permits. + +Unexpected changes are hard failures. Do not waive them because the notebook still opens. + +## Choose documentation location + +- Use existing Markdown for workflow purpose, scientific context shared by several cells, and state dependencies visible to readers. +- Use code comments for a local numerical convention, ordering constraint, or implementation safeguard. +- Avoid repeating the same explanation in adjacent Markdown and code. +- Prune narration that merely restates the next statement. + +## Hidden state and stale outputs + +Report reliance on prior-cell namespace, mutable globals, working directories, files, environment variables, random state, package versions, display hooks, or services. Do not silently reorder or merge cells to remove the dependency. + +When source is edited without execution, preserve outputs and execution counts. Disclose that the notebook was not executed and that displayed outputs were not recomputed from the modified source. + +## Serialization locality + +Use a structure-aware, locality-preserving editing mechanism. A small source edit should have a small diff. If a tool would normalize the whole JSON document, metadata, source arrays, or escaping, stop and report the limitation. + +Validate both: + +1. semantic structural preservation; and +2. textual diff locality. + +## Generated and review-only notebooks + +Do not edit a generated, cached, benchmark, result, publication, or unknown-role notebook without explicit source-edit authorization and a protection plan. Report the finding and direct changes to the authoritative source or generator when known. + +## Behavioral probes + +Do not execute the original notebook by default. Repository-designed fixture code may be extracted and run in an isolated deterministic probe to compare limited numerical, ordering, missing-value, or serialization behavior. Describe this accurately as additional fixture evidence, never as general notebook equivalence. diff --git a/skills/scientific-code-documenter/references/scientific-contracts.md b/skills/scientific-code-documenter/references/scientific-contracts.md new file mode 100644 index 0000000..c95c66a --- /dev/null +++ b/skills/scientific-code-documenter/references/scientific-contracts.md @@ -0,0 +1,62 @@ +# Scientific contract inventory + +Use this reference when documentation depends on data meaning, units, shapes, schemas, ordering, missing values, state, or interfaces. Record only items relevant to the requested scope. + +## Evidence classes + +### Evidence-backed + +Repository evidence establishes the contract: + +```python +def absorbance(intensity: np.ndarray) -> np.ndarray: + """Return dimensionless absorbance in the input wavelength order.""" +``` + +Use this wording only when tests, schemas, labels, types, documentation, or authoritative nearby code establish both the quantity and ordering. + +### Observable-only + +The operation is visible, but the scientific intent is not: + +```python +# Preserve the configured scaling operation; the source and target units are not documented. +scaled = signal * scale_factor +``` + +Do not turn multiplication by a familiar factor into an unsupported unit conversion. + +### Unknown + +No reliable contract can be established. Preserve the behavior, state the gap, and request expert confirmation. Do not invent units, physical roles, value ranges, sentinel meanings, or provenance. + +## Contract checklist + +- **Scientific meaning:** measured, calculated, fitted, simulated, raw, processed, absolute, or relative only when established. +- **Units:** record source and target units and the conversion point. If either is unknown, document only the operation. +- **Shapes:** include rank, axis order, and alignment rules where evidence supports them. +- **Schemas:** preserve column names, types, optional fields, and serialization order. +- **Ordering:** preserve row, column, iteration, and output ordering when positional consumers or tests depend on it. +- **Missing values:** distinguish rejection, propagation, imputation, and sentinel handling only from evidence. +- **State dependencies:** identify required prior calls, cells, mutable globals, seeds, files, environment variables, and working directories. +- **Interfaces:** preserve public symbols, exceptions, files, paths, formats, side effects, and output artifacts. + +## Safe wording patterns + +```python +"""Return values with the same shape and row order as `samples`.""" +``` + +Use when shape and ordering are tested. + +```python +# Values equal to the configured sentinel follow the existing exclusion branch. +``` + +Use when the effect is observable but the sentinel's scientific meaning is unknown. + +```python +# Expert confirmation required: the repository does not identify this factor's units or provenance. +``` + +Use in a review finding, not as a noisy inline comment, when no safe code wording is needed. From 12168ca5a33fd08257d334e00ead4e422a425ecf Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 12:50:40 +0800 Subject: [PATCH 02/17] test: add notebook contract evaluation --- evals/scientific-code-documenter/README.md | 19 +- evals/scientific-code-documenter/checks.sh | 11 +- .../fixtures/contracts/test_latex_fixtures.py | 30 +- .../fixtures/cross-language-contracts.md | 46 ++ .../editable_scientific.contract.json | 43 ++ .../notebook/editable_scientific.ipynb | 73 +++ .../notebook/editable_scientific.prompt.txt | 1 + .../generated_pipeline_report.contract.json | 7 + .../notebook/generated_pipeline_report.ipynb | 44 ++ .../generated_pipeline_report.prompt.txt | 1 + .../notebook/over_commented.contract.json | 25 + .../fixtures/notebook/over_commented.ipynb | 41 ++ .../notebook/over_commented.prompt.txt | 1 + .../review_only_stale_claim.contract.json | 7 + .../notebook/review_only_stale_claim.ipynb | 36 ++ .../review_only_stale_claim.prompt.txt | 1 + .../notebook/test_notebook_contract.py | 298 ++++++++++ .../notebook/validate_notebook_contract.py | 544 ++++++++++++++++++ evals/scientific-code-documenter/prompts.yaml | 80 +++ .../results/result.template.md | 22 + evals/scientific-code-documenter/rubric.md | 27 +- 21 files changed, 1339 insertions(+), 18 deletions(-) create mode 100644 evals/scientific-code-documenter/fixtures/notebook/editable_scientific.contract.json create mode 100644 evals/scientific-code-documenter/fixtures/notebook/editable_scientific.ipynb create mode 100644 evals/scientific-code-documenter/fixtures/notebook/editable_scientific.prompt.txt create mode 100644 evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.contract.json create mode 100644 evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.ipynb create mode 100644 evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.prompt.txt create mode 100644 evals/scientific-code-documenter/fixtures/notebook/over_commented.contract.json create mode 100644 evals/scientific-code-documenter/fixtures/notebook/over_commented.ipynb create mode 100644 evals/scientific-code-documenter/fixtures/notebook/over_commented.prompt.txt create mode 100644 evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.contract.json create mode 100644 evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.ipynb create mode 100644 evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.prompt.txt create mode 100644 evals/scientific-code-documenter/fixtures/notebook/test_notebook_contract.py create mode 100644 evals/scientific-code-documenter/fixtures/notebook/validate_notebook_contract.py diff --git a/evals/scientific-code-documenter/README.md b/evals/scientific-code-documenter/README.md index 6a775d9..6fe6b6b 100644 --- a/evals/scientific-code-documenter/README.md +++ b/evals/scientific-code-documenter/README.md @@ -1,16 +1,17 @@ # Scientific Code Documenter evaluation -This suite tests whether the skill improves documentation and readability without changing behavior or expanding scope. Tracked fixtures are synthetic and contain no unpublished research or third-party project code. Authorized real-repository runs retain raw source, diffs, and logs only in ignored local result directories; tracked summaries are sanitized. +This suite tests whether the skill improves documentation and readability without changing behavior or expanding scope. It includes contract-aware Jupyter Notebook cases without treating notebook JSON as ordinary text. Tracked fixtures are synthetic and contain no unpublished research or third-party project code. Authorized real-repository runs retain raw source, diffs, and logs only in ignored local result directories; tracked summaries are sanitized. ## Evaluation layers 1. Run collection structural validation to check frontmatter, naming, references, and portable runtime content. 2. Use `prompts.yaml` to test explicit invocation, positive implicit triggering, false triggers, and boundary decisions in fresh sessions. -3. Run `./scripts/run_fixture_checks.sh scientific-code-documenter` before and after each fixture edit. +3. Run `./scripts/run_fixture_checks.sh scientific-code-documenter` before and after each fixture edit. Notebook candidates must also pass their fixture-specific contract file. 4. Compare baseline and skill-assisted diffs from identical starting commits using the same model and prompt. 5. Score both outputs with `rubric.md`; preserve raw diffs and command logs under `results/`. -6. Exercise adversarial fixtures to confirm that unknown meaning is not invented and generated or vendored files remain untouched. -7. After synthetic acceptance, repeat review-only and limited-edit prompts on real repositories whose owners permit the work. +6. Exercise adversarial and notebook fixtures to confirm that unknown meaning is not invented and generated, review-only, or protected notebook state remains untouched. +7. Rerun the same skill prompt against every accepted editing candidate. The second pass must create no tracked diff; notebook bytes must be identical to pass 1. +8. After synthetic acceptance, repeat review-only and limited-edit prompts on real repositories whose owners explicitly permit the work. Structural success is necessary but does not establish trigger accuracy or effectiveness. Record those claims only after the full prompt matrix and controlled comparisons have been completed. @@ -21,6 +22,7 @@ Structural success is necessary but does not establish trigger accuracy or effec - `shell/`: a valid workflow with quoting and working-directory issues plus a long scientific-computing command. - `latex/`: a compilable report with scattered macros, long lines, inconsistent float structure, and an unexplained workaround. - `adversarial/`: trivial code, unknown scientific meaning, a protected unusual threshold, and generated or vendored files that must not be edited. +- `notebook/`: editable, review-only, over-commented, and generated notebooks plus a dependency-free structural/locality validator and a narrow isolated synthetic probe. Baseline fixtures are intentionally imperfect. A good edit may change their style defects but must keep executable checks and protected outputs passing. @@ -29,14 +31,21 @@ Baseline fixtures are intentionally imperfect. A good edit may change their styl - Create sibling worktrees outside the main checkout at the same starting commit. - Start fresh Codex sessions with the same model and prompt. - Run the baseline without explicit invocation and the comparison with `$scientific-code-documenter`. +- Compare the four notebook cases and representative Python, Shell, LaTeX, and HTML cases. Count the 75% win threshold across editing fixtures; both notebook editing fixtures must beat baseline. +- Use write-enabled workspaces for review-only and generated pairs so restraint is demonstrated rather than enforced by permissions. These cases may tie, but the skill must not lose or modify a byte. - Save each diff and validation log, then score both independently. +- Validate every accepted skill editing candidate again after a fresh second pass; retain the pass-2 raw log and zero-diff evidence. - Record the model, date, commit, fixture, trigger result, scores, validation outcome, strengths, regressions, and follow-up work. +Notebook candidates are invalid before qualitative scoring if any protected field, output, attachment, execution count, non-allowlisted source, or byte outside an allowlisted source value changes. Validator errors and ambiguous comparisons also fail closed. The deterministic fixture probe executes only selected trusted synthetic code cells in isolated Python processes. It does not run a notebook, provide a security sandbox, or prove general notebook equivalence. + Exact repository-specific commands are maintained in the root `README.md` after the shared scripts and commits exist. ## Real-repository acceptance -Completed acceptance now covers an authorized scientific Python repository and a production-style ORCA Shell workflow repository. The ORCA run reviewed nine scripts, exercised three controlled edit pairs with isolated mocks and dry runs, and preserved generated calculation inputs and workflow contracts without launching an expensive calculation. See the [v0.3.0 sanitized summary](../../docs/evaluations/scientific-code-documenter-v0.3.0.md). +Historical acceptance covers an authorized scientific Python repository and a production-style ORCA Shell workflow repository. The ORCA run reviewed nine scripts, exercised three controlled edit pairs with isolated mocks and dry runs, and preserved generated calculation inputs and workflow contracts without launching an expensive calculation. See the [v0.3.0 sanitized summary](../../docs/evaluations/scientific-code-documenter-v0.3.0.md). Those runs remain evidence for the predecessor runtime, not acceptance of the contract-aware v0.4 runtime. + +Real-notebook acceptance for v0.4 remains pending until a notebook owner explicitly authorizes an immutable review and controlled edit candidate. Do not substitute a personal or research notebook merely because it is locally available. Run real-repository candidates only from immutable snapshots, never from the live source tree. Protect and restore tracked source state, and determine validity from the frozen runtime, source inputs, prompts, harnesses, and captured outputs. Ignored calculation outputs and filesystem metadata remain outside that boundary unless explicitly frozen; record concurrent external activity separately without treating live-source quiescence as an evaluation gate. diff --git a/evals/scientific-code-documenter/checks.sh b/evals/scientific-code-documenter/checks.sh index c776889..1167939 100755 --- a/evals/scientific-code-documenter/checks.sh +++ b/evals/scientific-code-documenter/checks.sh @@ -19,6 +19,8 @@ run_required() { run_required "Python numerical behavior" \ python3 -m unittest discover -s "$eval_root/fixtures/python" -p 'test_*.py' +run_required "Notebook contract validator" \ + python3 -B -m unittest discover -s "$eval_root/fixtures/notebook" -p 'test_*.py' run_required "HTML and CSS fixture integrity" \ python3 "$eval_root/fixtures/html/validate_html.py" "$eval_root/fixtures/html/index.html" "$eval_root/fixtures/html/styles.css" run_required "Shell syntax" \ @@ -27,8 +29,13 @@ run_required "Adversarial protections" \ python3 -m unittest discover -s "$eval_root/fixtures/adversarial" -p 'test_*.py' run_required "Cross-language Shell contracts" \ python3 "$eval_root/fixtures/contracts/test_shell_fixtures.py" -run_required "Cross-language LaTeX contracts" \ - python3 "$eval_root/fixtures/contracts/test_latex_fixtures.py" +if command -v pdflatex >/dev/null 2>&1; then + run_required "Cross-language LaTeX contracts" \ + python3 "$eval_root/fixtures/contracts/test_latex_fixtures.py" +else + run_required "Cross-language LaTeX static contracts" \ + python3 "$eval_root/fixtures/contracts/test_latex_fixtures.py" static +fi run_required "Cross-language HTML contracts" \ python3 "$eval_root/fixtures/contracts/test_html_fixtures.py" diff --git a/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py b/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py index daf37d2..cedbf76 100644 --- a/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py +++ b/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py @@ -98,7 +98,7 @@ def labels_from_aux(aux_path: Path) -> set[str]: return set(re.findall(r"\\newlabel\{([^}]+)\}", aux_path.read_text(encoding="utf-8"))) -def validate_main(source: Path) -> None: +def validate_main(source: Path, *, compile_output: bool = True) -> None: require(source.is_file(), f"missing fixture: {source}") text = source.read_text(encoding="utf-8") compact = normalized_source(text) @@ -143,6 +143,10 @@ def validate_main(source: Path) -> None: require("Sample & Energy & Dipole & Orbital surface & Density surface" in compact, "descriptor columns changed") require(r"Sample & $E_{\mathrm{pa}}$ & $E_{\mathrm{pc}}$ & $R_{\mathrm{s}}$ & $R_{\mathrm{ct}}$ & $n$" in compact, "electrochemical columns changed") + if not compile_output: + print("PASS LaTeX main static contract") + return + pdf, aux, _, temporary = compile_tex(source, appendix=False, expected_pages=2) try: require(labels_from_aux(aux) == MAIN_LABELS, "non-appendix AUX labels changed") @@ -166,7 +170,7 @@ def validate_main(source: Path) -> None: print("PASS LaTeX main contract") -def validate_restraint(source: Path) -> None: +def validate_restraint(source: Path, *, compile_output: bool = True) -> None: require(source.is_file(), f"missing fixture: {source}") initial_bytes = source.read_bytes() require(sha256_bytes(initial_bytes) == FRAGILE_SHA256, "fragile source hash changed") @@ -192,6 +196,11 @@ def validate_restraint(source: Path) -> None: require("source and target units are not established" in compact, "factor ambiguity warning changed") require("physical interpretation is deliberately unspecified" in compact, "correction ambiguity warning changed") + if not compile_output: + require(source.read_bytes() == initial_bytes, "fragile source changed during validation") + print("PASS LaTeX restraint static contract") + return + pdf, _, _, temporary = compile_tex(source, appendix=False, expected_pages=1) try: visible = extract_normalized_text(pdf) @@ -206,11 +215,18 @@ def validate_restraint(source: Path) -> None: def main() -> int: mode = sys.argv[1] if len(sys.argv) > 1 else "all" explicit_path = Path(sys.argv[2]).resolve() if len(sys.argv) > 2 else None - if mode in {"all", "main"}: - validate_main(explicit_path or FIXTURE_ROOT / "latex/crowded_scientific_report.tex") - if mode in {"all", "restraint"}: - validate_restraint(explicit_path or FIXTURE_ROOT / "latex/fragile_template_snippet.tex") - if mode not in {"all", "main", "restraint"}: + compile_output = mode != "static" + if mode in {"all", "main", "static"}: + validate_main( + explicit_path or FIXTURE_ROOT / "latex/crowded_scientific_report.tex", + compile_output=compile_output, + ) + if mode in {"all", "restraint", "static"}: + validate_restraint( + explicit_path or FIXTURE_ROOT / "latex/fragile_template_snippet.tex", + compile_output=compile_output, + ) + if mode not in {"all", "main", "restraint", "static"}: raise SystemExit(f"unknown mode: {mode}") return 0 diff --git a/evals/scientific-code-documenter/fixtures/cross-language-contracts.md b/evals/scientific-code-documenter/fixtures/cross-language-contracts.md index 8120b7a..24af96d 100644 --- a/evals/scientific-code-documenter/fixtures/cross-language-contracts.md +++ b/evals/scientific-code-documenter/fixtures/cross-language-contracts.md @@ -110,3 +110,49 @@ preferred implementation or formatting style. `data-row-order` values, and the exact JSON byte payload. - Validation: source SHA-256, HTML parsing, exact structural and embedded-data assertions, and post-review byte identity. + +## Jupyter: `editable_scientific.ipynb` + +- Mode: Tier-1 documentation editing. +- Role: source notebook. +- Editable: only the allowlisted `source` fields of `contract-overview` and + `transform-records`. +- Scientific contract: input record order and `None` missing values remain + unchanged. `SCALE_FACTOR` has no established units or scientific meaning. +- Protected state: notebook and cell metadata, formats, cell count and order, + cell IDs and types, execution count 7, stored output, attachment, raw cell, + and every non-allowlisted source field. +- Validation: exact original hash, raw-byte locality outside changed source + values, Python AST equality after stripping docstrings, and an isolated + synthetic probe of numerical values, missing values, and serialized order. + The probe is not notebook execution or a general equivalence proof. + +## Jupyter: `review_only_stale_claim.ipynb` + +- Mode: Tier-0 review only. +- Role: analysis artifact. +- Editable: no; the notebook must remain byte-identical. +- Ambiguity: the stated time-unit conversion is not established by the code + and appears inconsistent with the configured arithmetic. Report the claim + with evidence and request expert confirmation; do not edit it. +- Validation: strict JSON parsing, exact original hash, and byte identity. + +## Jupyter: `over_commented.ipynb` + +- Mode: Tier-1 comment and Markdown pruning. +- Role: tutorial notebook. +- Editable: only the allowlisted source fields of `workflow-narration` and + `summarize-values`. +- Contract: remove or consolidate narration without changing executable AST, + stored output, execution state, notebook structure, metadata, or unrelated + serialization. +- Validation: exact original hash, changed-line budgets, Python AST equality, + protected-field comparison, and raw-byte locality. + +## Jupyter: `generated_pipeline_report.ipynb` + +- Mode: Tier-0 review only. +- Role: generated notebook with explicit generator provenance. +- Editable: no; direct changes belong in the upstream generator and the + notebook must remain byte-identical. +- Validation: strict JSON parsing, exact original hash, and byte identity. diff --git a/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.contract.json b/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.contract.json new file mode 100644 index 0000000..127f9b1 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.contract.json @@ -0,0 +1,43 @@ +{ + "schema_version": 1, + "original_sha256": "114daea3eb4d15611bd66414e879f12cc20e28a90d949be1ab7d3eecf1919707", + "notebook_role": "source", + "policy": "allowlisted-source-edits", + "allowed_source_changes": [ + { + "cell_index": 0, + "cell_id": "contract-overview", + "cell_type": "markdown", + "edit_kind": "markdown", + "max_changed_lines": 8 + }, + { + "cell_index": 1, + "cell_id": "transform-records", + "cell_type": "code", + "edit_kind": "python-comments-docstrings", + "max_changed_lines": 6 + } + ], + "locality": { + "max_changed_lines_total": 14 + }, + "behavior_probe": { + "selected_code_cells": [ + { + "cell_index": 1, + "cell_id": "transform-records" + } + ], + "callable": "notebook_contract_probe", + "timeout_seconds": 5, + "expected_json": { + "values": [ + 0.002, + null, + 0.001 + ], + "serialized_rows": "[[\"beta\",0.002],[\"alpha\",null],[\"gamma\",0.001]]" + } + } +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.ipynb b/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.ipynb new file mode 100644 index 0000000..8626d2f --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.ipynb @@ -0,0 +1,73 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "contract-overview", + "metadata": { + "tags": [ + "documentation-target" + ] + }, + "source": "# Scaling summary\n\nThe ordered records are transformed below." + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "transform-records", + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "text/plain": "{'values': [0.002, None, 0.001], 'serialized_rows': '[[\"beta\",0.002],[\"alpha\",null],[\"gamma\",0.001]]'}" + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": "SCALE_FACTOR = 1e-3\nINPUT_ROWS = [(\"beta\", 2.0), (\"alpha\", None), (\"gamma\", 1.0)]\n\ndef notebook_contract_probe():\n values = [None if value is None else value * SCALE_FACTOR for _, value in INPUT_ROWS]\n rows = [[label, scaled] for (label, _), scaled in zip(INPUT_ROWS, values)]\n import json\n return {\"values\": values, \"serialized_rows\": json.dumps(rows, separators=(\",\", \":\"))}\n" + }, + { + "attachments": { + "scale-note.txt": { + "text/plain": "Synthetic fixture attachment; preserve exactly." + } + }, + "cell_type": "markdown", + "id": "output-note", + "metadata": { + "tags": [ + "stored-output-note" + ] + }, + "source": "The displayed result is stored output from execution count 7." + }, + { + "cell_type": "raw", + "id": "provenance-note", + "metadata": { + "format": "text/plain" + }, + "source": "Synthetic public-safe fixture; no research data." + } + ], + "metadata": { + "data_contract": { + "missing_value": null, + "ordering": "input order" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.prompt.txt b/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.prompt.txt new file mode 100644 index 0000000..bc6f9b6 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.prompt.txt @@ -0,0 +1 @@ +Document the notebook's evidence-backed inputs, outputs, ordering, and missing-value behavior. Add only concise Markdown, comments, or docstrings to the existing allowlisted cells. Do not infer units or a scientific interpretation for SCALE_FACTOR, and do not execute, clear, regenerate, reorder, or otherwise normalize the notebook. diff --git a/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.contract.json b/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.contract.json new file mode 100644 index 0000000..ade610e --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.contract.json @@ -0,0 +1,7 @@ +{ + "schema_version": 1, + "original_sha256": "72e5a312bb04c6ab02724ec4cb611b882d37763d7aafd5993014f2ff6389462a", + "notebook_role": "generated", + "policy": "byte-identical", + "allowed_source_changes": [] +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.ipynb b/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.ipynb new file mode 100644 index 0000000..1b8a5dd --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.ipynb @@ -0,0 +1,44 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "generated-banner", + "metadata": { + "editable": false + }, + "source": "# Generated pipeline report\n\nThis notebook is regenerated by `tools/build_pipeline_report.py`." + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "generated-summary", + "metadata": { + "editable": false + }, + "outputs": [ + { + "data": { + "text/plain": "{'records': 3}" + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": "summary = {\"records\": 3}\nsummary\n" + } + ], + "metadata": { + "generated": { + "generator": "tools/build_pipeline_report.py", + "source_revision": "synthetic-001" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.prompt.txt b/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.prompt.txt new file mode 100644 index 0000000..56dfbe9 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.prompt.txt @@ -0,0 +1 @@ +Review the clarity of this generated notebook's documentation without modifying or executing it. Identify the upstream generator as the only appropriate edit target. The notebook must remain byte-identical. diff --git a/evals/scientific-code-documenter/fixtures/notebook/over_commented.contract.json b/evals/scientific-code-documenter/fixtures/notebook/over_commented.contract.json new file mode 100644 index 0000000..85b5d9a --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/over_commented.contract.json @@ -0,0 +1,25 @@ +{ + "schema_version": 1, + "original_sha256": "dcab864eaee5cdd365147f41e5e11a3e91888416060595d67a395664df3a6d9d", + "notebook_role": "tutorial", + "policy": "allowlisted-source-edits", + "allowed_source_changes": [ + { + "cell_index": 0, + "cell_id": "workflow-narration", + "cell_type": "markdown", + "edit_kind": "markdown", + "max_changed_lines": 6 + }, + { + "cell_index": 1, + "cell_id": "summarize-values", + "cell_type": "code", + "edit_kind": "python-comments-docstrings", + "max_changed_lines": 12 + } + ], + "locality": { + "max_changed_lines_total": 18 + } +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/over_commented.ipynb b/evals/scientific-code-documenter/fixtures/notebook/over_commented.ipynb new file mode 100644 index 0000000..8a7b28a --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/over_commented.ipynb @@ -0,0 +1,41 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "workflow-narration", + "metadata": {}, + "source": "# Summary\n\nThe next cell creates a list, loops over every value, checks whether each value is missing, and appends the non-missing values before sorting them." + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "summarize-values", + "metadata": { + "tags": [ + "documentation-target" + ] + }, + "outputs": [ + { + "data": { + "text/plain": "[1.0, 3.0]" + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": "def summarize(values):\n # Create an empty list.\n cleaned = []\n # Loop over each value.\n for value in values:\n # Check if the value is not None.\n if value is not None:\n # Append the value.\n cleaned.append(value)\n # Sort the list.\n cleaned.sort()\n # Return the list.\n return cleaned\n\nsummarize([3.0, None, 1.0])\n" + } + ], + "metadata": { + "fixture_mode": "comment-pruning", + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/over_commented.prompt.txt b/evals/scientific-code-documenter/fixtures/notebook/over_commented.prompt.txt new file mode 100644 index 0000000..6a93900 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/over_commented.prompt.txt @@ -0,0 +1 @@ +Prune or consolidate the Markdown and comments that merely narrate syntax. Preserve the function, call, stored output, execution state, metadata, cell structure, and serialization outside the two allowlisted source fields. Do not execute the notebook or add replacement commentary unless it records a real contract. diff --git a/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.contract.json b/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.contract.json new file mode 100644 index 0000000..a14d168 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.contract.json @@ -0,0 +1,7 @@ +{ + "schema_version": 1, + "original_sha256": "bfa6ca039930bcc922084d801ade9f6456fbcb548309034ec7894cbff97a6fd7", + "notebook_role": "analysis artifact", + "policy": "byte-identical", + "allowed_source_changes": [] +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.ipynb b/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.ipynb new file mode 100644 index 0000000..016ad71 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.ipynb @@ -0,0 +1,36 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "stale-claim", + "metadata": { + "review_state": "unverified" + }, + "source": "# Signal preparation\n\nMultiplying the signal by `1e-3` converts seconds to milliseconds." + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "scale-signal", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": "[0.001, 0.002]\n" + } + ], + "source": "signal = [1.0, 2.0]\nscaled = [value * 1e-3 for value in signal]\nprint(scaled)\n" + } + ], + "metadata": { + "fixture_mode": "review-only", + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.prompt.txt b/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.prompt.txt new file mode 100644 index 0000000..3bec698 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.prompt.txt @@ -0,0 +1 @@ +Review this notebook's scientific documentation. The scaling explanation may be stale or unsupported. Report the issue, evidence, affected contract, confidence, risk, proposed action, and whether expert confirmation is needed. Do not modify or execute the notebook; it must remain byte-identical. diff --git a/evals/scientific-code-documenter/fixtures/notebook/test_notebook_contract.py b/evals/scientific-code-documenter/fixtures/notebook/test_notebook_contract.py new file mode 100644 index 0000000..a1154f9 --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/test_notebook_contract.py @@ -0,0 +1,298 @@ +from __future__ import annotations + +import copy +import hashlib +import json +from pathlib import Path +import unittest + +import validate_notebook_contract as validator + + +FIXTURE_ROOT = Path(__file__).resolve().parent + + +def load_contract(stem: str) -> dict: + return json.loads((FIXTURE_ROOT / f"{stem}.contract.json").read_text(encoding="utf-8")) + + +def replace_value(data: bytes, path: tuple, value: object) -> bytes: + text = data.decode("utf-8") + spans = validator._SpanScanner(text).scan() + start, end = spans[path] + token = json.dumps(value, ensure_ascii=False, separators=(",", ":")) + return (text[:start] + token + text[end:]).encode("utf-8") + + +def source_at(data: bytes, index: int) -> str: + notebook = json.loads(data) + source = notebook["cells"][index]["source"] + return source if isinstance(source, str) else "".join(source) + + +class NotebookContractTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.editable = (FIXTURE_ROOT / "editable_scientific.ipynb").read_bytes() + cls.editable_contract = load_contract("editable_scientific") + cls.review = (FIXTURE_ROOT / "review_only_stale_claim.ipynb").read_bytes() + cls.review_contract = load_contract("review_only_stale_claim") + cls.over_commented = (FIXTURE_ROOT / "over_commented.ipynb").read_bytes() + cls.over_commented_contract = load_contract("over_commented") + cls.generated = (FIXTURE_ROOT / "generated_pipeline_report.ipynb").read_bytes() + cls.generated_contract = load_contract("generated_pipeline_report") + + def assert_valid(self, result: dict) -> None: + self.assertTrue(result["valid"], result["violations"]) + + def assert_category(self, result: dict, category: str) -> None: + categories = {item["category"] for item in result["violations"]} + self.assertIn(category, categories, result) + + def documented_editable_candidate(self) -> bytes: + markdown = ( + "# Scaling summary\n\n" + "The transformation preserves input order and propagates missing values. " + "The units and scientific meaning of `SCALE_FACTOR` are not documented." + ) + candidate = replace_value(self.editable, ("cells", 0, "source"), markdown) + code = source_at(candidate, 1).replace( + "def notebook_contract_probe():\n", + "def notebook_contract_probe():\n" + " \"\"\"Return ordered transformed values and their stable JSON form.\"\"\"\n" + " # Preserve missing values rather than applying the configured scale.\n", + ) + return replace_value(candidate, ("cells", 1, "source"), code) + + def test_allowlisted_documentation_edit_passes_with_probe(self) -> None: + result = validator.validate_bytes( + self.editable, + self.documented_editable_candidate(), + self.editable_contract, + ) + self.assert_valid(result) + self.assertEqual(result["probe"]["status"], "pass") + self.assertEqual( + [item["cell_id"] for item in result["intended_source_changes"]], + ["contract-overview", "transform-records"], + ) + + def test_comment_pruning_passes(self) -> None: + markdown = "# Summary\n\nMissing values are omitted and the remaining values are sorted in ascending order." + candidate = replace_value(self.over_commented, ("cells", 0, "source"), markdown) + code = "".join( + line + for line in source_at(candidate, 1).splitlines(keepends=True) + if not line.lstrip().startswith("#") + ) + candidate = replace_value(candidate, ("cells", 1, "source"), code) + result = validator.validate_bytes( + self.over_commented, + candidate, + self.over_commented_contract, + ) + self.assert_valid(result) + + def test_review_only_copy_is_byte_identical(self) -> None: + self.assert_valid( + validator.validate_bytes(self.review, self.review, self.review_contract) + ) + + def test_generated_copy_is_byte_identical(self) -> None: + self.assert_valid( + validator.validate_bytes(self.generated, self.generated, self.generated_contract) + ) + + def test_review_only_change_fails(self) -> None: + candidate = self.review.replace(b"Signal preparation", b"Signal processing") + result = validator.validate_bytes(self.review, candidate, self.review_contract) + self.assert_category(result, "serialization-locality") + + def test_malformed_candidate_fails_closed(self) -> None: + result = validator.validate_bytes(self.editable, b"{", self.editable_contract) + self.assert_category(result, "parseability") + + def test_duplicate_candidate_key_fails_closed(self) -> None: + candidate = self.editable.replace( + b'"nbformat": 4,', b'"nbformat": 4,\n "nbformat": 4,', 1 + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "parseability") + + def test_nonstandard_json_constant_fails_closed(self) -> None: + candidate = self.editable.replace( + b'"nbformat": 4,', b'"extension": NaN,\n "nbformat": 4,', 1 + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "parseability") + + def test_top_level_format_change_is_structural(self) -> None: + candidate = replace_value(self.editable, ("nbformat_minor",), 4) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "structural") + + def test_notebook_metadata_change_fails(self) -> None: + candidate = replace_value( + self.editable, + ("metadata", "data_contract", "ordering"), + "sorted order", + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "metadata") + + def test_cell_identifier_change_is_structural(self) -> None: + candidate = replace_value(self.editable, ("cells", 1, "id"), "renamed") + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "structural") + + def test_cell_type_change_is_structural(self) -> None: + candidate = replace_value(self.editable, ("cells", 3, "cell_type"), "markdown") + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "structural") + + def test_cell_order_change_is_structural(self) -> None: + notebook = json.loads(self.editable) + notebook["cells"][2], notebook["cells"][3] = notebook["cells"][3], notebook["cells"][2] + candidate = replace_value(self.editable, ("cells",), notebook["cells"]) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "structural") + + def test_cell_count_change_is_structural(self) -> None: + notebook = json.loads(self.editable) + candidate = replace_value(self.editable, ("cells",), notebook["cells"][:-1]) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "structural") + + def test_cell_metadata_change_fails(self) -> None: + candidate = replace_value(self.editable, ("cells", 1, "metadata", "collapsed"), True) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "metadata") + + def test_execution_count_change_fails(self) -> None: + candidate = replace_value(self.editable, ("cells", 1, "execution_count"), 8) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "execution-count") + + def test_output_change_fails(self) -> None: + candidate = replace_value( + self.editable, + ("cells", 1, "outputs", 0, "execution_count"), + 8, + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "output") + + def test_attachment_change_fails(self) -> None: + candidate = replace_value( + self.editable, + ("cells", 2, "attachments", "scale-note.txt", "text/plain"), + "changed", + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "attachment") + + def test_unknown_top_level_extension_is_protected(self) -> None: + candidate = self.editable.replace( + b' "nbformat": 4,', b' "custom_extension": {"enabled":true},\n "nbformat": 4,', 1 + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "protected-field") + + def test_nonallowlisted_source_change_fails(self) -> None: + candidate = replace_value( + self.editable, + ("cells", 2, "source"), + "This unlisted cell changed.", + ) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "unrelated-source") + + def test_source_container_representation_change_fails(self) -> None: + source = source_at(self.editable, 0) + candidate = replace_value(self.editable, ("cells", 0, "source"), [source]) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "serialization-locality") + + def test_whole_notebook_reserialization_fails_locality(self) -> None: + candidate = json.dumps(json.loads(self.editable), separators=(",", ":")).encode("utf-8") + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "serialization-locality") + + def test_unchanged_source_with_different_escape_fails_locality(self) -> None: + original_token = b'"# Scaling summary\\n\\nThe ordered records are transformed below."' + replacement = b'"# Scaling summary\\u000a\\u000aThe ordered records are transformed below."' + candidate = self.editable.replace(original_token, replacement, 1) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "serialization-locality") + + def test_source_edit_line_budget_is_enforced(self) -> None: + markdown = source_at(self.editable, 0) + "\n" + "\n".join(f"extra {i}" for i in range(20)) + candidate = replace_value(self.editable, ("cells", 0, "source"), markdown) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "serialization-locality") + + def test_numerical_change_fails_ast_and_probe(self) -> None: + code = source_at(self.editable, 1).replace("value * SCALE_FACTOR", "value + SCALE_FACTOR") + candidate = replace_value(self.editable, ("cells", 1, "source"), code) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "protected-field") + self.assert_category(result, "behavior") + + def test_call_order_change_fails_ast(self) -> None: + code = source_at(self.editable, 1).replace( + "json.dumps(rows, separators=(\",\", \":\"))", + "json.dumps(rows, separators=(\",\", \":\"), sort_keys=True)", + ) + candidate = replace_value(self.editable, ("cells", 1, "source"), code) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "protected-field") + + def test_probe_exception_fails(self) -> None: + code = source_at(self.editable, 1).replace( + "def notebook_contract_probe():\n", + "def notebook_contract_probe():\n raise RuntimeError(\"synthetic\")\n", + ) + candidate = replace_value(self.editable, ("cells", 1, "source"), code) + result = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_category(result, "behavior") + + def test_probe_timeout_fails(self) -> None: + contract = copy.deepcopy(self.editable_contract) + contract["behavior_probe"]["timeout_seconds"] = 0.1 + code = source_at(self.editable, 1).replace( + "def notebook_contract_probe():\n", + "def notebook_contract_probe():\n while True:\n pass\n", + ) + candidate = replace_value(self.editable, ("cells", 1, "source"), code) + result = validator.validate_bytes(self.editable, candidate, contract) + self.assert_category(result, "behavior") + + def test_stale_original_hash_is_an_input_error(self) -> None: + contract = copy.deepcopy(self.editable_contract) + contract["original_sha256"] = "0" * 64 + with self.assertRaises(validator.InputError): + validator.validate_bytes(self.editable, self.editable, contract) + + def test_stale_selector_is_an_input_error(self) -> None: + contract = copy.deepcopy(self.editable_contract) + contract["allowed_source_changes"][0]["cell_id"] = "stale-id" + with self.assertRaises(validator.InputError): + validator.validate_bytes(self.editable, self.editable, contract) + + def test_second_pass_byte_identity_is_enforced(self) -> None: + candidate = self.documented_editable_candidate() + first = validator.validate_bytes(self.editable, candidate, self.editable_contract) + self.assert_valid(first) + second_pass_contract = { + "schema_version": 1, + "original_sha256": hashlib.sha256(candidate).hexdigest(), + "notebook_role": "source", + "policy": "byte-identical", + "allowed_source_changes": [], + } + second = validator.validate_bytes(candidate, candidate, second_pass_contract) + self.assert_valid(second) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/scientific-code-documenter/fixtures/notebook/validate_notebook_contract.py b/evals/scientific-code-documenter/fixtures/notebook/validate_notebook_contract.py new file mode 100644 index 0000000..e54765d --- /dev/null +++ b/evals/scientific-code-documenter/fixtures/notebook/validate_notebook_contract.py @@ -0,0 +1,544 @@ +#!/usr/bin/env python3 +"""Validate narrowly allowlisted edits to a Jupyter notebook. + +The optional behavior probe executes only trusted synthetic fixture code. It is +not a security sandbox and does not establish general notebook equivalence. +""" + +from __future__ import annotations + +import argparse +import ast +import difflib +import hashlib +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +from typing import Any + + +class InputError(Exception): + """Raised when the original notebook or contract is invalid or ambiguous.""" + + +def _reject_constant(value: str) -> None: + raise ValueError(f"non-standard JSON constant: {value}") + + +def _reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValueError(f"duplicate object key: {key}") + result[key] = value + return result + + +def _parse_json(data: bytes, *, label: str) -> tuple[Any, str]: + try: + text = data.decode("utf-8") + except UnicodeDecodeError as exc: + raise ValueError(f"{label} is not valid UTF-8: {exc}") from exc + + try: + value = json.loads( + text, + object_pairs_hook=_reject_duplicate_keys, + parse_constant=_reject_constant, + ) + except (json.JSONDecodeError, ValueError) as exc: + raise ValueError(f"{label} is not strict JSON: {exc}") from exc + return value, text + + +def _notebook_shape_error(notebook: Any) -> str | None: + if not isinstance(notebook, dict): + return "notebook root must be an object" + if not isinstance(notebook.get("cells"), list): + return "notebook cells must be an array" + for index, cell in enumerate(notebook["cells"]): + if not isinstance(cell, dict): + return f"cells[{index}] must be an object" + if not isinstance(cell.get("id"), str) or not cell["id"]: + return f"cells[{index}].id must be a non-empty string" + if cell.get("cell_type") not in {"code", "markdown", "raw"}: + return f"cells[{index}].cell_type is invalid" + source = cell.get("source") + if not isinstance(source, (str, list)): + return f"cells[{index}].source must be a string or string array" + if isinstance(source, list) and not all(isinstance(line, str) for line in source): + return f"cells[{index}].source contains a non-string value" + return None + + +class _SpanScanner: + """Collect raw JSON value spans by structural path.""" + + def __init__(self, text: str) -> None: + self.text = text + self.decoder = json.JSONDecoder() + self.spans: dict[tuple[Any, ...], tuple[int, int]] = {} + + def scan(self) -> dict[tuple[Any, ...], tuple[int, int]]: + end = self._value(self._space(0), ()) + if self._space(end) != len(self.text): + raise ValueError("unexpected content after JSON document") + return self.spans + + def _space(self, index: int) -> int: + while index < len(self.text) and self.text[index] in " \t\r\n": + index += 1 + return index + + def _value(self, index: int, path: tuple[Any, ...]) -> int: + index = self._space(index) + start = index + if index >= len(self.text): + raise ValueError("unexpected end of JSON document") + if self.text[index] == "{": + end = self._object(index, path) + elif self.text[index] == "[": + end = self._array(index, path) + else: + _, end = self.decoder.raw_decode(self.text, index) + self.spans[path] = (start, end) + return end + + def _object(self, index: int, path: tuple[Any, ...]) -> int: + index = self._space(index + 1) + if index < len(self.text) and self.text[index] == "}": + return index + 1 + while True: + key, end = self.decoder.raw_decode(self.text, index) + if not isinstance(key, str): + raise ValueError("object key is not a string") + index = self._space(end) + if index >= len(self.text) or self.text[index] != ":": + raise ValueError("missing colon after object key") + index = self._value(index + 1, path + (key,)) + index = self._space(index) + if index >= len(self.text): + raise ValueError("unterminated object") + if self.text[index] == "}": + return index + 1 + if self.text[index] != ",": + raise ValueError("missing comma in object") + index = self._space(index + 1) + + def _array(self, index: int, path: tuple[Any, ...]) -> int: + index = self._space(index + 1) + if index < len(self.text) and self.text[index] == "]": + return index + 1 + item_index = 0 + while True: + index = self._value(index, path + (item_index,)) + item_index += 1 + index = self._space(index) + if index >= len(self.text): + raise ValueError("unterminated array") + if self.text[index] == "]": + return index + 1 + if self.text[index] != ",": + raise ValueError("missing comma in array") + index = self._space(index + 1) + + +def _source_text(source: str | list[str]) -> str: + return source if isinstance(source, str) else "".join(source) + + +def _changed_line_count(before: str, after: str) -> int: + matcher = difflib.SequenceMatcher(a=before.splitlines(), b=after.splitlines()) + return sum( + max(old_end - old_start, new_end - new_start) + for tag, old_start, old_end, new_start, new_end in matcher.get_opcodes() + if tag != "equal" + ) + + +class _DocstringStripper(ast.NodeTransformer): + def _strip(self, node: ast.AST) -> ast.AST: + body = getattr(node, "body", None) + if ( + isinstance(body, list) + and body + and isinstance(body[0], ast.Expr) + and isinstance(body[0].value, ast.Constant) + and isinstance(body[0].value.value, str) + ): + node.body = body[1:] + return self.generic_visit(node) + + visit_Module = _strip + visit_FunctionDef = _strip + visit_AsyncFunctionDef = _strip + visit_ClassDef = _strip + + +def _python_semantics(source: str) -> str: + tree = ast.parse(source) + stripped = _DocstringStripper().visit(tree) + ast.fix_missing_locations(stripped) + return ast.dump(stripped, include_attributes=False) + + +def _mask_spans( + text: str, + spans: dict[tuple[Any, ...], tuple[int, int]], + paths: set[tuple[Any, ...]], +) -> str: + replacements = [] + for path in paths: + if path not in spans: + raise ValueError(f"missing raw JSON span for {path}") + start, end = spans[path] + replacements.append((start, end)) + for start, end in sorted(replacements, reverse=True): + text = text[:start] + '"__ALLOWLISTED_SOURCE__"' + text[end:] + return text + + +def _violation(category: str, path: str, message: str) -> dict[str, str]: + return {"category": category, "path": path, "message": message} + + +def _validate_contract(contract: Any, original: dict[str, Any], original_data: bytes) -> None: + if not isinstance(contract, dict): + raise InputError("contract root must be an object") + if contract.get("schema_version") != 1: + raise InputError("contract schema_version must be 1") + if contract.get("policy") not in {"allowlisted-source-edits", "byte-identical"}: + raise InputError("contract policy is invalid") + expected_hash = contract.get("original_sha256") + actual_hash = hashlib.sha256(original_data).hexdigest() + if expected_hash != actual_hash: + raise InputError("contract original_sha256 does not match the original notebook") + if contract.get("notebook_role") not in { + "source", + "tutorial", + "analysis artifact", + "generated", + "unknown", + }: + raise InputError("contract notebook_role is invalid") + allowed = contract.get("allowed_source_changes") + if not isinstance(allowed, list): + raise InputError("allowed_source_changes must be an array") + seen: set[int] = set() + for selector in allowed: + if not isinstance(selector, dict): + raise InputError("each source selector must be an object") + index = selector.get("cell_index") + if not isinstance(index, int) or isinstance(index, bool): + raise InputError("source selector cell_index must be an integer") + if index in seen: + raise InputError(f"duplicate source selector for cells[{index}]") + seen.add(index) + if index < 0 or index >= len(original["cells"]): + raise InputError(f"source selector cells[{index}] is out of range") + cell = original["cells"][index] + if selector.get("cell_id") != cell.get("id"): + raise InputError(f"source selector cells[{index}] has a stale cell_id") + if selector.get("cell_type") != cell.get("cell_type"): + raise InputError(f"source selector cells[{index}] has a stale cell_type") + if selector.get("edit_kind") not in {"markdown", "python-comments-docstrings"}: + raise InputError(f"source selector cells[{index}] has an invalid edit_kind") + maximum = selector.get("max_changed_lines") + if not isinstance(maximum, int) or isinstance(maximum, bool) or maximum < 0: + raise InputError(f"source selector cells[{index}] has an invalid line limit") + if contract["policy"] == "byte-identical" and allowed: + raise InputError("byte-identical contracts cannot allow source changes") + + +def _run_probe( + notebook: dict[str, Any], probe: dict[str, Any], *, label: str +) -> tuple[Any | None, str | None]: + selectors = probe.get("selected_code_cells") + callable_name = probe.get("callable") + timeout = probe.get("timeout_seconds", 5) + if not isinstance(selectors, list) or not selectors: + return None, "selected_code_cells must be a non-empty array" + if not isinstance(callable_name, str) or not callable_name.isidentifier(): + return None, "probe callable must be a Python identifier" + if not isinstance(timeout, (int, float)) or isinstance(timeout, bool) or timeout <= 0: + return None, "probe timeout_seconds must be positive" + + sources = [] + for selector in selectors: + if not isinstance(selector, dict) or not isinstance(selector.get("cell_index"), int): + return None, "probe selector is malformed" + index = selector["cell_index"] + if index < 0 or index >= len(notebook["cells"]): + return None, f"probe selector cells[{index}] is out of range" + cell = notebook["cells"][index] + if cell.get("id") != selector.get("cell_id") or cell.get("cell_type") != "code": + return None, f"probe selector cells[{index}] does not match a code cell" + sources.append(_source_text(cell["source"])) + + wrapper = ( + "\n\n".join(sources) + + "\n\nimport json as __probe_json\n" + + f"__probe_value = {callable_name}()\n" + + "print('__NOTEBOOK_PROBE__' + __probe_json.dumps(" + + "__probe_value, ensure_ascii=False, sort_keys=True, separators=(',', ':')))\n" + ) + environment = { + "LC_ALL": "C", + "PATH": os.environ.get("PATH", ""), + "PYTHONIOENCODING": "utf-8", + } + try: + with tempfile.TemporaryDirectory(prefix="notebook-contract-probe-") as workdir: + completed = subprocess.run( + [sys.executable, "-I", "-c", wrapper], + cwd=workdir, + env=environment, + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except subprocess.TimeoutExpired: + return None, f"{label} probe timed out" + if completed.returncode != 0: + return None, f"{label} probe raised an exception" + marker = "__NOTEBOOK_PROBE__" + lines = completed.stdout.splitlines() + if len(lines) != 1 or not lines[0].startswith(marker) or completed.stderr: + return None, f"{label} probe produced unexpected stdout or stderr" + try: + return json.loads(lines[0][len(marker) :]), None + except json.JSONDecodeError: + return None, f"{label} probe returned invalid JSON" + + +def validate_bytes( + original_data: bytes, candidate_data: bytes, contract: dict[str, Any] +) -> dict[str, Any]: + """Return a deterministic validation record for two notebook byte streams.""" + + try: + original, original_text = _parse_json(original_data, label="original notebook") + except ValueError as exc: + raise InputError(str(exc)) from exc + shape_error = _notebook_shape_error(original) + if shape_error: + raise InputError(f"original notebook: {shape_error}") + _validate_contract(contract, original, original_data) + + violations: list[dict[str, str]] = [] + try: + candidate, candidate_text = _parse_json(candidate_data, label="candidate notebook") + except ValueError as exc: + return { + "valid": False, + "policy": contract["policy"], + "notebook_role": contract["notebook_role"], + "intended_source_changes": [], + "violations": [_violation("parseability", "$", str(exc))], + "probe": {"status": "not-run"}, + } + shape_error = _notebook_shape_error(candidate) + if shape_error: + violations.append(_violation("parseability", "$", shape_error)) + return { + "valid": False, + "policy": contract["policy"], + "notebook_role": contract["notebook_role"], + "intended_source_changes": [], + "violations": violations, + "probe": {"status": "not-run"}, + } + + if contract["policy"] == "byte-identical": + if original_data != candidate_data: + violations.append( + _violation( + "serialization-locality", + "$", + "byte-identical notebook changed", + ) + ) + return { + "valid": not violations, + "policy": contract["policy"], + "notebook_role": contract["notebook_role"], + "intended_source_changes": [], + "violations": violations, + "probe": {"status": "not-applicable"}, + } + + allowed = {entry["cell_index"]: entry for entry in contract["allowed_source_changes"]} + original_cells = original["cells"] + candidate_cells = candidate["cells"] + if len(original_cells) != len(candidate_cells): + violations.append(_violation("structural", "$.cells", "cell count changed")) + + original_identity = [(cell.get("id"), cell.get("cell_type")) for cell in original_cells] + candidate_identity = [(cell.get("id"), cell.get("cell_type")) for cell in candidate_cells] + if original_identity != candidate_identity: + violations.append( + _violation("structural", "$.cells", "cell order, IDs, or types changed") + ) + + for key in sorted(set(original) | set(candidate)): + if key == "cells": + continue + if original.get(key) == candidate.get(key) and (key in original) == (key in candidate): + continue + category = "metadata" if key == "metadata" else "structural" if key in {"nbformat", "nbformat_minor"} else "protected-field" + violations.append(_violation(category, f"$.{key}", "protected top-level field changed")) + + intended: list[dict[str, Any]] = [] + mask_paths: set[tuple[Any, ...]] = set() + total_changed_lines = 0 + for index in range(min(len(original_cells), len(candidate_cells))): + before = original_cells[index] + after = candidate_cells[index] + for key in sorted(set(before) | set(after)): + path = f"$.cells[{index}].{key}" + if key == "source": + if before.get(key) == after.get(key) and (key in before) == (key in after): + continue + selector = allowed.get(index) + if selector is None: + violations.append(_violation("unrelated-source", path, "non-allowlisted source changed")) + continue + if type(before.get(key)) is not type(after.get(key)): + violations.append( + _violation("serialization-locality", path, "source container representation changed") + ) + continue + before_text = _source_text(before[key]) + after_text = _source_text(after[key]) + changed_lines = _changed_line_count(before_text, after_text) + total_changed_lines += changed_lines + if changed_lines > selector["max_changed_lines"]: + violations.append( + _violation("serialization-locality", path, "source edit exceeds its changed-line limit") + ) + if selector["edit_kind"] == "markdown" and before.get("cell_type") != "markdown": + violations.append(_violation("protected-field", path, "markdown edit targets a non-Markdown cell")) + if selector["edit_kind"] == "python-comments-docstrings": + try: + before_ast = _python_semantics(before_text) + after_ast = _python_semantics(after_text) + except SyntaxError: + violations.append(_violation("parseability", path, "Python source does not parse")) + else: + if before_ast != after_ast: + violations.append( + _violation("protected-field", path, "Python executable AST changed") + ) + intended.append( + { + "cell_id": before.get("id"), + "cell_index": index, + "changed_lines": changed_lines, + "edit_kind": selector["edit_kind"], + } + ) + mask_paths.add(("cells", index, "source")) + continue + if before.get(key) == after.get(key) and (key in before) == (key in after): + continue + category = { + "attachments": "attachment", + "execution_count": "execution-count", + "metadata": "metadata", + "outputs": "output", + }.get(key, "structural" if key in {"cell_type", "id"} else "protected-field") + violations.append(_violation(category, path, "protected cell field changed")) + + locality = contract.get("locality", {}) + maximum_total = locality.get("max_changed_lines_total") + if not isinstance(maximum_total, int) or isinstance(maximum_total, bool) or maximum_total < 0: + raise InputError("locality.max_changed_lines_total must be a non-negative integer") + if total_changed_lines > maximum_total: + violations.append( + _violation("serialization-locality", "$.cells", "source edits exceed the total changed-line limit") + ) + + try: + original_spans = _SpanScanner(original_text).scan() + candidate_spans = _SpanScanner(candidate_text).scan() + masked_original = _mask_spans(original_text, original_spans, mask_paths) + masked_candidate = _mask_spans(candidate_text, candidate_spans, mask_paths) + except (json.JSONDecodeError, ValueError) as exc: + raise InputError(f"raw JSON span analysis failed: {exc}") from exc + if masked_original != masked_candidate: + violations.append( + _violation( + "serialization-locality", + "$", + "bytes outside intentionally changed source values differ", + ) + ) + + probe_record: dict[str, Any] = {"status": "not-applicable"} + probe = contract.get("behavior_probe") + if probe is not None: + if not isinstance(probe, dict) or "expected_json" not in probe: + raise InputError("behavior_probe is malformed") + original_result, original_error = _run_probe(original, probe, label="original") + candidate_result, candidate_error = _run_probe(candidate, probe, label="candidate") + expected = probe["expected_json"] + if original_error or candidate_error: + message = original_error or candidate_error or "probe failed" + violations.append(_violation("behavior", "$.behavior_probe", message)) + probe_record = {"status": "fail", "message": message} + elif original_result != expected or candidate_result != expected or original_result != candidate_result: + message = "original, candidate, and expected probe results are not identical" + violations.append(_violation("behavior", "$.behavior_probe", message)) + probe_record = {"status": "fail", "message": message} + else: + probe_record = {"status": "pass", "result": expected} + + violations.sort(key=lambda item: (item["category"], item["path"], item["message"])) + intended.sort(key=lambda item: item["cell_index"]) + return { + "valid": not violations, + "policy": contract["policy"], + "notebook_role": contract["notebook_role"], + "intended_source_changes": intended, + "violations": violations, + "probe": probe_record, + } + + +def validate_paths(original: Path, candidate: Path, contract_path: Path) -> dict[str, Any]: + original_data = original.read_bytes() + candidate_data = candidate.read_bytes() + try: + contract, _ = _parse_json(contract_path.read_bytes(), label="contract") + except ValueError as exc: + raise InputError(str(exc)) from exc + return validate_bytes(original_data, candidate_data, contract) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--original", required=True, type=Path) + parser.add_argument("--candidate", required=True, type=Path) + parser.add_argument("--contract", required=True, type=Path) + parser.add_argument("--json", action="store_true", dest="as_json") + args = parser.parse_args(argv) + try: + result = validate_paths(args.original, args.candidate, args.contract) + except (InputError, OSError) as exc: + print(f"VALIDATOR ERROR: {exc}", file=sys.stderr) + return 2 + if args.as_json: + print(json.dumps(result, ensure_ascii=False, sort_keys=True, separators=(",", ":"))) + else: + print("PASS" if result["valid"] else "FAIL") + for violation in result["violations"]: + print(f"{violation['category']}: {violation['path']}: {violation['message']}") + return 0 if result["valid"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/scientific-code-documenter/prompts.yaml b/evals/scientific-code-documenter/prompts.yaml index 95a376c..3a4d06f 100644 --- a/evals/scientific-code-documenter/prompts.yaml +++ b/evals/scientific-code-documenter/prompts.yaml @@ -66,6 +66,30 @@ positive_implicit: language: mixed mode: limited-edit prompt: "Apply focused documentation and readability improvements across the Python analysis and shell entry point, preserving APIs, files, and numerical outputs." + - id: positive-notebook-contracts + language: jupyter + mode: document-existing + prompt: "Document this scientific notebook's evidence-backed units, shapes, ordering, missing-value conventions, and stored-output state without executing it." + - id: positive-notebook-markdown-comments + language: jupyter + mode: comment-cleanup + prompt: "Improve the existing notebook Markdown and code comments, keeping cell structure, metadata, outputs, execution counts, and unrelated serialization unchanged." + - id: positive-notebook-cell-io + language: jupyter + mode: document-existing + prompt: "Clarify the inputs and outputs of the non-trivial cells and record which results are stored rather than recomputed." + - id: positive-data-contracts + language: mixed + mode: document-existing + prompt: "Document the table schema, row ordering, missing-value sentinel, array shape, and filesystem output contract without inventing scientific meaning." + - id: positive-comment-pruning + language: mixed + mode: comment-cleanup + prompt: "Prune the comments that narrate syntax or duplicate nearby documentation, retaining only evidence-backed constraints and interpretation." + - id: positive-notebook-readability + language: jupyter + mode: limited-edit + prompt: "Make this source notebook easier to follow through focused Markdown and comment edits while preserving its calculations, cells, execution state, and stored outputs." negative: - id: negative-debug @@ -110,6 +134,30 @@ negative: - id: negative-html-debug language: html prompt: "Fix the dashboard filter so selecting a sample updates the visible row count." + - id: negative-notebook-execute + language: jupyter + prompt: "Run every cell in this notebook and save fresh outputs." + - id: negative-notebook-crash + language: jupyter + prompt: "Find why cell 12 crashes and fix the exception." + - id: negative-notebook-analysis + language: jupyter + prompt: "Analyze these measurements in the notebook and report the most significant trend." + - id: negative-notebook-methodology + language: jupyter + prompt: "Optimize the fitting methodology in this notebook to reduce the residual error." + - id: negative-notebook-plot + language: jupyter + prompt: "Add a publication-ready plot of the calculated spectrum to this notebook." + - id: negative-notebook-regenerate + language: jupyter + prompt: "Clear all outputs, restart the kernel, and regenerate the notebook from top to bottom." + - id: negative-notebook-debug + language: jupyter + prompt: "Debug the wrong numerical result in the final notebook cell." + - id: negative-notebook-feature + language: jupyter + prompt: "Add an interactive molecule viewer and export button to the notebook." boundary: - id: boundary-shell-review-only @@ -160,6 +208,30 @@ boundary: language: json expected: trigger-and-use-nearby-doc prompt: "Explain the units in this JSON configuration without making the JSON invalid." + - id: boundary-notebook-cleanup-ambiguous + language: jupyter + expected: trigger-and-classify-before-editing + prompt: "Clean up this notebook." + - id: boundary-notebook-review-only + language: jupyter + expected: trigger-review-only + prompt: "Review the notebook for stale scientific claims and missing data contracts. Do not modify or execute it." + - id: boundary-notebook-generated + language: jupyter + expected: trigger-and-skip-generated + prompt: "Improve the comments in this generated pipeline notebook, but leave the generated artifact byte-identical and identify the upstream edit target." + - id: boundary-notebook-analysis-artifact + language: jupyter + expected: trigger-and-report-risk + prompt: "Assess whether this executed analysis notebook can safely receive documentation edits while preserving every stored output and execution count." + - id: boundary-notebook-unknown-role + language: jupyter + expected: trigger-and-report-risk + prompt: "Document this notebook, but its source/generated status and execution provenance are unknown." + - id: boundary-notebook-questionable-claim + language: jupyter + expected: trigger-review-only + prompt: "A Markdown cell assigns units to an unexplained scale factor. Determine whether the claim is supported, but do not change the notebook." explicit_invocation: - id: explicit-review-only @@ -186,3 +258,11 @@ explicit_invocation: language: latex mode: review-only prompt: "$scientific-code-documenter\nReview report.tex for source readability, label and macro contracts, and unsupported scientific interpretation. Do not modify it." + - id: explicit-notebook-review + language: jupyter + mode: review-only + prompt: "$scientific-code-documenter\nReview this notebook's role, scientific and data contracts, execution state, outputs, and documentation risks. Do not modify or execute it." + - id: explicit-notebook-limited + language: jupyter + mode: limited-edit + prompt: "$scientific-code-documenter\nApply only evidence-backed Markdown, comment, and docstring improvements to the explicitly allowlisted cells. Preserve all other notebook bytes and do not execute it." diff --git a/evals/scientific-code-documenter/results/result.template.md b/evals/scientific-code-documenter/results/result.template.md index 889c6db..d813c8a 100644 --- a/evals/scientific-code-documenter/results/result.template.md +++ b/evals/scientific-code-documenter/results/result.template.md @@ -9,6 +9,10 @@ - Skill run invoked explicitly: yes / no - Baseline validation result: - Skill validation result: +- Notebook role (if applicable): +- Risk tier and Tier-3 reasons: +- Evidence classes recorded: +- Permitted notebook source fields: ## Scores @@ -22,6 +26,24 @@ | Scope discipline | | | | | **Total** | | | | +## Notebook hard gates + +| Gate | Baseline | Skill | Evidence path | +| --- | --- | --- | --- | +| Original identity | | | | +| Protected structure | | | | +| Source allowlist | | | | +| Textual locality | | | | +| Review/generated byte identity | | | | +| Deterministic fixture probe | | | | +| Validator health | | | | +| Second-pass idempotence | not required | | | + +- Validator JSON: +- Pass-2 raw log: +- Pass-2 diff: +- Notebook bytes identical after pass 2: yes / no / not applicable + ## Observations - Strengths: diff --git a/evals/scientific-code-documenter/rubric.md b/evals/scientific-code-documenter/rubric.md index d590060..e16f28a 100644 --- a/evals/scientific-code-documenter/rubric.md +++ b/evals/scientific-code-documenter/rubric.md @@ -5,11 +5,11 @@ Score each category from 0 to 3. Record evidence from the diff and validation lo | Category | 0 | 1 | 2 | 3 | | --- | --- | --- | --- | --- | | Readability | Harder to scan or substantially churned | Minor cosmetic gains; major density remains | Clearer structure with a few avoidable issues | Focused structure, names, spacing, and flow materially improve scanning | -| Function documentation | Missing, misleading, or mechanical | Some non-trivial interfaces documented poorly | Important interfaces documented with small omissions | Concise, language-appropriate contracts cover purpose and meaningful failures | -| Scientific context | Invented or materially wrong | Key units, shapes, or conventions remain ambiguous | Most relevant context is explicit without guessing | Units, shapes, domains, assumptions, and distinctions are precise and evidence-based | -| Comment quality | Narrates syntax or uses repetitive AI-like phrasing | Too sparse, verbose, or generic | Mostly useful with limited redundancy | Human-sounding comments explain only intent, constraints, conventions, or safeguards | +| Function documentation | Missing, misleading, or mechanical | Some non-trivial interfaces or cell inputs/outputs documented poorly | Important interfaces and notebook flow documented with small omissions | Concise, language-appropriate contracts cover purpose, cell inputs/outputs, and meaningful failures | +| Scientific context | Invented or materially wrong | Key units, shapes, conventions, or evidence limits remain ambiguous | Most relevant context and uncertainty are explicit without guessing | Contracts are precise, evidence-classified, and preserve unknown scientific meaning | +| Comment quality | Narrates syntax, duplicates Markdown, or uses repetitive AI-like phrasing | Too sparse, verbose, generic, or reluctant to prune | Mostly useful with limited redundancy | Human-sounding comments explain only real contracts; obvious or duplicate commentary is removed or consolidated | | Behavior preservation | Any syntax, API, format, test, or numerical regression | No detected regression but validation is inadequate | Checks pass but material behavior remains unprotected | All relevant checks pass and public, numerical, ordering, and file-format behavior is preserved | -| Scope discipline | Unrequested rewrite or generated/vendor edits | Significant unrelated formatting or risky refactor | Mostly focused with small avoidable churn | Minimal, reviewable diff; risky or suspicious logic is reported rather than changed | +| Scope discipline | Unrequested rewrite or generated/vendor edits | Significant unrelated formatting, notebook churn, or risky refactor | Mostly focused with small avoidable churn | Minimal, reviewable diff; role and risk are classified, and Tier-3 or suspicious work is reported rather than changed | ## Acceptance criteria @@ -23,6 +23,23 @@ Score each category from 0 to 3. Record evidence from the diff and validation lo Do not average away a behavior regression. A fixture that scores below any mandatory criterion fails even if the overall mean passes. +## Notebook hard gates + +Evaluate these gates before qualitative scoring. Every applicable gate must pass. + +| Gate | Required result | +| --- | --- | +| Original identity | Contract SHA-256 matches the exact immutable starting notebook | +| Protected structure | 100% preservation of formats, cells, order, IDs, types, metadata, attachments, execution counts, and outputs | +| Source allowlist | Zero changes outside explicitly permitted cell source fields | +| Textual locality | Zero byte changes outside intentionally changed allowlisted source values | +| Review/generated restraint | Review-only and generated notebooks remain byte-identical | +| Behavioral probe | Exact numerical values, missing-value behavior, and serialized order for applicable trusted synthetic fixtures | +| Idempotence | Fresh second skill pass creates zero tracked diff; notebook bytes remain identical to pass 1 | +| Validator health | Zero parse ambiguities, internal errors, timeouts, or unexplained changes | + +A notebook editing candidate must beat its baseline. Review-only and generated candidates may tie, but they must not lose or change a byte. + ## Result record For each run, capture: @@ -34,5 +51,7 @@ For each run, capture: - baseline and skill scores by category; - validation commands and exit status; - saved diff and log paths; +- notebook role, risk tier and reasons, evidence classes, allowlisted source fields, validator report, and hard-gate outcomes; +- second-pass prompt, raw log, diff, and byte-identity result for accepted editing candidates; - observed strengths and regressions; - follow-up changes or unresolved ambiguity. From 3c631207b6dd31c921e9faaeede559af1ef69098 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 13:15:13 +0800 Subject: [PATCH 03/17] fix: make documentation passes idempotent --- skills/scientific-code-documenter/SKILL.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/scientific-code-documenter/SKILL.md index 541c76f..1d81626 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/scientific-code-documenter/SKILL.md @@ -21,8 +21,9 @@ Improve scientific and technical code without obscuring its logic or silently ch 4. For notebooks, classify the artifact role as `source`, `tutorial`, `analysis artifact`, `generated`, or `unknown` before deciding whether an edit is safe. 5. Grade the proposed work from Tier 0 through Tier 3 and perform only the authorized, protectable tier. 6. Define the smallest coherent scope and apply repository-specific conventions before general preferences. -7. Preserve public interfaces, calculations, numerical behavior, outputs, notebook state, and file formats unless the user explicitly authorizes and protects a change. -8. Run the narrowest relevant validation. Fail closed on unexplained contract changes and report exactly what ran. +7. Treat documentation that already satisfies the requested contracts as complete. Do not create a diff merely to rephrase, expand, rename local variables, or apply another plausible cleanup. +8. Preserve public interfaces, calculations, numerical behavior, outputs, notebook state, and file formats unless the user explicitly authorizes and protects a change. +9. Run the narrowest relevant validation. Fail closed on unexplained contract changes and report exactly what ran. ### Treat operating mode as a hard boundary @@ -72,6 +73,7 @@ For review findings, report: location, issue, evidence, contract affected, confi ### Refactor for readability - Simplify dense expressions, deep nesting, long functions, or opaque transformations only where understanding materially improves. +- Require a concrete current readability defect before applying a Tier-2 refactor. A broad readability request is not sufficient when the existing implementation and documentation already expose the relevant contracts. - Use clear intermediate variables and coherent helpers. - Preserve evaluation order, floating-point behavior, side effects, exception behavior, and data ordering unless change is requested. - Keep public names stable. Treat symbol renaming and extraction across module boundaries as higher-risk changes. @@ -84,6 +86,20 @@ For review findings, report: location, issue, evidence, contract affected, confi - Cite concrete files and tight line ranges, explain impact, and suggest the smallest remedy. - State when no actionable issue is found. +## Make repeated invocation stable + +Before editing, ask whether the current artifact already meets the request. If its high-value contracts are documented, unsupported meaning remains explicitly unknown, and no stale or redundant commentary remains, make no change and report that the existing documentation is sufficient. + +Do not use a repeated invocation to: + +- add lower-value detail omitted by a focused prior pass; +- rephrase accurate documentation without a demonstrated clarity defect; +- add more comments around already documented behavior; +- rename locals, expand formatting, or perform a new Tier-2 refactor merely because another form is possible; +- alternate between equivalent wording or representation choices. + +Make the first accepted edit coherent enough that the same request against its result produces no tracked change. In an evaluation or release workflow, verify this with a fresh second pass and fail the candidate if content or notebook bytes drift. + ## Readability rules - Prefer descriptive names that convey scientific or computational meaning. From a220b82a6461101c8bb61126c5e1b7f843faae68 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 13:15:35 +0800 Subject: [PATCH 04/17] test: support targeted static LaTeX validation --- .../fixtures/contracts/test_latex_fixtures.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py b/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py index cedbf76..0fce308 100644 --- a/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py +++ b/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py @@ -215,18 +215,18 @@ def validate_restraint(source: Path, *, compile_output: bool = True) -> None: def main() -> int: mode = sys.argv[1] if len(sys.argv) > 1 else "all" explicit_path = Path(sys.argv[2]).resolve() if len(sys.argv) > 2 else None - compile_output = mode != "static" - if mode in {"all", "main", "static"}: + compile_output = mode not in {"static", "main-static", "restraint-static"} + if mode in {"all", "main", "static", "main-static"}: validate_main( explicit_path or FIXTURE_ROOT / "latex/crowded_scientific_report.tex", compile_output=compile_output, ) - if mode in {"all", "restraint", "static"}: + if mode in {"all", "restraint", "static", "restraint-static"}: validate_restraint( explicit_path or FIXTURE_ROOT / "latex/fragile_template_snippet.tex", compile_output=compile_output, ) - if mode not in {"all", "main", "restraint", "static"}: + if mode not in {"all", "main", "restraint", "static", "main-static", "restraint-static"}: raise SystemExit(f"unknown mode: {mode}") return 0 From 68a0c12fd478ff36c1808cb3fdbe9ecbe91b7a3d Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 13:23:02 +0800 Subject: [PATCH 05/17] fix: preserve existing documentation diffs --- skills/scientific-code-documenter/SKILL.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/scientific-code-documenter/SKILL.md index 1d81626..2772998 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/scientific-code-documenter/SKILL.md @@ -16,6 +16,7 @@ Improve scientific and technical code without obscuring its logic or silently ch - Read nearby code to learn naming, layout, docstring, and comment conventions. - Identify generated and vendored files that should remain untouched. - Check the working tree and preserve unrelated user changes. + - If a tracked target already has documentation or readability changes, treat them as user or prior-pass work. Do not refine, replace, or stack another pass unless the user explicitly asks to continue from those existing changes. 2. Classify the operating mode and treat review-only or no-edit language as a hard boundary. 3. Build a compact contract inventory for the requested scope. Classify each relevant item as `evidence-backed`, `observable-only`, or `unknown`. 4. For notebooks, classify the artifact role as `source`, `tutorial`, `analysis artifact`, `generated`, or `unknown` before deciding whether an edit is safe. @@ -90,6 +91,8 @@ For review findings, report: location, issue, evidence, contract affected, confi Before editing, ask whether the current artifact already meets the request. If its high-value contracts are documented, unsupported meaning remains explicitly unknown, and no stale or redundant commentary remains, make no change and report that the existing documentation is sufficient. +Use the initial working-tree state as a preservation boundary. When the requested tracked file already contains in-scope documentation or readability changes, a generic request to improve that file is not permission to revise those changes. Inspect and validate them read-only, report any remaining concern separately, and leave the file unchanged unless the user explicitly authorizes building on the existing diff. In a controlled second pass, this existing diff is the prior candidate and must remain byte-identical. + Do not use a repeated invocation to: - add lower-value detail omitted by a focused prior pass; From f5945677d57f8189811f25bddcab07994b95fd16 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 13:29:42 +0800 Subject: [PATCH 06/17] fix: retain contracts while pruning comments --- skills/scientific-code-documenter/SKILL.md | 1 + .../references/comment-examples.md | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/scientific-code-documenter/SKILL.md index 2772998..9f47e73 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/scientific-code-documenter/SKILL.md @@ -181,6 +181,7 @@ Maintain distinctions that affect interpretation, including measured versus calc - Avoid vague praise such as “robust,” “seamless,” “comprehensive,” or “efficient” unless technically defined. - Avoid repeatedly stating “This ensures that” when the consequence is already evident. - Remove comments that narrate syntax, duplicate nearby Markdown, contradict the code, or bury the important contract. +- When pruning narration, preserve or consolidate any evidence-backed data contract it obscures. If missing-value handling, ordering, units, or another non-obvious contract would otherwise be undocumented, keep one concise authoritative explanation instead of deleting all context. - Replace or remove an unsupported scientific claim only when repository evidence establishes the correction or the claim is clearly redundant. Otherwise report the ambiguity and require expert confirmation, even when editing is authorized. Consult [references/comment-examples.md](references/comment-examples.md) when concrete wording examples would help. Do not load it when the repository's own conventions already settle the choice. diff --git a/skills/scientific-code-documenter/references/comment-examples.md b/skills/scientific-code-documenter/references/comment-examples.md index a010c0b..fbca406 100644 --- a/skills/scientific-code-documenter/references/comment-examples.md +++ b/skills/scientific-code-documenter/references/comment-examples.md @@ -96,6 +96,18 @@ for row in rows: The code is clearer without either comment. Add documentation only if ordering, encoding, or failure behavior is non-obvious and evidence-backed. +If the removed narration was the only description of a real data contract, +consolidate that contract once: + +```python +def summarize(values): + """Omit missing values and return the remainder in ascending order.""" + ... +``` + +This is preferable to either narrating the loop line by line or deleting the +only statement of the missing-value and ordering policy. + ## Unsupported explanations and expert boundaries Replace an unsupported claim with neutral wording only when repository evidence establishes that the neutral wording is accurate: From 8dedc6718a67d82376e5963991fd1bbd48057d79 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 13:53:27 +0800 Subject: [PATCH 07/17] fix: document observable contracts neutrally --- skills/scientific-code-documenter/SKILL.md | 4 ++-- .../references/comment-examples.md | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/scientific-code-documenter/SKILL.md index 9f47e73..77e117a 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/scientific-code-documenter/SKILL.md @@ -45,7 +45,7 @@ Do not infer a scientific contract from domain familiarity or an arithmetic patt ## Grade risk before acting - **Tier 0 — review/planning only:** make no changes when the request is non-editing, intent is too uncertain, the artifact is generated or unknown, or preservation cannot be verified. -- **Tier 1 — documentation only:** edit comments, docstrings, or Markdown; prune redundant or misleading commentary; document only evidence-backed contracts. Do not change behavior. +- **Tier 1 — documentation only:** edit comments, docstrings, or Markdown; prune redundant or misleading commentary; state evidence-backed contracts precisely and observable-only operations neutrally. Leave unknown meaning unresolved. Do not change behavior. - **Tier 2 — local readability refactoring:** refactor only when explicitly requested, locally protectable, and free of Tier-3 effects. Require targeted behavior checks. - **Tier 3 — protected/high risk:** report by default. This includes numerical evaluation order, public interfaces, notebook structure or identity, execution state, stored outputs, metadata, and artifact contracts. Edit only with explicit authorization and suitable protection. @@ -181,7 +181,7 @@ Maintain distinctions that affect interpretation, including measured versus calc - Avoid vague praise such as “robust,” “seamless,” “comprehensive,” or “efficient” unless technically defined. - Avoid repeatedly stating “This ensures that” when the consequence is already evident. - Remove comments that narrate syntax, duplicate nearby Markdown, contradict the code, or bury the important contract. -- When pruning narration, preserve or consolidate any evidence-backed data contract it obscures. If missing-value handling, ordering, units, or another non-obvious contract would otherwise be undocumented, keep one concise authoritative explanation instead of deleting all context. +- When pruning narration, preserve or consolidate any evidence-backed or directly observable data contract it obscures. A visible conditional, sort operation, schema, or test can establish neutral missing-value or ordering behavior without establishing its scientific purpose. If such a non-obvious contract would otherwise be undocumented, keep one concise authoritative explanation instead of deleting all context. - Replace or remove an unsupported scientific claim only when repository evidence establishes the correction or the claim is clearly redundant. Otherwise report the ambiguity and require expert confirmation, even when editing is authorized. Consult [references/comment-examples.md](references/comment-examples.md) when concrete wording examples would help. Do not load it when the repository's own conventions already settle the choice. diff --git a/skills/scientific-code-documenter/references/comment-examples.md b/skills/scientific-code-documenter/references/comment-examples.md index fbca406..47bbb81 100644 --- a/skills/scientific-code-documenter/references/comment-examples.md +++ b/skills/scientific-code-documenter/references/comment-examples.md @@ -96,8 +96,10 @@ for row in rows: The code is clearer without either comment. Add documentation only if ordering, encoding, or failure behavior is non-obvious and evidence-backed. -If the removed narration was the only description of a real data contract, -consolidate that contract once: +If the removed narration was the only description of a real or directly +observable data contract, consolidate that contract once. A visible `None` +filter and ascending `sort()` establish the behavior below without establishing +why the scientific workflow chose it: ```python def summarize(values): From 605e4fe280a36c5966586ce2e18931d463b0b71a Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 13:57:43 +0800 Subject: [PATCH 08/17] fix: consolidate notebook pruning contracts --- skills/scientific-code-documenter/SKILL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/scientific-code-documenter/SKILL.md index 77e117a..8740492 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/scientific-code-documenter/SKILL.md @@ -70,6 +70,7 @@ For review findings, report: location, issue, evidence, contract affected, confi - Remove or correct stale, misleading, redundant, and obviously generated-sounding comments when confidence is high. - Leave suspicious logic unchanged and flag it separately when its intended behavior is uncertain. - Prefer deleting or consolidating stale, obvious, duplicated, or unsupported commentary over adding another explanation. +- A pruning pass is incomplete if it deletes the only documentation of a non-obvious observable contract. Replace line-by-line narration with one concise Markdown statement or docstring when the code visibly filters missing values, imposes ordering, or changes representation. ### Refactor for readability @@ -242,6 +243,7 @@ nanoseconds to seconds” when the units are not documented. - Preserve `nbformat`, `nbformat_minor`, notebook and cell metadata, cell count and order, cell IDs and types, execution counts, outputs, attachments, and untouched source by default. - Do not execute, clear or regenerate outputs, normalize JSON or metadata, convert to a script, add or remove cells, reorder cells, or repair hidden state unless explicitly requested and protected. - Change only the minimum existing Markdown or code-cell source needed. Prefer one authoritative explanation instead of duplicated Markdown and comments. +- During notebook comment pruning, do not reduce explanatory Markdown to a bare heading when the allowlisted code visibly filters missing values, sorts records, or otherwise imposes a data contract. Consolidate that observable behavior once without inventing its scientific purpose. - Require both structural preservation and a focused textual diff. If the editing mechanism would rewrite unrelated serialization, report the limitation and do not edit. - Report hidden or prior-state dependencies without silently restructuring the notebook. - When source changes but stored outputs remain, say the notebook was not executed and do not imply that displayed outputs were recomputed. From 714c27a1f54df0cd8ce682030da51554cb446bea Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 14:05:56 +0800 Subject: [PATCH 09/17] docs: summarize v0.4 evaluation evidence --- CHANGELOG.md | 33 +++++++++ GETTING_STARTED.md | 13 +++- README.md | 19 +++-- .../scientific-code-documenter-v0.4.0.md | 70 +++++++++++++++++++ .../scientific-code-documenter-v0.4.0.md | 43 ++++++++++++ 5 files changed, 171 insertions(+), 7 deletions(-) create mode 100644 docs/evaluations/scientific-code-documenter-v0.4.0.md create mode 100644 docs/releases/scientific-code-documenter-v0.4.0.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e0f9b4..bbc06b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,39 @@ All notable changes to this collection are documented here. No remote release ha No changes yet. +## [0.4.0] - 2026-08-28 + +### Added + +- Scientific and artifact contract inventory with `evidence-backed`, `observable-only`, and `unknown` classifications. +- Tier 0–3 risk grading and standardized review findings. +- First-class conservative Jupyter Notebook policy, role classification, and progressive references. +- Dependency-free notebook structural/locality validator, deterministic synthetic probe, four notebook fixtures, and hard-gate tests. +- Comment-pruning rules that consolidate real data contracts instead of adding narration or erasing useful context. +- Fresh-pass idempotence checks and dirty-target preservation. + +### Changed + +- Runtime behavior now covers scientific contracts, notebook artifacts, pruning, and explicit preservation boundaries. +- Controlled A/B evaluation now includes notebooks plus Python, Shell, LaTeX, and HTML regressions. +- Historical v0.1–v0.3 results remain predecessor evidence rather than acceptance of the changed runtime. + +### Validated + +- Six editing fixtures averaged 17.5/18, with six skill wins and behavior preservation at 3/3 throughout. +- Editable, review-only, pruning, and generated notebook hard gates, including exact probe output and byte-level locality. +- Six accepted editing candidates remained byte-identical on fresh second passes. +- Positive implicit triggering at 10/10 and false triggering at 1/10. + +### Known limitations + +- Explicitly authorized real-notebook acceptance remains pending. +- One notebook feature-development prompt falsely triggered the skill. +- Manual A/B scoring was not independently blind. +- ShellCheck and rendered LaTeX compilation were unavailable in the release environment. +- Deterministic fixture probes do not prove general notebook equivalence or scientific correctness. +- Human scientific review remains required; the release remains experimental. + ## [0.3.0] - 2026-07-29 ### Added diff --git a/GETTING_STARTED.md b/GETTING_STARTED.md index 3a78c45..b73a809 100644 --- a/GETTING_STARTED.md +++ b/GETTING_STARTED.md @@ -1,6 +1,6 @@ # Getting started -This guide shows how to install and use the `scientific-code-documenter` skill in Codex. The skill helps review and improve scientific or technical code documentation and readability without intentionally changing behavior. +This guide shows how to install and use the `scientific-code-documenter` skill in Codex. The skill helps review, document, and prune scientific code or Jupyter notebooks while preserving explicit scientific, data, interface, execution-state, and artifact contracts. ## What you need @@ -38,7 +38,16 @@ Apply only high-confidence documentation and readability improvements to this fi Attach the file, open its repository in Codex, or name the file path in the prompt. Codex may also select the skill automatically when a request clearly matches its purpose. -Always review scientific edits before accepting them. The skill preserves behavior by design, but it cannot independently confirm undocumented scientific meaning or every environment-specific interaction. +For a conservative notebook review: + +```text +$scientific-code-documenter +Review this notebook's role, scientific and data contracts, execution state, outputs, and documentation risks. Do not modify or execute it. +``` + +For an authorized notebook documentation edit, name the permitted cells or source fields and the protected state. The skill does not execute notebooks, clear outputs, add or reorder cells, or normalize notebook JSON by default. + +Always review scientific edits before accepting them. The skill preserves behavior by design, but it cannot independently confirm undocumented scientific meaning, general notebook equivalence, stored-output validity, or every environment-specific interaction. ## Manual installation from a clone diff --git a/README.md b/README.md index 0b0bb77..4cf7e99 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,10 @@ codex_skills/ │ └── scientific-code-documenter/ │ ├── SKILL.md │ ├── agents/openai.yaml -│ └── references/comment-examples.md +│ └── references/ +│ ├── comment-examples.md +│ ├── jupyter-notebooks.md +│ └── scientific-contracts.md ├── evals/ │ ├── README.md │ └── scientific-code-documenter/ @@ -37,7 +40,7 @@ codex_skills/ | Skill | Purpose | Status | | --- | --- | --- | -| `scientific-code-documenter` | Improve scientific and technical code documentation and readability while preserving behavior | Experimental v0.3.0; validated with scientific Python and ORCA Shell real-repository acceptance plus controlled Python, Shell, LaTeX, HTML/CSS/JavaScript, trigger, and boundary tests | +| `scientific-code-documenter` | Document and prune scientific code and Jupyter notebooks using explicit contract inventory, evidence classes, and risk grading | Experimental v0.4.0 release candidate; synthetic notebook and controlled Python, Shell, LaTeX, HTML, trigger, restraint, and idempotence gates pass; authorized real-notebook acceptance is pending | ## Validate and install @@ -81,6 +84,8 @@ Codex may also select it implicitly for requests such as: - “Clarify this Shell workflow's aggregate-error behavior without changing its exit codes.” - “Organize these LaTeX macros while preserving labels, rendered content, and the template workaround.” - “Make this scientific dashboard easier to maintain without changing its DOM or CSV contracts.” +- “Document this notebook's cell inputs, missing-value rules, ordering, and stored-output state without executing it.” +- “Prune syntax-narrating notebook comments while preserving outputs, metadata, execution counts, and cell identity.” For an editing-mode request: @@ -89,7 +94,9 @@ $scientific-code-documenter Apply only high-confidence documentation and readability improvements to this file. Preserve calculations, public interfaces, file formats, and numerical behaviour. ``` -Units, shapes, conventions, physical roles, and numerical context should be documented only when supported by code, tests, configuration, repository documentation, or user-provided evidence. Uncertainty should be stated rather than converted into scientific fact. A review-only or “do not edit” instruction takes precedence over opportunities to fix stale comments or improve code. +Units, shapes, conventions, physical roles, and numerical context should be documented precisely only when supported by code, tests, configuration, repository documentation, or user-provided evidence. Directly observable operations may be described neutrally; unknown scientific meaning stays unresolved. A review-only or “do not edit” instruction takes precedence over opportunities to fix stale comments or improve code. + +Jupyter notebooks are treated as structured artifacts. The skill classifies notebook role and risk before editing, preserves cell identity, metadata, execution counts, attachments, and stored outputs by default, and does not execute notebooks merely to refresh state. Generated, unknown-role, and protected analysis notebooks normally remain review-only. Ordinary debugging, dependency upgrades, test execution, and unrelated feature work should not trigger the skill merely because code is present. @@ -109,9 +116,11 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for the complete workflow. Structural validation proves that a skill can be discovered and parsed; it does not prove usefulness. Evaluation therefore combines controlled A/B runs, trigger and scope-boundary measurements, independent behavior checks, adversarial fixtures, and authorized real-repository acceptance. -`scientific-code-documenter` v0.3.0 retains the accepted v0.2.0 runtime unchanged. Its real-repository ORCA Shell acceptance reviewed nine production-style scripts (4,319 lines), made zero review-only edits, and tested three controlled A/B pairs. The skill scored 89/90 versus 80/90 baseline, won all three scripts, and preserved generated ORCA inputs, command order, CLI and environment contracts, failures, restarts, cleanup, logs, outputs, and exit codes under independent mocks and dry runs. No expensive scientific calculation was launched. +`scientific-code-documenter` v0.4.0 changes runtime behavior. It adds scientific/artifact contract inventory, `evidence-backed` / `observable-only` / `unknown` classification, Tier 0–3 risk grading, structured review findings, conservative notebook policy, comment pruning, and repeated-pass stability. + +Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. Positive implicit triggering was 10/10; one of ten negative prompts falsely triggered. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable, so Shell syntax/dynamic contracts and static LaTeX contracts were used. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/scientific-code-documenter-v0.4.0.md). -The prior scientific Python real-repository acceptance and controlled Shell, LaTeX, HTML, CSS, and JavaScript evidence remain applicable because the runtime hash did not change. Mock and dry-run equivalence cannot prove every scheduler, cluster, MPI, or operating-system interaction. See the [sanitized v0.3.0 evaluation summary](docs/evaluations/scientific-code-documenter-v0.3.0.md) for exact results and limitations. The skill remains experimental and scientific edits still require human review. +Historical v0.1–v0.3 scientific Python and ORCA Shell evidence remains useful predecessor evidence, but it is not accepted as proof for the changed v0.4 runtime. No real notebook was supplied with explicit authorization, so real-notebook acceptance remains pending. The skill remains experimental; limited probes do not prove notebook equivalence or scientific correctness, and human scientific review remains mandatory. ## Privacy and publication diff --git a/docs/evaluations/scientific-code-documenter-v0.4.0.md b/docs/evaluations/scientific-code-documenter-v0.4.0.md new file mode 100644 index 0000000..d674931 --- /dev/null +++ b/docs/evaluations/scientific-code-documenter-v0.4.0.md @@ -0,0 +1,70 @@ +# scientific-code-documenter v0.4.0 validation + +## Version status + +- Version: `scientific-code-documenter` v0.4.0, before the final `angze-code-style` migration +- Release status: experimental local release candidate +- Runtime behavior changed from v0.3.0: yes +- Real-notebook acceptance: pending explicit authorization +- Provisional verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING** + +## Methodology + +The controlled comparison used fresh ephemeral Codex CLI sessions with `gpt-5.6-terra`, medium reasoning, workspace-write sandboxes, identical synthetic starting files, and a zero-user-skill baseline. Candidate sessions exposed one frozen runtime snapshot and explicitly invoked the skill for A/B editing and restraint runs. Independent fixture validators ran after every candidate. + +The eight pairs covered editable notebook documentation, notebook review restraint, notebook comment pruning, generated-notebook restraint, Python, Shell, LaTeX, and HTML. Fresh second sessions repeated the same skill prompt against all six accepted editing candidates. Raw prompts, event logs, diffs, responses, validation records, failed attempts, and second-pass evidence remain in the ignored local result bundle. + +Scoring was a manual evidence-based side-by-side review of saved diffs and validation results. It was not randomized or independently blind; this limits the strength of comparative claims. + +## A/B result + +| Fixture | Baseline | Skill | Outcome | +| --- | ---: | ---: | --- | +| Editable notebook | 17/18 | 18/18 | Skill win | +| Notebook pruning | 14/18 | 17/18 | Skill win | +| Python | 14/18 | 17/18 | Skill win | +| Shell | 16/18 | 17/18 | Skill win | +| LaTeX | 16/18 | 18/18 | Skill win | +| HTML | 15/18 | 18/18 | Skill win | +| **Average** | **15.3/18** | **17.5/18** | **6 wins, 0 ties, 0 losses** | + +Every accepted skill fixture scored at least 17/18 and behavior preservation was 3/3 throughout. Review-only and generated notebooks were unscored byte-identical ties. + +The baseline Python candidate added public annotations and asserted more scientific interpretation than the repository established. The accepted skill candidate retained public signatures and described the threshold neutrally. The skill notebook candidate explicitly kept the scale factor's units and scientific meaning unknown. The pruning candidate consolidated visible `None` filtering and ascending ordering once while deleting syntax narration. Shell, LaTeX, and HTML candidates documented concrete contracts more locally and concisely. + +## Notebook hard gates + +| Gate | Result | +| --- | --- | +| Protected notebook structure | 100% preserved | +| Unexpected output, metadata, attachment, cell-ID, or execution-count changes | 0 | +| Non-allowlisted source changes | 0 | +| Bytes outside intended source values | 0 changes | +| Review-only and generated notebooks | 0 changed bytes | +| Deterministic numerical, missing-value, and ordering probe | Exact pass | +| Fresh second-pass idempotence | 6/6 editing candidates byte-identical | +| Accepted-candidate validator failures | 0 | + +The validator uses strict standard-library JSON parsing, exact original hashes, explicit source selectors, protected-field comparison, raw JSON span masking, line budgets, Python AST comparison for comment/docstring-only edits, and an optional trusted synthetic probe. It fails closed on malformed input, stale selectors, ambiguity, serialization churn, or internal error. + +## Trigger result + +Ten positive implicit prompts covered Python, Shell, LaTeX, HTML, notebook documentation, notebook readability, data contracts, and comment pruning. All ten activated or read the skill. Ten negative prompts covered debugging, dependencies, tests, performance, data analysis, notebook execution, crashes, methodology, and features. One notebook feature prompt read the skill, for a 10% false-trigger rate. + +Thus positive accuracy was 100% and false triggering was 10%, meeting the release thresholds. The feature false positive is retained as a known limitation. + +## Retained failed attempts + +- One orchestration attempt used incompatible current CLI flags and exited before model launch; no candidate changed. +- Initial Python, Shell, and HTML candidates drifted on a second pass. The runtime added an explicit dirty-target preservation boundary, and all affected reruns became byte-identical. +- Deletion-only notebook-pruning attempts tied the baseline. The runtime's evidence-class wording was reconciled so visible missing-value and ordering operations are documented neutrally without inventing scientific purpose; the final retained candidate passed and won. + +These attempts remain in the local result bundle and were not removed or scored as accepted outputs. + +## Existing evidence and limitations + +Historical v0.1–v0.3 scientific Python and ORCA Shell acceptance remains predecessor evidence only because v0.4 changes runtime instructions. + +No notebook owner explicitly authorized a real immutable notebook snapshot. Per the release boundary, no arbitrary personal or research notebook was substituted. Real-notebook zero-edit review and controlled baseline-versus-skill editing remain pending. + +ShellCheck and `pdflatex` were unavailable. Required Shell syntax and dynamic contract checks passed; LaTeX static contracts passed, but rendered compilation did not run. The deterministic notebook probe is limited fixture evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. diff --git a/docs/releases/scientific-code-documenter-v0.4.0.md b/docs/releases/scientific-code-documenter-v0.4.0.md new file mode 100644 index 0000000..0ee6c4b --- /dev/null +++ b/docs/releases/scientific-code-documenter-v0.4.0.md @@ -0,0 +1,43 @@ +# scientific-code-documenter v0.4.0 + +Release preparation date: 2026-08-28 + +This experimental release candidate changes runtime behavior. It upgrades scientific documentation from style guidance to contract-aware, risk-graded work and adds conservative first-class support for Jupyter Notebook artifacts. + +The final v0.4 package will be renamed to `angze-code-style` after the pre-rename implementation, evaluation, and local release preparation are stable. Final renamed runtime and manifest hashes will be recorded after that migration. + +## Added + +- A pre-edit inventory of scientific meaning, units, shapes, schemas, ordering, missing values, state dependencies, interfaces, outputs, and artifact structure. +- Explicit `evidence-backed`, `observable-only`, and `unknown` classifications. +- Tier 0 review through Tier 3 protected-risk decisions, with standardized finding fields and reason labels. +- Notebook role classification: source, tutorial, analysis artifact, generated, or unknown. +- Structured notebook preservation for formats, cells, IDs, types, metadata, attachments, execution counts, outputs, untouched source, and serialization locality. +- Comment pruning that removes syntax narration while retaining concise observable data contracts. +- A standard-library notebook validator and isolated deterministic fixture probe. +- Four public-safe notebook fixtures plus Python, Shell, LaTeX, and HTML regression coverage. + +## Evaluation result + +Six editing comparisons averaged 17.5/18, with a minimum of 17/18, behavior preservation at 3/3 for every accepted fixture, and six skill wins. Review-only and generated notebooks were byte-identical ties. All notebook hard gates passed, including the exact numerical/missing-value/ordering probe and fresh-pass idempotence. + +Positive implicit triggering was 10/10. One of ten negative prompts—a notebook feature request—falsely triggered, meeting but not exceeding the 10% limit. + +## Notebook safety boundary + +Notebooks are not executed, cleared, regenerated, reordered, or normalized by default. Source edits are restricted to explicit cells or fields and require structural plus textual-locality validation. Stored outputs remain stored outputs; their presence does not imply recomputation after a source edit. + +The deterministic probe executes only selected trusted synthetic fixture code in isolated Python processes. It is not a security sandbox, notebook execution engine, or proof of general notebook equivalence. + +## Limitations and release status + +- No real notebook was supplied with explicit authorization. Required real-notebook review and controlled-edit acceptance remain pending. +- Scoring was manual side-by-side review, not an independent blind judge. +- ShellCheck and `pdflatex` were unavailable. Shell behavior contracts and static LaTeX contracts passed, but rendered LaTeX validation did not run. +- One notebook feature prompt falsely triggered the skill. +- Notebook structure preservation does not establish scientific correctness, output freshness, provenance completeness, or hidden-state safety. +- Jupyter is the only newly supported environment; R, MATLAB, Julia, and Fortran remain out of scope. +- Human scientific and maintainer review remains mandatory. +- The release remains experimental. + +Provisional verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING**. From 33442db8f41f990732ff5fdf10b28043752ee681 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 14:31:38 +0800 Subject: [PATCH 10/17] feat: rename skill to angze-code-style --- CHANGELOG.md | 4 +- GETTING_STARTED.md | 14 +-- README.md | 26 ++--- .../angze-code-style-v0.4.0.manifest.json | 105 ++++++++++++++++++ ...r-v0.4.0.md => angze-code-style-v0.4.0.md} | 15 ++- .../scientific-code-documenter-v0.1.0.md | 2 +- .../scientific-code-documenter-v0.2.0.md | 2 +- ...r-v0.4.0.md => angze-code-style-v0.4.0.md} | 16 ++- .../README.md | 6 +- .../checks.sh | 2 +- .../adversarial/ambiguous_units_and_roles.py | 0 .../fixtures/adversarial/generated_client.py | 0 .../fixtures/adversarial/protected_cases.py | 0 .../review_only_stale_comment/prompts.yaml | 0 .../review_only_transform.py | 0 .../test_review_only_transform.py | 0 .../adversarial/test_protected_cases.py | 0 .../fixtures/adversarial/vendor_curve.js | 0 .../fixtures/contracts/test_html_fixtures.py | 0 .../fixtures/contracts/test_latex_fixtures.py | 0 .../fixtures/contracts/test_shell_fixtures.py | 0 .../contracts/xps_dashboard_harness.js | 0 .../fixtures/cross-language-contracts.md | 0 .../fixtures/html/cramped_xps_dashboard.html | 0 .../html/generated_report_snapshot.html | 0 .../fixtures/html/index.html | 0 .../fixtures/html/styles.css | 0 .../fixtures/html/synthetic-spectrum.svg | 0 .../fixtures/html/validate_html.py | 0 .../fixtures/latex/Makefile | 0 .../latex/crowded_scientific_report.tex | 0 .../latex/fragile_template_snippet.tex | 0 .../fixtures/latex/report.tex | 0 .../editable_scientific.contract.json | 0 .../notebook/editable_scientific.ipynb | 0 .../notebook/editable_scientific.prompt.txt | 0 .../generated_pipeline_report.contract.json | 0 .../notebook/generated_pipeline_report.ipynb | 0 .../generated_pipeline_report.prompt.txt | 0 .../notebook/over_commented.contract.json | 0 .../fixtures/notebook/over_commented.ipynb | 0 .../notebook/over_commented.prompt.txt | 0 .../review_only_stale_claim.contract.json | 0 .../notebook/review_only_stale_claim.ipynb | 0 .../review_only_stale_claim.prompt.txt | 0 .../notebook/test_notebook_contract.py | 0 .../notebook/validate_notebook_contract.py | 0 .../cramped/cramped_analysis_pipeline.py | 0 .../spectroscopy_pipeline_documentation.py | 0 .../photocatalysis_analysis_workflow.py | 0 .../quantum/quantum_chemistry_workflow.py | 0 .../restraint/dangerous_restraint_test.py | 0 .../fixtures/python/spectrum_pipeline.py | 0 .../fixtures/python/test_spectrum_pipeline.py | 0 .../fixtures/shell/cramped_orca_batch.sh | 0 .../fixtures/shell/review_only_publish.sh | 0 .../fixtures/shell/run_campaign.sh | 0 .../prompts.yaml | 18 +-- .../results/.gitkeep | 0 .../results/result.template.md | 2 +- .../rubric.md | 0 .../SKILL.md | 4 +- skills/angze-code-style/agents/openai.yaml | 4 + .../references/comment-examples.md | 0 .../references/jupyter-notebooks.md | 0 .../references/scientific-contracts.md | 0 .../agents/openai.yaml | 4 - 67 files changed, 173 insertions(+), 51 deletions(-) create mode 100644 docs/evaluations/angze-code-style-v0.4.0.manifest.json rename docs/evaluations/{scientific-code-documenter-v0.4.0.md => angze-code-style-v0.4.0.md} (80%) rename docs/releases/{scientific-code-documenter-v0.4.0.md => angze-code-style-v0.4.0.md} (67%) rename evals/{scientific-code-documenter => angze-code-style}/README.md (95%) rename evals/{scientific-code-documenter => angze-code-style}/checks.sh (96%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/ambiguous_units_and_roles.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/generated_client.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/protected_cases.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/review_only_stale_comment/prompts.yaml (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/review_only_stale_comment/review_only_transform.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/review_only_stale_comment/test_review_only_transform.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/test_protected_cases.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/adversarial/vendor_curve.js (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/contracts/test_html_fixtures.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/contracts/test_latex_fixtures.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/contracts/test_shell_fixtures.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/contracts/xps_dashboard_harness.js (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/cross-language-contracts.md (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/html/cramped_xps_dashboard.html (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/html/generated_report_snapshot.html (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/html/index.html (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/html/styles.css (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/html/synthetic-spectrum.svg (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/html/validate_html.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/latex/Makefile (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/latex/crowded_scientific_report.tex (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/latex/fragile_template_snippet.tex (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/latex/report.tex (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/editable_scientific.contract.json (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/editable_scientific.ipynb (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/editable_scientific.prompt.txt (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/generated_pipeline_report.contract.json (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/generated_pipeline_report.ipynb (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/generated_pipeline_report.prompt.txt (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/over_commented.contract.json (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/over_commented.ipynb (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/over_commented.prompt.txt (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/review_only_stale_claim.contract.json (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/review_only_stale_claim.ipynb (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/review_only_stale_claim.prompt.txt (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/test_notebook_contract.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/notebook/validate_notebook_contract.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/cramped/cramped_analysis_pipeline.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/documentation/spectroscopy_pipeline_documentation.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/photocatalysis/photocatalysis_analysis_workflow.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/quantum/quantum_chemistry_workflow.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/restraint/dangerous_restraint_test.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/spectrum_pipeline.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/python/test_spectrum_pipeline.py (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/shell/cramped_orca_batch.sh (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/shell/review_only_publish.sh (100%) rename evals/{scientific-code-documenter => angze-code-style}/fixtures/shell/run_campaign.sh (100%) rename evals/{scientific-code-documenter => angze-code-style}/prompts.yaml (89%) rename evals/{scientific-code-documenter => angze-code-style}/results/.gitkeep (100%) rename evals/{scientific-code-documenter => angze-code-style}/results/result.template.md (96%) rename evals/{scientific-code-documenter => angze-code-style}/rubric.md (100%) rename skills/{scientific-code-documenter => angze-code-style}/SKILL.md (99%) create mode 100644 skills/angze-code-style/agents/openai.yaml rename skills/{scientific-code-documenter => angze-code-style}/references/comment-examples.md (100%) rename skills/{scientific-code-documenter => angze-code-style}/references/jupyter-notebooks.md (100%) rename skills/{scientific-code-documenter => angze-code-style}/references/scientific-contracts.md (100%) delete mode 100644 skills/scientific-code-documenter/agents/openai.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index bbc06b7..0b26a30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ No changes yet. ### Changed +- Renamed the active skill from `scientific-code-documenter` to `angze-code-style` as a clean break; the old invocation is not retained as an alias. - Runtime behavior now covers scientific contracts, notebook artifacts, pruning, and explicit preservation boundaries. - Controlled A/B evaluation now includes notebooks plus Python, Shell, LaTeX, and HTML regressions. - Historical v0.1–v0.3 results remain predecessor evidence rather than acceptance of the changed runtime. @@ -29,11 +30,12 @@ No changes yet. - Editable, review-only, pruning, and generated notebook hard gates, including exact probe output and byte-level locality. - Six accepted editing candidates remained byte-identical on fresh second passes. - Positive implicit triggering at 10/10 and false triggering at 1/10. +- Full post-rename trigger rerun at 10/10 positives and 0/10 negatives, plus one renamed editable-notebook validation and byte-identical fresh second pass. ### Known limitations - Explicitly authorized real-notebook acceptance remains pending. -- One notebook feature-development prompt falsely triggered the skill. +- One notebook feature-development prompt falsely triggered the skill in the original matrix; the post-rename rerun did not reproduce it. - Manual A/B scoring was not independently blind. - ShellCheck and rendered LaTeX compilation were unavailable in the release environment. - Deterministic fixture probes do not prove general notebook equivalence or scientific correctness. diff --git a/GETTING_STARTED.md b/GETTING_STARTED.md index b73a809..8a573f0 100644 --- a/GETTING_STARTED.md +++ b/GETTING_STARTED.md @@ -1,6 +1,6 @@ # Getting started -This guide shows how to install and use the `scientific-code-documenter` skill in Codex. The skill helps review, document, and prune scientific code or Jupyter notebooks while preserving explicit scientific, data, interface, execution-state, and artifact contracts. +This guide shows how to install and use the `angze-code-style` skill in Codex. The skill helps review, document, and prune scientific code or Jupyter notebooks while preserving explicit scientific, data, interface, execution-state, and artifact contracts. ## What you need @@ -15,7 +15,7 @@ Open a new Codex task and paste this prompt: ```text $skill-installer -Install https://github.com/angzeli/codex-skills/tree/main/skills/scientific-code-documenter +Install https://github.com/angzeli/codex-skills/tree/main/skills/angze-code-style ``` The skill should be available on your next turn. If it does not appear, start a new task or restart Codex. @@ -25,14 +25,14 @@ The skill should be available on your next turn. If it does not appear, start a For a review that does not modify files: ```text -$scientific-code-documenter +$angze-code-style Review this file for readability and documentation quality. Do not modify it. ``` For focused documentation and readability improvements: ```text -$scientific-code-documenter +$angze-code-style Apply only high-confidence documentation and readability improvements to this file. Preserve calculations, public interfaces, file formats, and numerical behavior. ``` @@ -41,7 +41,7 @@ Attach the file, open its repository in Codex, or name the file path in the prom For a conservative notebook review: ```text -$scientific-code-documenter +$angze-code-style Review this notebook's role, scientific and data contracts, execution state, outputs, and documentation risks. Do not modify or execute it. ``` @@ -56,7 +56,7 @@ This alternative is useful for contributors or anyone who wants updates in a loc ```sh git clone https://github.com/angzeli/codex-skills.git cd codex-skills -./scripts/install_skill.sh scientific-code-documenter +./scripts/install_skill.sh angze-code-style ``` The script validates the skill and links it into `$HOME/.agents/skills`. Start a new Codex task after the first installation. To update later, pull the latest repository changes; the link will continue to use the checked-out skill. @@ -65,7 +65,7 @@ The script validates the skill and links it into `$HOME/.agents/skills`. Start a - **The skill is not listed:** Start a new task or restart Codex after installation. - **GitHub access fails:** Confirm that you can open or clone the repository, especially if it is private. -- **The manual installer will not run:** From the repository root, try `bash ./scripts/install_skill.sh scientific-code-documenter`. +- **The manual installer will not run:** From the repository root, try `bash ./scripts/install_skill.sh angze-code-style`. - **You only want feedback:** Include `Do not modify files` in the prompt. For maintainer commands, validation details, and evaluation status, see the main [README](README.md). diff --git a/README.md b/README.md index 4cf7e99..a7919bb 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ This repository is the editable source of truth for a growing collection of custom Codex skills. Each direct child of `skills/` is an independent runtime package; collection documentation, evaluation suites, and shared tooling stay outside those packages. -New to Codex skills? Follow [Getting started](GETTING_STARTED.md) to install and use `scientific-code-documenter`. +New to Codex skills? Follow [Getting started](GETTING_STARTED.md) to install and use `angze-code-style`. ## Source and runtime installations @@ -12,7 +12,7 @@ Edit skills in this repository. Runtime discovery uses one symbolic link per ski codex_skills/ ├── skills/ │ ├── README.md -│ └── scientific-code-documenter/ +│ └── angze-code-style/ │ ├── SKILL.md │ ├── agents/openai.yaml │ └── references/ @@ -21,7 +21,7 @@ codex_skills/ │ └── scientific-contracts.md ├── evals/ │ ├── README.md -│ └── scientific-code-documenter/ +│ └── angze-code-style/ │ ├── fixtures/ │ ├── results/ │ ├── checks.sh @@ -40,7 +40,7 @@ codex_skills/ | Skill | Purpose | Status | | --- | --- | --- | -| `scientific-code-documenter` | Document and prune scientific code and Jupyter notebooks using explicit contract inventory, evidence classes, and risk grading | Experimental v0.4.0 release candidate; synthetic notebook and controlled Python, Shell, LaTeX, HTML, trigger, restraint, and idempotence gates pass; authorized real-notebook acceptance is pending | +| `angze-code-style` | Document and prune scientific code and Jupyter notebooks using explicit contract inventory, evidence classes, and risk grading | Experimental v0.4.0 release candidate; synthetic notebook and controlled Python, Shell, LaTeX, HTML, trigger, restraint, and idempotence gates pass; authorized real-notebook acceptance is pending | ## Validate and install @@ -48,12 +48,12 @@ The collection-level commands work from any checkout path: ```sh ./scripts/list_skills.sh -./scripts/check_skill.sh scientific-code-documenter +./scripts/check_skill.sh angze-code-style ./scripts/check_all_skills.sh -./scripts/run_fixture_checks.sh scientific-code-documenter +./scripts/run_fixture_checks.sh angze-code-style -./scripts/install_skill.sh --dry-run scientific-code-documenter -./scripts/install_skill.sh scientific-code-documenter +./scripts/install_skill.sh --dry-run angze-code-style +./scripts/install_skill.sh angze-code-style ./scripts/install_all_skills.sh --dry-run ./scripts/install_all_skills.sh ``` @@ -70,10 +70,10 @@ Installers derive the repository root from their own location and create individ ## Invoke a skill -Use `scientific-code-documenter` for review-only documentation assessments, focused docstring and comment work, or behavior-preserving readability improvements. Invoke it explicitly for a review-only request: +Use `angze-code-style` for review-only documentation assessments, focused docstring and comment work, or behavior-preserving readability improvements. Invoke it explicitly for a review-only request: ```text -$scientific-code-documenter +$angze-code-style Review this file for readability and documentation quality. Do not modify it. ``` @@ -90,7 +90,7 @@ Codex may also select it implicitly for requests such as: For an editing-mode request: ```text -$scientific-code-documenter +$angze-code-style Apply only high-confidence documentation and readability improvements to this file. Preserve calculations, public interfaces, file formats, and numerical behaviour. ``` @@ -116,9 +116,9 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for the complete workflow. Structural validation proves that a skill can be discovered and parsed; it does not prove usefulness. Evaluation therefore combines controlled A/B runs, trigger and scope-boundary measurements, independent behavior checks, adversarial fixtures, and authorized real-repository acceptance. -`scientific-code-documenter` v0.4.0 changes runtime behavior. It adds scientific/artifact contract inventory, `evidence-backed` / `observable-only` / `unknown` classification, Tier 0–3 risk grading, structured review findings, conservative notebook policy, comment pruning, and repeated-pass stability. +`angze-code-style` v0.4.0 changes runtime behavior and completes a clean-break rename from `scientific-code-documenter`. It adds scientific/artifact contract inventory, `evidence-backed` / `observable-only` / `unknown` classification, Tier 0–3 risk grading, structured review findings, conservative notebook policy, comment pruning, and repeated-pass stability. -Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. Positive implicit triggering was 10/10; one of ten negative prompts falsely triggered. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable, so Shell syntax/dynamic contracts and static LaTeX contracts were used. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/scientific-code-documenter-v0.4.0.md). +Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. The initial trigger matrix scored 10/10 positives with one false trigger; the full post-rename rerun scored 10/10 positives and 0/10 negatives. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable, so Shell syntax/dynamic contracts and static LaTeX contracts were used. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/angze-code-style-v0.4.0.md) and [integrity manifest](docs/evaluations/angze-code-style-v0.4.0.manifest.json). Historical v0.1–v0.3 scientific Python and ORCA Shell evidence remains useful predecessor evidence, but it is not accepted as proof for the changed v0.4 runtime. No real notebook was supplied with explicit authorization, so real-notebook acceptance remains pending. The skill remains experimental; limited probes do not prove notebook equivalence or scientific correctness, and human scientific review remains mandatory. diff --git a/docs/evaluations/angze-code-style-v0.4.0.manifest.json b/docs/evaluations/angze-code-style-v0.4.0.manifest.json new file mode 100644 index 0000000..0abc224 --- /dev/null +++ b/docs/evaluations/angze-code-style-v0.4.0.manifest.json @@ -0,0 +1,105 @@ +{ + "schema_version": 1, + "skill": "angze-code-style", + "version": "0.4.0", + "release_date": "2026-08-28", + "runtime_snapshot": { + "algorithm": "sha256 of tracked runtime files as C-sorted LF-delimited file_sha256, two spaces, and runtime-relative path records", + "sha256": "7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84", + "files": [ + { + "path": "skills/angze-code-style/SKILL.md", + "sha256": "5e1c49383585d6aa1f5694e1fc7add25e4b7ccf3e3a62185606b685b3b556851", + "bytes": 23187 + }, + { + "path": "skills/angze-code-style/agents/openai.yaml", + "sha256": "767b50465dd17eb490efa1d4a8569803c597d03247bb60b0f4818c5ec67f8690", + "bytes": 253 + }, + { + "path": "skills/angze-code-style/references/comment-examples.md", + "sha256": "55eff6a16876d47b97670afe5b681d9b4735181cbb5d9e7d9f63530001e41c3b", + "bytes": 6217 + }, + { + "path": "skills/angze-code-style/references/jupyter-notebooks.md", + "sha256": "24dfddd405d91e586447027b386919b44274bc0c00bc96e0c35ff2d4f50975c5", + "bytes": 2996 + }, + { + "path": "skills/angze-code-style/references/scientific-contracts.md", + "sha256": "4d9837faf31d7d5e23f1fd7ba5703546a5a9a7a249ce5122455ae0a8ccc45948", + "bytes": 2588 + } + ] + }, + "notebook_acceptance_fixtures": { + "scope": "tracked synthetic fixtures, contracts, validator, and validator tests", + "files": [ + { + "path": "evals/angze-code-style/fixtures/notebook/editable_scientific.contract.json", + "sha256": "56ee49c547dfe3fdcd84226fb256976c3381cc8a91a333cfb69fdc130bb6b046", + "bytes": 1009 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/editable_scientific.ipynb", + "sha256": "114daea3eb4d15611bd66414e879f12cc20e28a90d949be1ab7d3eecf1919707", + "bytes": 1877 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/generated_pipeline_report.contract.json", + "sha256": "7a80ca5ed2f16380401c22d34beb70ae2bd028125327f9d8ba97f970b9e26069", + "bytes": 209 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/generated_pipeline_report.ipynb", + "sha256": "72e5a312bb04c6ab02724ec4cb611b882d37763d7aafd5993014f2ff6389462a", + "bytes": 874 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/over_commented.contract.json", + "sha256": "25f2136a7f64b98adaf69b7bd58f2462b2855ada25d94d65e9ce28673be01ad3", + "bytes": 620 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/over_commented.ipynb", + "sha256": "dcab864eaee5cdd365147f41e5e11a3e91888416060595d67a395664df3a6d9d", + "bytes": 1208 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/review_only_stale_claim.contract.json", + "sha256": "62f79d2b99dca0e4dac4599ada7c80d7744915477b92287994f9a49454f9ce71", + "bytes": 217 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/review_only_stale_claim.ipynb", + "sha256": "bfa6ca039930bcc922084d801ade9f6456fbcb548309034ec7894cbff97a6fd7", + "bytes": 748 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/test_notebook_contract.py", + "sha256": "1a04595fc2d1ae8fb8c92c15d3a56f794f331fea79c98e3337696d71dd4c4147", + "bytes": 14247 + }, + { + "path": "evals/angze-code-style/fixtures/notebook/validate_notebook_contract.py", + "sha256": "cfd8e33838c36acd3fad34a350c975c11035fad1efb82bc10f0e426e651855c9", + "bytes": 23033 + } + ] + }, + "controlled_results": { + "editing_pairs": 6, + "skill_wins": 6, + "skill_score_average": 17.5, + "skill_score_minimum": 17, + "behavior_preservation_score": "3/3 for every accepted editing fixture", + "accepted_second_passes_byte_identical": "6/6", + "post_rename_positive_triggers": "10/10", + "post_rename_negative_triggers": "0/10", + "post_rename_representative_notebook_validation": "pass", + "post_rename_representative_second_pass_byte_identical": true, + "real_notebook_acceptance": "pending explicit authorization" + } +} diff --git a/docs/evaluations/scientific-code-documenter-v0.4.0.md b/docs/evaluations/angze-code-style-v0.4.0.md similarity index 80% rename from docs/evaluations/scientific-code-documenter-v0.4.0.md rename to docs/evaluations/angze-code-style-v0.4.0.md index d674931..14bc17d 100644 --- a/docs/evaluations/scientific-code-documenter-v0.4.0.md +++ b/docs/evaluations/angze-code-style-v0.4.0.md @@ -1,12 +1,15 @@ -# scientific-code-documenter v0.4.0 validation +# angze-code-style v0.4.0 validation ## Version status -- Version: `scientific-code-documenter` v0.4.0, before the final `angze-code-style` migration +- Version: `angze-code-style` v0.4.0 +- Migration: clean-break rename from `scientific-code-documenter`; no compatibility alias - Release status: experimental local release candidate - Runtime behavior changed from v0.3.0: yes - Real-notebook acceptance: pending explicit authorization -- Provisional verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING** +- Verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING** +- Tracked runtime snapshot SHA-256: `7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84` +- Integrity manifest: [`angze-code-style-v0.4.0.manifest.json`](angze-code-style-v0.4.0.manifest.json) ## Methodology @@ -53,6 +56,12 @@ Ten positive implicit prompts covered Python, Shell, LaTeX, HTML, notebook docum Thus positive accuracy was 100% and false triggering was 10%, meeting the release thresholds. The feature false positive is retained as a known limitation. +After the clean-break rename, the same full 20-prompt matrix was rerun in fresh isolated sessions. All twenty sessions were valid: positives were 10/10 and negative activations were 0/10. The global skill inventory matched exactly after restoration. An initial path-sensitive detector undercount was corrected over the saved JSONL because the CLI resolved the isolated symlink to `runtime-snapshot/SKILL.md`; no session was rerun or replaced. + +## Post-rename representative check + +One explicit `$angze-code-style` editable-notebook run changed only the allowlisted source values and passed the strict notebook contract validator. A fresh second session read the same frozen runtime and left the first candidate byte-identical; the validator passed again. This is a rename-sensitive synthetic migration check, not real-notebook acceptance. + ## Retained failed attempts - One orchestration attempt used incompatible current CLI flags and exited before model launch; no candidate changed. diff --git a/docs/evaluations/scientific-code-documenter-v0.1.0.md b/docs/evaluations/scientific-code-documenter-v0.1.0.md index 3214fa8..eb716e0 100644 --- a/docs/evaluations/scientific-code-documenter-v0.1.0.md +++ b/docs/evaluations/scientific-code-documenter-v0.1.0.md @@ -57,7 +57,7 @@ The skill source, target source, global skill state, and prior evaluation eviden Evaluation used neutral repositories and public-safe synthetic fixtures, fresh ephemeral candidate sessions, isolated skill discovery, and identical controlled A/B prompts except for explicit skill invocation. Outputs were judged blind to condition, and behavior was checked independently of qualitative scoring. Real-repository candidates came from immutable, remote-free source and skill snapshots so acceptance could be tied to one exact runtime hash. -Tracked materials are available in the [evaluation overview](../../evals/scientific-code-documenter/README.md), [prompt matrix](../../evals/scientific-code-documenter/prompts.yaml), [scoring rubric](../../evals/scientific-code-documenter/rubric.md), and [synthetic fixtures](../../evals/scientific-code-documenter/fixtures/). Detailed raw runs remain local, ignored audit evidence and are not part of the release. +Tracked materials are available in the [evaluation overview](../../evals/angze-code-style/README.md), [prompt matrix](../../evals/angze-code-style/prompts.yaml), [scoring rubric](../../evals/angze-code-style/rubric.md), and [synthetic fixtures](../../evals/angze-code-style/fixtures/). Detailed raw runs remain local, ignored audit evidence and are not part of the release. ## Known limitations diff --git a/docs/evaluations/scientific-code-documenter-v0.2.0.md b/docs/evaluations/scientific-code-documenter-v0.2.0.md index 1464e85..fd69946 100644 --- a/docs/evaluations/scientific-code-documenter-v0.2.0.md +++ b/docs/evaluations/scientific-code-documenter-v0.2.0.md @@ -97,7 +97,7 @@ No new five-fixture Python A/B or real-repository acceptance run was required be The six controlled fixtures are synthetic and public-safe. Each pair started from one neutral commit and used fresh ephemeral sessions with the same model, reasoning effort, prompt body, sandbox, CLI, and independent post-run contract. The comparison differed only by explicit skill invocation. Skill arms used an immutable runtime snapshot and recorded a skill read; baseline and blind-judge conditions exposed no user skills. Candidate mappings were randomized and hidden until structured judge output passed validation. -Tracked materials include the [evaluation overview](../../evals/scientific-code-documenter/README.md), [prompt matrix](../../evals/scientific-code-documenter/prompts.yaml), [rubric](../../evals/scientific-code-documenter/rubric.md), [fixture contracts](../../evals/scientific-code-documenter/fixtures/cross-language-contracts.md), and [synthetic fixtures](../../evals/scientific-code-documenter/fixtures/). Raw model events, diffs, mappings, and generated outputs remain ignored local audit evidence. +Tracked materials include the [evaluation overview](../../evals/angze-code-style/README.md), [prompt matrix](../../evals/angze-code-style/prompts.yaml), [rubric](../../evals/angze-code-style/rubric.md), [fixture contracts](../../evals/angze-code-style/fixtures/cross-language-contracts.md), and [synthetic fixtures](../../evals/angze-code-style/fixtures/). Raw model events, diffs, mappings, and generated outputs remain ignored local audit evidence. ## Limitations diff --git a/docs/releases/scientific-code-documenter-v0.4.0.md b/docs/releases/angze-code-style-v0.4.0.md similarity index 67% rename from docs/releases/scientific-code-documenter-v0.4.0.md rename to docs/releases/angze-code-style-v0.4.0.md index 0ee6c4b..f8171dd 100644 --- a/docs/releases/scientific-code-documenter-v0.4.0.md +++ b/docs/releases/angze-code-style-v0.4.0.md @@ -1,10 +1,10 @@ -# scientific-code-documenter v0.4.0 +# angze-code-style v0.4.0 Release preparation date: 2026-08-28 This experimental release candidate changes runtime behavior. It upgrades scientific documentation from style guidance to contract-aware, risk-graded work and adds conservative first-class support for Jupyter Notebook artifacts. -The final v0.4 package will be renamed to `angze-code-style` after the pre-rename implementation, evaluation, and local release preparation are stable. Final renamed runtime and manifest hashes will be recorded after that migration. +The active package is now `angze-code-style`, a clean-break rename from `scientific-code-documenter`. The authoritative invocation is `$angze-code-style`; no compatibility skill or alias is included. ## Added @@ -21,7 +21,13 @@ The final v0.4 package will be renamed to `angze-code-style` after the pre-renam Six editing comparisons averaged 17.5/18, with a minimum of 17/18, behavior preservation at 3/3 for every accepted fixture, and six skill wins. Review-only and generated notebooks were byte-identical ties. All notebook hard gates passed, including the exact numerical/missing-value/ordering probe and fresh-pass idempotence. -Positive implicit triggering was 10/10. One of ten negative prompts—a notebook feature request—falsely triggered, meeting but not exceeding the 10% limit. +Positive implicit triggering was 10/10. One of ten negative prompts—a notebook feature request—falsely triggered in the original controlled matrix, meeting but not exceeding the 10% limit. A full 20-prompt rerun after the clean-break rename scored 10/10 positives and 0/10 negatives. The previously observed false trigger remains a limitation rather than being discarded. + +One renamed editable-notebook candidate then passed the strict contract validator, and a fresh second session produced a byte-identical candidate. Both sessions read the frozen `angze-code-style` runtime. + +## Integrity + +The tracked runtime snapshot SHA-256 is `7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84`. It covers the five tracked runtime files as sorted file-hash and runtime-relative-path records. The [machine-readable manifest](../evaluations/angze-code-style-v0.4.0.manifest.json) records the algorithm, per-file hashes and sizes, notebook fixtures, validator files, and accepted result summary without personal paths. ## Notebook safety boundary @@ -34,10 +40,10 @@ The deterministic probe executes only selected trusted synthetic fixture code in - No real notebook was supplied with explicit authorization. Required real-notebook review and controlled-edit acceptance remain pending. - Scoring was manual side-by-side review, not an independent blind judge. - ShellCheck and `pdflatex` were unavailable. Shell behavior contracts and static LaTeX contracts passed, but rendered LaTeX validation did not run. -- One notebook feature prompt falsely triggered the skill. +- One notebook feature prompt falsely triggered the skill in the original controlled matrix; a post-rename rerun did not reproduce it. - Notebook structure preservation does not establish scientific correctness, output freshness, provenance completeness, or hidden-state safety. - Jupyter is the only newly supported environment; R, MATLAB, Julia, and Fortran remain out of scope. - Human scientific and maintainer review remains mandatory. - The release remains experimental. -Provisional verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING**. +Verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING**. diff --git a/evals/scientific-code-documenter/README.md b/evals/angze-code-style/README.md similarity index 95% rename from evals/scientific-code-documenter/README.md rename to evals/angze-code-style/README.md index 6fe6b6b..9c30a9d 100644 --- a/evals/scientific-code-documenter/README.md +++ b/evals/angze-code-style/README.md @@ -1,4 +1,4 @@ -# Scientific Code Documenter evaluation +# Angze Code Style evaluation This suite tests whether the skill improves documentation and readability without changing behavior or expanding scope. It includes contract-aware Jupyter Notebook cases without treating notebook JSON as ordinary text. Tracked fixtures are synthetic and contain no unpublished research or third-party project code. Authorized real-repository runs retain raw source, diffs, and logs only in ignored local result directories; tracked summaries are sanitized. @@ -6,7 +6,7 @@ This suite tests whether the skill improves documentation and readability withou 1. Run collection structural validation to check frontmatter, naming, references, and portable runtime content. 2. Use `prompts.yaml` to test explicit invocation, positive implicit triggering, false triggers, and boundary decisions in fresh sessions. -3. Run `./scripts/run_fixture_checks.sh scientific-code-documenter` before and after each fixture edit. Notebook candidates must also pass their fixture-specific contract file. +3. Run `./scripts/run_fixture_checks.sh angze-code-style` before and after each fixture edit. Notebook candidates must also pass their fixture-specific contract file. 4. Compare baseline and skill-assisted diffs from identical starting commits using the same model and prompt. 5. Score both outputs with `rubric.md`; preserve raw diffs and command logs under `results/`. 6. Exercise adversarial and notebook fixtures to confirm that unknown meaning is not invented and generated, review-only, or protected notebook state remains untouched. @@ -30,7 +30,7 @@ Baseline fixtures are intentionally imperfect. A good edit may change their styl - Create sibling worktrees outside the main checkout at the same starting commit. - Start fresh Codex sessions with the same model and prompt. -- Run the baseline without explicit invocation and the comparison with `$scientific-code-documenter`. +- Run the baseline without explicit invocation and the comparison with `$angze-code-style`. - Compare the four notebook cases and representative Python, Shell, LaTeX, and HTML cases. Count the 75% win threshold across editing fixtures; both notebook editing fixtures must beat baseline. - Use write-enabled workspaces for review-only and generated pairs so restraint is demonstrated rather than enforced by permissions. These cases may tie, but the skill must not lose or modify a byte. - Save each diff and validation log, then score both independently. diff --git a/evals/scientific-code-documenter/checks.sh b/evals/angze-code-style/checks.sh similarity index 96% rename from evals/scientific-code-documenter/checks.sh rename to evals/angze-code-style/checks.sh index 1167939..f8f76c2 100755 --- a/evals/scientific-code-documenter/checks.sh +++ b/evals/angze-code-style/checks.sh @@ -48,7 +48,7 @@ else fi if command -v pdflatex >/dev/null 2>&1; then - tex_output=$(mktemp -d "${TMPDIR:-/tmp}/scientific-code-documenter-tex.XXXXXX") + tex_output=$(mktemp -d "${TMPDIR:-/tmp}/angze-code-style-tex.XXXXXX") run_required "LaTeX compilation" \ pdflatex -interaction=nonstopmode -halt-on-error -output-directory "$tex_output" "$eval_root/fixtures/latex/report.tex" else diff --git a/evals/scientific-code-documenter/fixtures/adversarial/ambiguous_units_and_roles.py b/evals/angze-code-style/fixtures/adversarial/ambiguous_units_and_roles.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/ambiguous_units_and_roles.py rename to evals/angze-code-style/fixtures/adversarial/ambiguous_units_and_roles.py diff --git a/evals/scientific-code-documenter/fixtures/adversarial/generated_client.py b/evals/angze-code-style/fixtures/adversarial/generated_client.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/generated_client.py rename to evals/angze-code-style/fixtures/adversarial/generated_client.py diff --git a/evals/scientific-code-documenter/fixtures/adversarial/protected_cases.py b/evals/angze-code-style/fixtures/adversarial/protected_cases.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/protected_cases.py rename to evals/angze-code-style/fixtures/adversarial/protected_cases.py diff --git a/evals/scientific-code-documenter/fixtures/adversarial/review_only_stale_comment/prompts.yaml b/evals/angze-code-style/fixtures/adversarial/review_only_stale_comment/prompts.yaml similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/review_only_stale_comment/prompts.yaml rename to evals/angze-code-style/fixtures/adversarial/review_only_stale_comment/prompts.yaml diff --git a/evals/scientific-code-documenter/fixtures/adversarial/review_only_stale_comment/review_only_transform.py b/evals/angze-code-style/fixtures/adversarial/review_only_stale_comment/review_only_transform.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/review_only_stale_comment/review_only_transform.py rename to evals/angze-code-style/fixtures/adversarial/review_only_stale_comment/review_only_transform.py diff --git a/evals/scientific-code-documenter/fixtures/adversarial/review_only_stale_comment/test_review_only_transform.py b/evals/angze-code-style/fixtures/adversarial/review_only_stale_comment/test_review_only_transform.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/review_only_stale_comment/test_review_only_transform.py rename to evals/angze-code-style/fixtures/adversarial/review_only_stale_comment/test_review_only_transform.py diff --git a/evals/scientific-code-documenter/fixtures/adversarial/test_protected_cases.py b/evals/angze-code-style/fixtures/adversarial/test_protected_cases.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/test_protected_cases.py rename to evals/angze-code-style/fixtures/adversarial/test_protected_cases.py diff --git a/evals/scientific-code-documenter/fixtures/adversarial/vendor_curve.js b/evals/angze-code-style/fixtures/adversarial/vendor_curve.js similarity index 100% rename from evals/scientific-code-documenter/fixtures/adversarial/vendor_curve.js rename to evals/angze-code-style/fixtures/adversarial/vendor_curve.js diff --git a/evals/scientific-code-documenter/fixtures/contracts/test_html_fixtures.py b/evals/angze-code-style/fixtures/contracts/test_html_fixtures.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/contracts/test_html_fixtures.py rename to evals/angze-code-style/fixtures/contracts/test_html_fixtures.py diff --git a/evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py b/evals/angze-code-style/fixtures/contracts/test_latex_fixtures.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/contracts/test_latex_fixtures.py rename to evals/angze-code-style/fixtures/contracts/test_latex_fixtures.py diff --git a/evals/scientific-code-documenter/fixtures/contracts/test_shell_fixtures.py b/evals/angze-code-style/fixtures/contracts/test_shell_fixtures.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/contracts/test_shell_fixtures.py rename to evals/angze-code-style/fixtures/contracts/test_shell_fixtures.py diff --git a/evals/scientific-code-documenter/fixtures/contracts/xps_dashboard_harness.js b/evals/angze-code-style/fixtures/contracts/xps_dashboard_harness.js similarity index 100% rename from evals/scientific-code-documenter/fixtures/contracts/xps_dashboard_harness.js rename to evals/angze-code-style/fixtures/contracts/xps_dashboard_harness.js diff --git a/evals/scientific-code-documenter/fixtures/cross-language-contracts.md b/evals/angze-code-style/fixtures/cross-language-contracts.md similarity index 100% rename from evals/scientific-code-documenter/fixtures/cross-language-contracts.md rename to evals/angze-code-style/fixtures/cross-language-contracts.md diff --git a/evals/scientific-code-documenter/fixtures/html/cramped_xps_dashboard.html b/evals/angze-code-style/fixtures/html/cramped_xps_dashboard.html similarity index 100% rename from evals/scientific-code-documenter/fixtures/html/cramped_xps_dashboard.html rename to evals/angze-code-style/fixtures/html/cramped_xps_dashboard.html diff --git a/evals/scientific-code-documenter/fixtures/html/generated_report_snapshot.html b/evals/angze-code-style/fixtures/html/generated_report_snapshot.html similarity index 100% rename from evals/scientific-code-documenter/fixtures/html/generated_report_snapshot.html rename to evals/angze-code-style/fixtures/html/generated_report_snapshot.html diff --git a/evals/scientific-code-documenter/fixtures/html/index.html b/evals/angze-code-style/fixtures/html/index.html similarity index 100% rename from evals/scientific-code-documenter/fixtures/html/index.html rename to evals/angze-code-style/fixtures/html/index.html diff --git a/evals/scientific-code-documenter/fixtures/html/styles.css b/evals/angze-code-style/fixtures/html/styles.css similarity index 100% rename from evals/scientific-code-documenter/fixtures/html/styles.css rename to evals/angze-code-style/fixtures/html/styles.css diff --git a/evals/scientific-code-documenter/fixtures/html/synthetic-spectrum.svg b/evals/angze-code-style/fixtures/html/synthetic-spectrum.svg similarity index 100% rename from evals/scientific-code-documenter/fixtures/html/synthetic-spectrum.svg rename to evals/angze-code-style/fixtures/html/synthetic-spectrum.svg diff --git a/evals/scientific-code-documenter/fixtures/html/validate_html.py b/evals/angze-code-style/fixtures/html/validate_html.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/html/validate_html.py rename to evals/angze-code-style/fixtures/html/validate_html.py diff --git a/evals/scientific-code-documenter/fixtures/latex/Makefile b/evals/angze-code-style/fixtures/latex/Makefile similarity index 100% rename from evals/scientific-code-documenter/fixtures/latex/Makefile rename to evals/angze-code-style/fixtures/latex/Makefile diff --git a/evals/scientific-code-documenter/fixtures/latex/crowded_scientific_report.tex b/evals/angze-code-style/fixtures/latex/crowded_scientific_report.tex similarity index 100% rename from evals/scientific-code-documenter/fixtures/latex/crowded_scientific_report.tex rename to evals/angze-code-style/fixtures/latex/crowded_scientific_report.tex diff --git a/evals/scientific-code-documenter/fixtures/latex/fragile_template_snippet.tex b/evals/angze-code-style/fixtures/latex/fragile_template_snippet.tex similarity index 100% rename from evals/scientific-code-documenter/fixtures/latex/fragile_template_snippet.tex rename to evals/angze-code-style/fixtures/latex/fragile_template_snippet.tex diff --git a/evals/scientific-code-documenter/fixtures/latex/report.tex b/evals/angze-code-style/fixtures/latex/report.tex similarity index 100% rename from evals/scientific-code-documenter/fixtures/latex/report.tex rename to evals/angze-code-style/fixtures/latex/report.tex diff --git a/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.contract.json b/evals/angze-code-style/fixtures/notebook/editable_scientific.contract.json similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/editable_scientific.contract.json rename to evals/angze-code-style/fixtures/notebook/editable_scientific.contract.json diff --git a/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.ipynb b/evals/angze-code-style/fixtures/notebook/editable_scientific.ipynb similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/editable_scientific.ipynb rename to evals/angze-code-style/fixtures/notebook/editable_scientific.ipynb diff --git a/evals/scientific-code-documenter/fixtures/notebook/editable_scientific.prompt.txt b/evals/angze-code-style/fixtures/notebook/editable_scientific.prompt.txt similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/editable_scientific.prompt.txt rename to evals/angze-code-style/fixtures/notebook/editable_scientific.prompt.txt diff --git a/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.contract.json b/evals/angze-code-style/fixtures/notebook/generated_pipeline_report.contract.json similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.contract.json rename to evals/angze-code-style/fixtures/notebook/generated_pipeline_report.contract.json diff --git a/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.ipynb b/evals/angze-code-style/fixtures/notebook/generated_pipeline_report.ipynb similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.ipynb rename to evals/angze-code-style/fixtures/notebook/generated_pipeline_report.ipynb diff --git a/evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.prompt.txt b/evals/angze-code-style/fixtures/notebook/generated_pipeline_report.prompt.txt similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/generated_pipeline_report.prompt.txt rename to evals/angze-code-style/fixtures/notebook/generated_pipeline_report.prompt.txt diff --git a/evals/scientific-code-documenter/fixtures/notebook/over_commented.contract.json b/evals/angze-code-style/fixtures/notebook/over_commented.contract.json similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/over_commented.contract.json rename to evals/angze-code-style/fixtures/notebook/over_commented.contract.json diff --git a/evals/scientific-code-documenter/fixtures/notebook/over_commented.ipynb b/evals/angze-code-style/fixtures/notebook/over_commented.ipynb similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/over_commented.ipynb rename to evals/angze-code-style/fixtures/notebook/over_commented.ipynb diff --git a/evals/scientific-code-documenter/fixtures/notebook/over_commented.prompt.txt b/evals/angze-code-style/fixtures/notebook/over_commented.prompt.txt similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/over_commented.prompt.txt rename to evals/angze-code-style/fixtures/notebook/over_commented.prompt.txt diff --git a/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.contract.json b/evals/angze-code-style/fixtures/notebook/review_only_stale_claim.contract.json similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.contract.json rename to evals/angze-code-style/fixtures/notebook/review_only_stale_claim.contract.json diff --git a/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.ipynb b/evals/angze-code-style/fixtures/notebook/review_only_stale_claim.ipynb similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.ipynb rename to evals/angze-code-style/fixtures/notebook/review_only_stale_claim.ipynb diff --git a/evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.prompt.txt b/evals/angze-code-style/fixtures/notebook/review_only_stale_claim.prompt.txt similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/review_only_stale_claim.prompt.txt rename to evals/angze-code-style/fixtures/notebook/review_only_stale_claim.prompt.txt diff --git a/evals/scientific-code-documenter/fixtures/notebook/test_notebook_contract.py b/evals/angze-code-style/fixtures/notebook/test_notebook_contract.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/test_notebook_contract.py rename to evals/angze-code-style/fixtures/notebook/test_notebook_contract.py diff --git a/evals/scientific-code-documenter/fixtures/notebook/validate_notebook_contract.py b/evals/angze-code-style/fixtures/notebook/validate_notebook_contract.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/notebook/validate_notebook_contract.py rename to evals/angze-code-style/fixtures/notebook/validate_notebook_contract.py diff --git a/evals/scientific-code-documenter/fixtures/python/cramped/cramped_analysis_pipeline.py b/evals/angze-code-style/fixtures/python/cramped/cramped_analysis_pipeline.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/cramped/cramped_analysis_pipeline.py rename to evals/angze-code-style/fixtures/python/cramped/cramped_analysis_pipeline.py diff --git a/evals/scientific-code-documenter/fixtures/python/documentation/spectroscopy_pipeline_documentation.py b/evals/angze-code-style/fixtures/python/documentation/spectroscopy_pipeline_documentation.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/documentation/spectroscopy_pipeline_documentation.py rename to evals/angze-code-style/fixtures/python/documentation/spectroscopy_pipeline_documentation.py diff --git a/evals/scientific-code-documenter/fixtures/python/photocatalysis/photocatalysis_analysis_workflow.py b/evals/angze-code-style/fixtures/python/photocatalysis/photocatalysis_analysis_workflow.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/photocatalysis/photocatalysis_analysis_workflow.py rename to evals/angze-code-style/fixtures/python/photocatalysis/photocatalysis_analysis_workflow.py diff --git a/evals/scientific-code-documenter/fixtures/python/quantum/quantum_chemistry_workflow.py b/evals/angze-code-style/fixtures/python/quantum/quantum_chemistry_workflow.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/quantum/quantum_chemistry_workflow.py rename to evals/angze-code-style/fixtures/python/quantum/quantum_chemistry_workflow.py diff --git a/evals/scientific-code-documenter/fixtures/python/restraint/dangerous_restraint_test.py b/evals/angze-code-style/fixtures/python/restraint/dangerous_restraint_test.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/restraint/dangerous_restraint_test.py rename to evals/angze-code-style/fixtures/python/restraint/dangerous_restraint_test.py diff --git a/evals/scientific-code-documenter/fixtures/python/spectrum_pipeline.py b/evals/angze-code-style/fixtures/python/spectrum_pipeline.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/spectrum_pipeline.py rename to evals/angze-code-style/fixtures/python/spectrum_pipeline.py diff --git a/evals/scientific-code-documenter/fixtures/python/test_spectrum_pipeline.py b/evals/angze-code-style/fixtures/python/test_spectrum_pipeline.py similarity index 100% rename from evals/scientific-code-documenter/fixtures/python/test_spectrum_pipeline.py rename to evals/angze-code-style/fixtures/python/test_spectrum_pipeline.py diff --git a/evals/scientific-code-documenter/fixtures/shell/cramped_orca_batch.sh b/evals/angze-code-style/fixtures/shell/cramped_orca_batch.sh similarity index 100% rename from evals/scientific-code-documenter/fixtures/shell/cramped_orca_batch.sh rename to evals/angze-code-style/fixtures/shell/cramped_orca_batch.sh diff --git a/evals/scientific-code-documenter/fixtures/shell/review_only_publish.sh b/evals/angze-code-style/fixtures/shell/review_only_publish.sh similarity index 100% rename from evals/scientific-code-documenter/fixtures/shell/review_only_publish.sh rename to evals/angze-code-style/fixtures/shell/review_only_publish.sh diff --git a/evals/scientific-code-documenter/fixtures/shell/run_campaign.sh b/evals/angze-code-style/fixtures/shell/run_campaign.sh similarity index 100% rename from evals/scientific-code-documenter/fixtures/shell/run_campaign.sh rename to evals/angze-code-style/fixtures/shell/run_campaign.sh diff --git a/evals/scientific-code-documenter/prompts.yaml b/evals/angze-code-style/prompts.yaml similarity index 89% rename from evals/scientific-code-documenter/prompts.yaml rename to evals/angze-code-style/prompts.yaml index 3a4d06f..43ef30f 100644 --- a/evals/scientific-code-documenter/prompts.yaml +++ b/evals/angze-code-style/prompts.yaml @@ -1,5 +1,5 @@ schema_version: 1 -skill: scientific-code-documenter +skill: angze-code-style positive_implicit: - id: positive-python-docstrings @@ -237,32 +237,32 @@ explicit_invocation: - id: explicit-review-only language: mixed mode: review-only - prompt: "$scientific-code-documenter\nReview this file for readability and documentation quality. Do not modify it. Identify the five highest-value improvements." + prompt: "$angze-code-style\nReview this file for readability and documentation quality. Do not modify it. Identify the five highest-value improvements." - id: explicit-limited-edit language: mixed mode: limited-edit - prompt: "$scientific-code-documenter\nApply only high-confidence documentation and readability improvements to this file. Preserve calculations, public interfaces, file formats, and numerical behaviour." + prompt: "$angze-code-style\nApply only high-confidence documentation and readability improvements to this file. Preserve calculations, public interfaces, file formats, and numerical behaviour." - id: explicit-new-function language: python mode: write-new-code - prompt: "$scientific-code-documenter\nImplement the requested function using this repository's conventions. Document its units, array shapes, assumptions, failure modes, and return values without over-commenting it." + prompt: "$angze-code-style\nImplement the requested function using this repository's conventions. Document its units, array shapes, assumptions, failure modes, and return values without over-commenting it." - id: explicit-shell-review language: shell mode: review-only - prompt: "$scientific-code-documenter\nReview this workflow for comment quality and unstated path assumptions. Do not edit or run it." + prompt: "$angze-code-style\nReview this workflow for comment quality and unstated path assumptions. Do not edit or run it." - id: explicit-html-limited language: html-css mode: limited-edit - prompt: "$scientific-code-documenter\nImprove only semantic structure, accessibility documentation, and opaque names in this report component; leave generated CSS untouched." + prompt: "$angze-code-style\nImprove only semantic structure, accessibility documentation, and opaque names in this report component; leave generated CSS untouched." - id: explicit-latex-review language: latex mode: review-only - prompt: "$scientific-code-documenter\nReview report.tex for source readability, label and macro contracts, and unsupported scientific interpretation. Do not modify it." + prompt: "$angze-code-style\nReview report.tex for source readability, label and macro contracts, and unsupported scientific interpretation. Do not modify it." - id: explicit-notebook-review language: jupyter mode: review-only - prompt: "$scientific-code-documenter\nReview this notebook's role, scientific and data contracts, execution state, outputs, and documentation risks. Do not modify or execute it." + prompt: "$angze-code-style\nReview this notebook's role, scientific and data contracts, execution state, outputs, and documentation risks. Do not modify or execute it." - id: explicit-notebook-limited language: jupyter mode: limited-edit - prompt: "$scientific-code-documenter\nApply only evidence-backed Markdown, comment, and docstring improvements to the explicitly allowlisted cells. Preserve all other notebook bytes and do not execute it." + prompt: "$angze-code-style\nApply only evidence-backed Markdown, comment, and docstring improvements to the explicitly allowlisted cells. Preserve all other notebook bytes and do not execute it." diff --git a/evals/scientific-code-documenter/results/.gitkeep b/evals/angze-code-style/results/.gitkeep similarity index 100% rename from evals/scientific-code-documenter/results/.gitkeep rename to evals/angze-code-style/results/.gitkeep diff --git a/evals/scientific-code-documenter/results/result.template.md b/evals/angze-code-style/results/result.template.md similarity index 96% rename from evals/scientific-code-documenter/results/result.template.md rename to evals/angze-code-style/results/result.template.md index d813c8a..915eb89 100644 --- a/evals/scientific-code-documenter/results/result.template.md +++ b/evals/angze-code-style/results/result.template.md @@ -1,4 +1,4 @@ -# Scientific Code Documenter evaluation result +# Angze Code Style evaluation result - Date: - Model and interface: diff --git a/evals/scientific-code-documenter/rubric.md b/evals/angze-code-style/rubric.md similarity index 100% rename from evals/scientific-code-documenter/rubric.md rename to evals/angze-code-style/rubric.md diff --git a/skills/scientific-code-documenter/SKILL.md b/skills/angze-code-style/SKILL.md similarity index 99% rename from skills/scientific-code-documenter/SKILL.md rename to skills/angze-code-style/SKILL.md index 8740492..86e5ce6 100644 --- a/skills/scientific-code-documenter/SKILL.md +++ b/skills/angze-code-style/SKILL.md @@ -1,9 +1,9 @@ --- -name: scientific-code-documenter +name: angze-code-style description: Document, prune, refactor, or review scientific code and Jupyter notebooks while preserving scientific, data, interface, execution-state, and artifact contracts. Use for evidence-backed comments, docstrings, notebook Markdown, units, shapes, schemas, ordering, missing-value conventions, assumptions, numerical choices, or readability work that must not change analysis semantics. Do not trigger for ordinary notebook execution, data analysis, debugging, performance work, methodology changes, output regeneration, dependency updates, or feature implementation. --- -# Scientific Code Documenter +# Angze Code Style Improve scientific and technical code without obscuring its logic or silently changing its behavior. Match the repository's established conventions when they are coherent and intentional. diff --git a/skills/angze-code-style/agents/openai.yaml b/skills/angze-code-style/agents/openai.yaml new file mode 100644 index 0000000..591c027 --- /dev/null +++ b/skills/angze-code-style/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Angze Code Style" + short_description: "Document scientific code and notebooks" + default_prompt: "Use $angze-code-style to improve this scientific code or notebook documentation while preserving its contracts and behavior." diff --git a/skills/scientific-code-documenter/references/comment-examples.md b/skills/angze-code-style/references/comment-examples.md similarity index 100% rename from skills/scientific-code-documenter/references/comment-examples.md rename to skills/angze-code-style/references/comment-examples.md diff --git a/skills/scientific-code-documenter/references/jupyter-notebooks.md b/skills/angze-code-style/references/jupyter-notebooks.md similarity index 100% rename from skills/scientific-code-documenter/references/jupyter-notebooks.md rename to skills/angze-code-style/references/jupyter-notebooks.md diff --git a/skills/scientific-code-documenter/references/scientific-contracts.md b/skills/angze-code-style/references/scientific-contracts.md similarity index 100% rename from skills/scientific-code-documenter/references/scientific-contracts.md rename to skills/angze-code-style/references/scientific-contracts.md diff --git a/skills/scientific-code-documenter/agents/openai.yaml b/skills/scientific-code-documenter/agents/openai.yaml deleted file mode 100644 index 7f13d71..0000000 --- a/skills/scientific-code-documenter/agents/openai.yaml +++ /dev/null @@ -1,4 +0,0 @@ -interface: - display_name: "Scientific Code Documenter" - short_description: "Document scientific code and notebooks" - default_prompt: "Use $scientific-code-documenter to improve this scientific code or notebook documentation while preserving its contracts and behavior." From a17160fc15c6c6274799cc05811f40dbedd58027 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 15:59:59 +0800 Subject: [PATCH 11/17] test(skill): complete real-notebook acceptance for angze-code-style v0.4.0 --- CHANGELOG.md | 4 +- README.md | 6 +-- .../angze-code-style-v0.4.0-real-notebook.md | 48 +++++++++++++++++++ .../angze-code-style-v0.4.0.manifest.json | 27 ++++++++++- docs/evaluations/angze-code-style-v0.4.0.md | 17 +++++-- docs/releases/angze-code-style-v0.4.0.md | 7 +-- 6 files changed, 95 insertions(+), 14 deletions(-) create mode 100644 docs/evaluations/angze-code-style-v0.4.0-real-notebook.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b26a30..4f53b24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,14 +31,14 @@ No changes yet. - Six accepted editing candidates remained byte-identical on fresh second passes. - Positive implicit triggering at 10/10 and false triggering at 1/10. - Full post-rename trigger rerun at 10/10 positives and 0/10 negatives, plus one renamed editable-notebook validation and byte-identical fresh second pass. +- Authorized immutable real tutorial-notebook acceptance: zero-edit review, controlled Markdown-only A/B, strict notebook validation, unchanged executable sources, and byte-identical fresh-session idempotence. ### Known limitations -- Explicitly authorized real-notebook acceptance remains pending. - One notebook feature-development prompt falsely triggered the skill in the original matrix; the post-rename rerun did not reproduce it. - Manual A/B scoring was not independently blind. - ShellCheck and rendered LaTeX compilation were unavailable in the release environment. -- Deterministic fixture probes do not prove general notebook equivalence or scientific correctness. +- Deterministic fixtures and bounded Markdown-only real-notebook evidence do not prove general notebook equivalence or scientific correctness. - Human scientific review remains required; the release remains experimental. ## [0.3.0] - 2026-07-29 diff --git a/README.md b/README.md index a7919bb..bfbccc8 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ codex_skills/ | Skill | Purpose | Status | | --- | --- | --- | -| `angze-code-style` | Document and prune scientific code and Jupyter notebooks using explicit contract inventory, evidence classes, and risk grading | Experimental v0.4.0 release candidate; synthetic notebook and controlled Python, Shell, LaTeX, HTML, trigger, restraint, and idempotence gates pass; authorized real-notebook acceptance is pending | +| `angze-code-style` | Document and prune scientific code and Jupyter notebooks using explicit contract inventory, evidence classes, and risk grading | Experimental v0.4.0 release candidate; synthetic and authorized real-notebook acceptance plus controlled Python, Shell, LaTeX, HTML, trigger, restraint, and idempotence gates pass | ## Validate and install @@ -118,9 +118,9 @@ Structural validation proves that a skill can be discovered and parsed; it does `angze-code-style` v0.4.0 changes runtime behavior and completes a clean-break rename from `scientific-code-documenter`. It adds scientific/artifact contract inventory, `evidence-backed` / `observable-only` / `unknown` classification, Tier 0–3 risk grading, structured review findings, conservative notebook policy, comment pruning, and repeated-pass stability. -Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. The initial trigger matrix scored 10/10 positives with one false trigger; the full post-rename rerun scored 10/10 positives and 0/10 negatives. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable, so Shell syntax/dynamic contracts and static LaTeX contracts were used. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/angze-code-style-v0.4.0.md) and [integrity manifest](docs/evaluations/angze-code-style-v0.4.0.manifest.json). +Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. The initial trigger matrix scored 10/10 positives with one false trigger; the full post-rename rerun scored 10/10 positives and 0/10 negatives. A separately authorized real tutorial-notebook snapshot passed zero-edit review, controlled A/B editing, strict preservation validation, behavior-preservation, and fresh-session idempotence. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable, so Shell syntax/dynamic contracts and static LaTeX contracts were used. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/angze-code-style-v0.4.0.md), [real-notebook acceptance record](docs/evaluations/angze-code-style-v0.4.0-real-notebook.md), and [integrity manifest](docs/evaluations/angze-code-style-v0.4.0.manifest.json). -Historical v0.1–v0.3 scientific Python and ORCA Shell evidence remains useful predecessor evidence, but it is not accepted as proof for the changed v0.4 runtime. No real notebook was supplied with explicit authorization, so real-notebook acceptance remains pending. The skill remains experimental; limited probes do not prove notebook equivalence or scientific correctness, and human scientific review remains mandatory. +Historical v0.1–v0.3 scientific Python and ORCA Shell evidence remains useful predecessor evidence, but it is not accepted as proof for the changed v0.4 runtime. The real-notebook acceptance established bounded Markdown-only preservation, not general notebook equivalence or scientific correctness. The skill remains experimental, and human scientific review remains mandatory. ## Privacy and publication diff --git a/docs/evaluations/angze-code-style-v0.4.0-real-notebook.md b/docs/evaluations/angze-code-style-v0.4.0-real-notebook.md new file mode 100644 index 0000000..3a8d168 --- /dev/null +++ b/docs/evaluations/angze-code-style-v0.4.0-real-notebook.md @@ -0,0 +1,48 @@ +# angze-code-style v0.4.0 real-notebook acceptance + +## Scope + +- Sanitized notebook identifier: `part-5-worked-tutorial-01` +- Role: tutorial +- Immutable snapshot SHA-256: `e160f1ac42055375c6d498957d64538fff4f76694ac26e4374074a2831721395` +- Size and structure: 527,494 bytes; 44 cells (30 Markdown, 14 code); notebook format 4.5 +- Stored state: nine output items, 14 execution counts, 44 cell IDs, and no attachments +- Original integrity: unchanged at the final check + +The original notebook was not executed, cleared, regenerated, normalized, or modified. Raw snapshots, candidates, prompts, logs, and private paths remain in an ignored local evidence bundle and are not tracked. + +## Zero-edit review + +A fresh `$angze-code-style` review-only session changed zero bytes. It reported six finding categories: public/tutorial interface, reproducibility and execution state, shapes/order/missing-value contracts, stored-output provenance, scientific interpretation, and navigation/readability. + +The review identified a stale interface description that required expert confirmation. That item was excluded from editing rather than guessed. + +## Controlled task and A/B + +The bounded task appended at most 12 lines to one existing Markdown cell to document only observable input/output shapes, append ordering, terminal-candidate state, missing/non-finite validation boundaries, and unspecified units. It prohibited code changes, cell operations, output or metadata changes, execution, and normalization. + +Fresh baseline and skill sessions began from the exact snapshot hash. Both produced a legitimate eight-line Markdown change, introduced no unsupported scientific claims, and passed the same strict contract validator. Side-by-side review found equivalent correctness, restraint, locality, contract awareness, and notebook safety: **tie**. The skill therefore did not lose to baseline. + +## Hard gates and behavior evidence + +| Gate | Unexpected changes | +| --- | ---: | +| Structure, order, IDs, and cell types | 0 | +| Notebook and cell metadata | 0 | +| Outputs and execution counts | 0 | +| Attachments | 0 | +| Non-allowlisted source | 0 | +| Bytes outside the intended source value | 0 | +| Serialization locality | 0 | + +All 14 code-cell source values were exactly equal to the immutable snapshot; their aggregate source hash was `451a689da8f62471cb24943fb4c44a5974fde48f91b56717f520b417762f3402`. This establishes behavior-preservation evidence for a Markdown-only edit. It does not claim general notebook equivalence. + +## Idempotence and runtime integrity + +The accepted skill candidate SHA-256 was `19e0bec2539426c7d25f5f74f4b3562300daa184f475cccf1a37ca72cb0b5ee3`. A fresh session repeated the exact request and changed zero bytes; the second-pass hash was identical and the strict validator again reported zero failures. + +An initial harness check misclassified the saved session's frozen reference read because it followed a runtime-snapshot path rather than the detector's expected direct path. The detector was corrected over the retained JSONL; the model session was not rerun or replaced, and every substantive idempotence gate had already passed. + +The tracked runtime SHA-256 before and after acceptance was `7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84`. Acceptance added evidence only; it did not mutate the frozen runtime. + +Verdict: **REAL-NOTEBOOK ACCEPTANCE PASSED**. diff --git a/docs/evaluations/angze-code-style-v0.4.0.manifest.json b/docs/evaluations/angze-code-style-v0.4.0.manifest.json index 0abc224..26c5976 100644 --- a/docs/evaluations/angze-code-style-v0.4.0.manifest.json +++ b/docs/evaluations/angze-code-style-v0.4.0.manifest.json @@ -100,6 +100,31 @@ "post_rename_negative_triggers": "0/10", "post_rename_representative_notebook_validation": "pass", "post_rename_representative_second_pass_byte_identical": true, - "real_notebook_acceptance": "pending explicit authorization" + "real_notebook_acceptance": { + "status": "pass", + "sanitized_identifier": "part-5-worked-tutorial-01", + "role": "tutorial", + "snapshot_sha256": "e160f1ac42055375c6d498957d64538fff4f76694ac26e4374074a2831721395", + "bytes": 527494, + "notebook_format": "4.5", + "cell_count": 44, + "cell_types": { + "markdown": 30, + "code": 14 + }, + "stored_output_items": 9, + "attachments": 0, + "zero_edit_changed_bytes": 0, + "zero_edit_findings": 6, + "ab_outcome": "tie", + "unexpected_protected_changes": 0, + "non_allowlisted_source_changes": 0, + "executable_sources_unchanged": "14/14", + "unsupported_scientific_claims_introduced": 0, + "accepted_candidate_sha256": "19e0bec2539426c7d25f5f74f4b3562300daa184f475cccf1a37ca72cb0b5ee3", + "second_pass_changed_bytes": 0, + "notebook_executed": false, + "general_notebook_equivalence_claimed": false + } } } diff --git a/docs/evaluations/angze-code-style-v0.4.0.md b/docs/evaluations/angze-code-style-v0.4.0.md index 14bc17d..6464c92 100644 --- a/docs/evaluations/angze-code-style-v0.4.0.md +++ b/docs/evaluations/angze-code-style-v0.4.0.md @@ -6,10 +6,11 @@ - Migration: clean-break rename from `scientific-code-documenter`; no compatibility alias - Release status: experimental local release candidate - Runtime behavior changed from v0.3.0: yes -- Real-notebook acceptance: pending explicit authorization -- Verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING** +- Real-notebook acceptance: passed +- Verdict: **READY FOR LOCAL REVIEW** - Tracked runtime snapshot SHA-256: `7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84` - Integrity manifest: [`angze-code-style-v0.4.0.manifest.json`](angze-code-style-v0.4.0.manifest.json) +- Real-notebook record: [`angze-code-style-v0.4.0-real-notebook.md`](angze-code-style-v0.4.0-real-notebook.md) ## Methodology @@ -62,6 +63,14 @@ After the clean-break rename, the same full 20-prompt matrix was rerun in fresh One explicit `$angze-code-style` editable-notebook run changed only the allowlisted source values and passed the strict notebook contract validator. A fresh second session read the same frozen runtime and left the first candidate byte-identical; the validator passed again. This is a rename-sensitive synthetic migration check, not real-notebook acceptance. +## Authorized real-notebook acceptance + +An immutable snapshot of one authorized real tutorial notebook was accepted without exposing its personal path or substantive research content in tracked evidence. The snapshot contained 44 cells (30 Markdown and 14 code), nine stored outputs, no attachments, and notebook format 4.5. + +Review-only inspection changed zero bytes and reported six categories of documentation or contract findings. A bounded task added a concise observable data/history contract to one existing Markdown cell. Baseline and skill candidates began from the same snapshot; both passed the strict validator and the honest human comparison was a tie. The skill introduced no unsupported scientific claims and was not materially worse than baseline. + +For the skill candidate, protected-structure, metadata, output, attachment, execution-count, cell-ID, non-allowlisted-source, and serialization-locality violations were all zero. All 14 executable source values remained exactly equal to the snapshot. A fresh second skill session changed zero bytes and passed the validator again. The original notebook was unchanged and was not executed. This is bounded Markdown-only preservation evidence; general notebook equivalence is not claimed. Full sanitized details are in the [real-notebook acceptance record](angze-code-style-v0.4.0-real-notebook.md). + ## Retained failed attempts - One orchestration attempt used incompatible current CLI flags and exited before model launch; no candidate changed. @@ -74,6 +83,4 @@ These attempts remain in the local result bundle and were not removed or scored Historical v0.1–v0.3 scientific Python and ORCA Shell acceptance remains predecessor evidence only because v0.4 changes runtime instructions. -No notebook owner explicitly authorized a real immutable notebook snapshot. Per the release boundary, no arbitrary personal or research notebook was substituted. Real-notebook zero-edit review and controlled baseline-versus-skill editing remain pending. - -ShellCheck and `pdflatex` were unavailable. Required Shell syntax and dynamic contract checks passed; LaTeX static contracts passed, but rendered compilation did not run. The deterministic notebook probe is limited fixture evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. +ShellCheck and `pdflatex` were unavailable. Required Shell syntax and dynamic contract checks passed; LaTeX static contracts passed, but rendered compilation did not run. Manual A/B scoring was not independently blind. The deterministic probe and bounded real-notebook acceptance are limited evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. diff --git a/docs/releases/angze-code-style-v0.4.0.md b/docs/releases/angze-code-style-v0.4.0.md index f8171dd..07f606d 100644 --- a/docs/releases/angze-code-style-v0.4.0.md +++ b/docs/releases/angze-code-style-v0.4.0.md @@ -25,6 +25,8 @@ Positive implicit triggering was 10/10. One of ten negative prompts—a notebook One renamed editable-notebook candidate then passed the strict contract validator, and a fresh second session produced a byte-identical candidate. Both sessions read the frozen `angze-code-style` runtime. +An explicitly authorized immutable snapshot of a real tutorial notebook also passed. Review-only inspection changed zero bytes. A bounded one-cell Markdown task produced a controlled A/B tie, introduced no unsupported scientific claims, preserved every protected notebook field, left all 14 executable source values unchanged, and was byte-identical on a fresh second skill pass. The original notebook was neither modified nor executed. See the [sanitized acceptance record](../evaluations/angze-code-style-v0.4.0-real-notebook.md). + ## Integrity The tracked runtime snapshot SHA-256 is `7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84`. It covers the five tracked runtime files as sorted file-hash and runtime-relative-path records. The [machine-readable manifest](../evaluations/angze-code-style-v0.4.0.manifest.json) records the algorithm, per-file hashes and sizes, notebook fixtures, validator files, and accepted result summary without personal paths. @@ -37,13 +39,12 @@ The deterministic probe executes only selected trusted synthetic fixture code in ## Limitations and release status -- No real notebook was supplied with explicit authorization. Required real-notebook review and controlled-edit acceptance remain pending. - Scoring was manual side-by-side review, not an independent blind judge. - ShellCheck and `pdflatex` were unavailable. Shell behavior contracts and static LaTeX contracts passed, but rendered LaTeX validation did not run. - One notebook feature prompt falsely triggered the skill in the original controlled matrix; a post-rename rerun did not reproduce it. -- Notebook structure preservation does not establish scientific correctness, output freshness, provenance completeness, or hidden-state safety. +- The accepted real-notebook edit was Markdown-only. Unchanged executable sources are bounded preservation evidence, not proof of general notebook equivalence, scientific correctness, output freshness, provenance completeness, or hidden-state safety. - Jupyter is the only newly supported environment; R, MATLAB, Julia, and Fortran remain out of scope. - Human scientific and maintainer review remains mandatory. - The release remains experimental. -Verdict: **PARTIALLY READY — REAL-NOTEBOOK ACCEPTANCE PENDING**. +Verdict: **READY FOR LOCAL REVIEW**. From aaab4f919192469f6dc5c414afc62034ebc201ef Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 16:35:02 +0800 Subject: [PATCH 12/17] docs(skill): clarify v0.4.0 validation limitations --- CHANGELOG.md | 2 +- docs/evaluations/angze-code-style-v0.4.0.md | 2 +- docs/releases/angze-code-style-v0.4.0.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f53b24..3e5fdaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,7 +37,7 @@ No changes yet. - One notebook feature-development prompt falsely triggered the skill in the original matrix; the post-rename rerun did not reproduce it. - Manual A/B scoring was not independently blind. -- ShellCheck and rendered LaTeX compilation were unavailable in the release environment. +- ShellCheck and rendered LaTeX compilation were unavailable and did not run. Deterministic Shell contracts and static LaTeX contracts passed; the unavailable tools remain disclosed, non-blocking confidence-depth limitations for this experimental release. - Deterministic fixtures and bounded Markdown-only real-notebook evidence do not prove general notebook equivalence or scientific correctness. - Human scientific review remains required; the release remains experimental. diff --git a/docs/evaluations/angze-code-style-v0.4.0.md b/docs/evaluations/angze-code-style-v0.4.0.md index 6464c92..636b35d 100644 --- a/docs/evaluations/angze-code-style-v0.4.0.md +++ b/docs/evaluations/angze-code-style-v0.4.0.md @@ -83,4 +83,4 @@ These attempts remain in the local result bundle and were not removed or scored Historical v0.1–v0.3 scientific Python and ORCA Shell acceptance remains predecessor evidence only because v0.4 changes runtime instructions. -ShellCheck and `pdflatex` were unavailable. Required Shell syntax and dynamic contract checks passed; LaTeX static contracts passed, but rendered compilation did not run. Manual A/B scoring was not independently blind. The deterministic probe and bounded real-notebook acceptance are limited evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. +ShellCheck and rendered `pdflatex` compilation were unavailable and did not run. Required Shell syntax and deterministic dynamic contract checks passed, and required LaTeX static contracts passed. The unavailable tools remain disclosed, non-blocking confidence-depth limitations for this experimental release; neither is represented as executed validation. Manual A/B scoring was not independently blind. The deterministic probe and bounded real-notebook acceptance are limited evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. diff --git a/docs/releases/angze-code-style-v0.4.0.md b/docs/releases/angze-code-style-v0.4.0.md index 07f606d..6651d3c 100644 --- a/docs/releases/angze-code-style-v0.4.0.md +++ b/docs/releases/angze-code-style-v0.4.0.md @@ -40,7 +40,7 @@ The deterministic probe executes only selected trusted synthetic fixture code in ## Limitations and release status - Scoring was manual side-by-side review, not an independent blind judge. -- ShellCheck and `pdflatex` were unavailable. Shell behavior contracts and static LaTeX contracts passed, but rendered LaTeX validation did not run. +- ShellCheck and rendered `pdflatex` compilation were unavailable and did not run. Required Shell syntax and deterministic behavior contracts passed, and required LaTeX static contracts passed. The unavailable tools remain disclosed, non-blocking confidence-depth limitations for this experimental release. - One notebook feature prompt falsely triggered the skill in the original controlled matrix; a post-rename rerun did not reproduce it. - The accepted real-notebook edit was Markdown-only. Unchanged executable sources are bounded preservation evidence, not proof of general notebook equivalence, scientific correctness, output freshness, provenance completeness, or hidden-state safety. - Jupyter is the only newly supported environment; R, MATLAB, Julia, and Fortran remain out of scope. From d998dea85af5fab6200d2ecc455383f299f04720 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 16:45:28 +0800 Subject: [PATCH 13/17] docs(skill): finalize angze-code-style v0.4.0 identity --- evals/angze-code-style/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/angze-code-style/README.md b/evals/angze-code-style/README.md index 9c30a9d..b4fc4a2 100644 --- a/evals/angze-code-style/README.md +++ b/evals/angze-code-style/README.md @@ -45,7 +45,7 @@ Exact repository-specific commands are maintained in the root `README.md` after Historical acceptance covers an authorized scientific Python repository and a production-style ORCA Shell workflow repository. The ORCA run reviewed nine scripts, exercised three controlled edit pairs with isolated mocks and dry runs, and preserved generated calculation inputs and workflow contracts without launching an expensive calculation. See the [v0.3.0 sanitized summary](../../docs/evaluations/scientific-code-documenter-v0.3.0.md). Those runs remain evidence for the predecessor runtime, not acceptance of the contract-aware v0.4 runtime. -Real-notebook acceptance for v0.4 remains pending until a notebook owner explicitly authorizes an immutable review and controlled edit candidate. Do not substitute a personal or research notebook merely because it is locally available. +Real-notebook acceptance for v0.4 passed using an explicitly authorized immutable tutorial-notebook snapshot. The original notebook remained unchanged and unexecuted; tracked evidence is sanitized in the [acceptance record](../../docs/evaluations/angze-code-style-v0.4.0-real-notebook.md). Run real-repository candidates only from immutable snapshots, never from the live source tree. Protect and restore tracked source state, and determine validity from the frozen runtime, source inputs, prompts, harnesses, and captured outputs. Ignored calculation outputs and filesystem metadata remain outside that boundary unless explicitly frozen; record concurrent external activity separately without treating live-source quiescence as an evaluation gate. From bbe2604baf7f120693e455e6b93acc46b51a9746 Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 17:14:25 +0800 Subject: [PATCH 14/17] docs(eval): clarify synthetic and real-notebook A/B gates --- evals/angze-code-style/rubric.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/evals/angze-code-style/rubric.md b/evals/angze-code-style/rubric.md index e16f28a..f71c7a7 100644 --- a/evals/angze-code-style/rubric.md +++ b/evals/angze-code-style/rubric.md @@ -13,6 +13,8 @@ Score each category from 0 to 3. Record evidence from the diff and validation lo ## Acceptance criteria +The quantitative score and comparative-win criteria in this section apply to controlled synthetic scored fixtures. Authorized real-notebook field acceptance uses the separate safety and restraint rule below. + - Average score: at least 15 out of 18. - Per-fixture score: at least 13 out of 18. - Behavior preservation: exactly 3 for every accepted fixture. @@ -38,7 +40,13 @@ Evaluate these gates before qualitative scoring. Every applicable gate must pass | Idempotence | Fresh second skill pass creates zero tracked diff; notebook bytes remain identical to pass 1 | | Validator health | Zero parse ambiguities, internal errors, timeouts, or unexplained changes | -A notebook editing candidate must beat its baseline. Review-only and generated candidates may tie, but they must not lose or change a byte. +### Synthetic scored editing fixtures + +For controlled synthetic editing fixtures used to establish skill-added value, every notebook editing candidate must beat its baseline and the aggregate editing win threshold still applies. Review-only and generated restraint candidates may tie, but they must not lose or change a byte. + +### Authorized real-notebook acceptance + +The separately authorized real-notebook field acceptance tests safety and restraint rather than adding another synthetic skill-value benchmark. The skill candidate must not lose to baseline; an honest tie is acceptable only when the documentation task is legitimate rather than manufactured through under-editing and every applicable notebook hard gate passes. ## Result record From 0400756ce3ea0753f69015a86eaea06c48de9c4e Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 17:20:58 +0800 Subject: [PATCH 15/17] ci: add deterministic validation workflow --- .github/workflows/validate.yml | 79 ++++++++ scripts/check_release_integrity.py | 315 +++++++++++++++++++++++++++++ 2 files changed, 394 insertions(+) create mode 100644 .github/workflows/validate.yml create mode 100644 scripts/check_release_integrity.py diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml new file mode 100644 index 0000000..b87b0a4 --- /dev/null +++ b/.github/workflows/validate.yml @@ -0,0 +1,79 @@ +name: Validation + +on: + pull_request: + branches: + - main + push: + branches: + - main + workflow_dispatch: + +permissions: + contents: read + +jobs: + validate: + name: Python ${{ matrix.python-version }} + runs-on: ubuntu-latest + timeout-minutes: 10 + strategy: + fail-fast: false + matrix: + python-version: + - "3.11" + - "3.12" + + steps: + - name: Check out the tested revision + uses: actions/checkout@v4 + with: + fetch-depth: 0 + ref: ${{ github.event.pull_request.head.sha || github.sha }} + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + + - name: Confirm the tested SHA + env: + EXPECTED_SHA: ${{ github.event.pull_request.head.sha || github.sha }} + run: | + actual_sha=$(git rev-parse HEAD) + test "$actual_sha" = "$EXPECTED_SHA" + printf 'Tested SHA: %s\n' "$actual_sha" + + - name: Validate skill packages + run: ./scripts/check_all_skills.sh + + - name: Run deterministic angze-code-style fixtures + run: ./scripts/run_fixture_checks.sh angze-code-style + + - name: Verify v0.4.0 release integrity + run: >- + python3 scripts/check_release_integrity.py + docs/evaluations/angze-code-style-v0.4.0.manifest.json + --expected-runtime-sha + 7f79b22c4694717963966132967b1a03b4cf5790529e6c229ef2ef31920fad84 + --expected-skill angze-code-style + --expected-version 0.4.0 + --forbid-invocation '$scientific-code-documenter' + + - name: Check changed whitespace + env: + PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} + PUSH_BEFORE_SHA: ${{ github.event.before }} + run: | + if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then + git diff --check "$PR_BASE_SHA...HEAD" + elif [ "$GITHUB_EVENT_NAME" = "push" ] && + [ -n "$PUSH_BEFORE_SHA" ] && + [ "$PUSH_BEFORE_SHA" != "0000000000000000000000000000000000000000" ]; then + git diff --check "$PUSH_BEFORE_SHA..HEAD" + else + git diff-tree --check --no-commit-id --root -r HEAD + fi + + - name: Confirm checks did not modify tracked files + run: git diff --exit-code diff --git a/scripts/check_release_integrity.py b/scripts/check_release_integrity.py new file mode 100644 index 0000000..b43c746 --- /dev/null +++ b/scripts/check_release_integrity.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +"""Validate a release manifest, frozen runtime, and public release surfaces.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import re +import subprocess +import sys +from pathlib import Path, PurePosixPath + + +class IntegrityError(RuntimeError): + """Raised when a release-integrity contract is not satisfied.""" + + +def require(condition: bool, message: str) -> None: + if not condition: + raise IntegrityError(message) + + +def safe_repo_path(repo_root: Path, raw_path: str) -> Path: + posix_path = PurePosixPath(raw_path) + require( + raw_path == posix_path.as_posix() + and not posix_path.is_absolute() + and ".." not in posix_path.parts, + f"manifest path is not a safe repository-relative path: {raw_path!r}", + ) + return repo_root.joinpath(*posix_path.parts) + + +def sha256_bytes(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def tracked_paths(repo_root: Path, pathspec: str) -> set[str]: + output = subprocess.check_output( + ["git", "ls-files", "-z", "--", pathspec], cwd=repo_root + ) + return {path for path in output.decode().split("\0") if path} + + +def validate_file_records( + repo_root: Path, records: object, label: str +) -> list[tuple[str, str]]: + require(isinstance(records, list) and records, f"{label} files must be a non-empty list") + + validated: list[tuple[str, str]] = [] + seen: set[str] = set() + for record in records: + require(isinstance(record, dict), f"{label} file record must be an object") + raw_path = record.get("path") + expected_hash = record.get("sha256") + expected_size = record.get("bytes") + require(isinstance(raw_path, str), f"{label} file path must be a string") + require(raw_path not in seen, f"duplicate {label} file path: {raw_path}") + require( + isinstance(expected_hash, str) + and re.fullmatch(r"[0-9a-f]{64}", expected_hash) is not None, + f"invalid SHA-256 for {raw_path}", + ) + require( + isinstance(expected_size, int) and expected_size >= 0, + f"invalid byte size for {raw_path}", + ) + + path = safe_repo_path(repo_root, raw_path) + require(path.is_file() and not path.is_symlink(), f"missing regular file: {raw_path}") + data = path.read_bytes() + actual_hash = sha256_bytes(data) + require(actual_hash == expected_hash, f"SHA-256 mismatch: {raw_path}") + require(len(data) == expected_size, f"byte-size mismatch: {raw_path}") + seen.add(raw_path) + validated.append((raw_path, actual_hash)) + + require( + [path for path, _ in validated] == sorted(path for path, _ in validated), + f"{label} file records must be C-sorted by path", + ) + return validated + + +def validate_markdown_links(repo_root: Path, files: list[Path]) -> int: + link_count = 0 + for source in files: + for match in re.findall(r"\]\(([^)]+)\)", source.read_text(encoding="utf-8")): + link_count += 1 + reference = match.strip("<>").split("#", 1)[0].split("?", 1)[0] + if not reference or reference.startswith( + ("#", "http://", "https://", "mailto:", "skill://", "$") + ): + continue + target = (source.parent / reference).resolve() + try: + target.relative_to(repo_root) + except ValueError as error: + raise IntegrityError( + f"relative link escapes repository in {source.relative_to(repo_root)}: {reference}" + ) from error + require( + target.exists(), + f"broken relative link in {source.relative_to(repo_root)}: {reference}", + ) + return link_count + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("manifest", type=Path) + parser.add_argument("--expected-runtime-sha", required=True) + parser.add_argument("--expected-skill", required=True) + parser.add_argument("--expected-version", required=True) + parser.add_argument("--forbid-invocation", action="append", default=[]) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + repo_root = Path(__file__).resolve().parents[1] + manifest_path = args.manifest + if not manifest_path.is_absolute(): + manifest_path = repo_root / manifest_path + manifest_path = manifest_path.resolve() + try: + manifest_path.relative_to(repo_root) + except ValueError as error: + raise IntegrityError("manifest must be inside the repository") from error + + require( + re.fullmatch(r"[0-9a-f]{64}", args.expected_runtime_sha) is not None, + "expected runtime SHA-256 must be 64 lowercase hexadecimal characters", + ) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + require(manifest.get("schema_version") == 1, "unsupported manifest schema") + require(manifest.get("skill") == args.expected_skill, "manifest skill mismatch") + require(manifest.get("version") == args.expected_version, "manifest version mismatch") + + runtime = manifest.get("runtime_snapshot") + require(isinstance(runtime, dict), "missing runtime_snapshot object") + require( + runtime.get("sha256") == args.expected_runtime_sha, + "manifest runtime SHA-256 does not match the accepted release hash", + ) + runtime_records = validate_file_records( + repo_root, runtime.get("files"), "runtime snapshot" + ) + + runtime_root = f"skills/{args.expected_skill}" + listed_runtime_paths = {path for path, _ in runtime_records} + require( + listed_runtime_paths == tracked_paths(repo_root, runtime_root), + "manifest runtime file set does not match tracked runtime files", + ) + aggregate_records = [] + for raw_path, digest in runtime_records: + runtime_relative = PurePosixPath(raw_path).relative_to(runtime_root).as_posix() + aggregate_records.append(f"{digest} {runtime_relative}") + actual_runtime_sha = sha256_bytes( + ("\n".join(aggregate_records) + "\n").encode() + ) + require( + actual_runtime_sha == args.expected_runtime_sha, + "tracked runtime aggregate SHA-256 mismatch", + ) + + notebook_section = manifest.get("notebook_acceptance_fixtures") + require(isinstance(notebook_section, dict), "missing notebook fixture section") + notebook_records = validate_file_records( + repo_root, notebook_section.get("files"), "notebook acceptance fixture" + ) + recorded_notebooks = { + path for path, _ in notebook_records if PurePosixPath(path).suffix == ".ipynb" + } + tracked_eval_paths = tracked_paths(repo_root, f"evals/{args.expected_skill}") + tracked_notebooks = { + path for path in tracked_eval_paths if PurePosixPath(path).suffix == ".ipynb" + } + require( + tracked_notebooks == recorded_notebooks, + "tracked notebooks do not match the synthetic notebooks in the manifest", + ) + raw_log_paths = { + path + for path in tracked_eval_paths + if PurePosixPath(path).suffix in {".jsonl", ".log"} + } + require(not raw_log_paths, "raw evaluation logs are tracked") + allowed_result_paths = { + f"evals/{args.expected_skill}/results/.gitkeep", + f"evals/{args.expected_skill}/results/result.template.md", + } + require( + tracked_paths(repo_root, f"evals/{args.expected_skill}/results") + <= allowed_result_paths, + "unexpected raw result artifacts are tracked", + ) + + version_label = f"v{args.expected_version}" + release_path = repo_root / f"docs/releases/{args.expected_skill}-{version_label}.md" + evaluation_path = ( + repo_root / f"docs/evaluations/{args.expected_skill}-{version_label}.md" + ) + acceptance_path = repo_root / ( + f"docs/evaluations/{args.expected_skill}-{version_label}-real-notebook.md" + ) + current_surfaces = [ + repo_root / "README.md", + repo_root / "GETTING_STARTED.md", + repo_root / "CHANGELOG.md", + release_path, + evaluation_path, + acceptance_path, + repo_root / f"evals/{args.expected_skill}/README.md", + repo_root / f"evals/{args.expected_skill}/prompts.yaml", + repo_root / runtime_root / "SKILL.md", + repo_root / runtime_root / "agents/openai.yaml", + ] + for path in current_surfaces: + require(path.is_file(), f"missing current release surface: {path.relative_to(repo_root)}") + + invocation = f"${args.expected_skill}" + skill_text = (repo_root / runtime_root / "SKILL.md").read_text(encoding="utf-8") + require( + re.search(rf"(?m)^name:\s*{re.escape(args.expected_skill)}\s*$", skill_text) + is not None, + "SKILL.md frontmatter name mismatch", + ) + agent_text = (repo_root / runtime_root / "agents/openai.yaml").read_text( + encoding="utf-8" + ) + prompts_text = (repo_root / f"evals/{args.expected_skill}/prompts.yaml").read_text( + encoding="utf-8" + ) + require(invocation in agent_text, "agent metadata lacks the current invocation") + require( + re.search(rf"(?m)^skill:\s*{re.escape(args.expected_skill)}\s*$", prompts_text) + is not None + and invocation in prompts_text, + "evaluation prompts lack the current skill identity or invocation", + ) + require( + f"{args.expected_skill} {version_label}" in release_path.read_text(encoding="utf-8") + and f"{args.expected_skill} {version_label}" + in evaluation_path.read_text(encoding="utf-8"), + "current release documents have inconsistent identity", + ) + require( + "experimental" in release_path.read_text(encoding="utf-8").lower(), + "release note does not state experimental status", + ) + require( + "REAL-NOTEBOOK ACCEPTANCE PASSED" + in acceptance_path.read_text(encoding="utf-8"), + "real-notebook acceptance is not recorded as passed", + ) + controlled_results = manifest.get("controlled_results") + require(isinstance(controlled_results, dict), "missing controlled_results object") + real_acceptance = controlled_results.get("real_notebook_acceptance") + require( + isinstance(real_acceptance, dict) and real_acceptance.get("status") == "pass", + "manifest real-notebook acceptance is not passed", + ) + + combined_current_text = "\n".join( + path.read_text(encoding="utf-8", errors="replace") for path in current_surfaces + ) + for forbidden in args.forbid_invocation: + require( + forbidden not in combined_current_text, + f"forbidden current-facing invocation found: {forbidden}", + ) + require( + re.search( + r"REAL-NOTEBOOK ACCEPTANCE PENDING|PARTIALLY READY|" + r"real-notebook acceptance[^.]*pending", + combined_current_text, + flags=re.IGNORECASE, + ) + is None, + "current release surfaces still describe real-notebook acceptance as pending", + ) + + personal_path_pattern = re.compile( + rb"/Users/[^/$\s]+/|/home/[^/$\s]+/|[A-Za-z]:\\Users\\[^\\]+\\" + ) + privacy_paths = { + path for path, _ in runtime_records + notebook_records + } | {str(path.relative_to(repo_root)) for path in current_surfaces} + for raw_path in sorted(privacy_paths): + require( + personal_path_pattern.search((repo_root / raw_path).read_bytes()) is None, + f"personal absolute path found: {raw_path}", + ) + + markdown_surfaces = [path for path in current_surfaces if path.suffix == ".md"] + link_count = validate_markdown_links(repo_root, markdown_surfaces) + + print(f"PASS release identity: {args.expected_skill} {version_label} ({invocation})") + print(f"PASS runtime files: {len(runtime_records)}") + print(f"PASS runtime SHA-256: {actual_runtime_sha}") + print(f"PASS notebook fixture records: {len(notebook_records)}") + print(f"PASS current release links: {link_count}") + print("PASS tracked evidence and path sanitation") + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except (IntegrityError, json.JSONDecodeError, OSError, subprocess.CalledProcessError) as error: + print(f"error: {error}", file=sys.stderr) + raise SystemExit(1) From cae49afe984eb9a329e75ca0ed7057ea2fea161e Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 17:26:31 +0800 Subject: [PATCH 16/17] test(eval): satisfy ShellCheck in synthetic fixture --- evals/angze-code-style/fixtures/shell/run_campaign.sh | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/evals/angze-code-style/fixtures/shell/run_campaign.sh b/evals/angze-code-style/fixtures/shell/run_campaign.sh index fc47e30..432c934 100644 --- a/evals/angze-code-style/fixtures/shell/run_campaign.sh +++ b/evals/angze-code-style/fixtures/shell/run_campaign.sh @@ -34,9 +34,7 @@ for temperature in $temperatures; do resume_args="" fi - $solver simulate --input $input_file --temperature-k $temperature --basis synthetic-dz --convergence-energy 1e-8 --convergence-density 1e-6 --max-iterations 300 --grid radial-96-angular-302 $resume_args --output $output_file - - if [ "$?" -ne 0 ]; then + if ! $solver simulate --input $input_file --temperature-k $temperature --basis synthetic-dz --convergence-energy 1e-8 --convergence-density 1e-6 --max-iterations 300 --grid radial-96-angular-302 $resume_args --output $output_file; then printf 'solver failed at temperature %s K\n' "$temperature" >&2 exit 1 fi From 3f913ee9d34fbeda645a5c564231966bd395e51b Mon Sep 17 00:00:00 2001 From: Squiddy Date: Fri, 28 Aug 2026 18:25:00 +0800 Subject: [PATCH 17/17] test(eval): refresh Shell A/B provenance for v0.4.0 --- CHANGELOG.md | 2 +- README.md | 2 +- .../angze-code-style-v0.4.0.manifest.json | 27 +++++++++++++++++++ docs/evaluations/angze-code-style-v0.4.0.md | 9 ++++++- docs/releases/angze-code-style-v0.4.0.md | 2 +- 5 files changed, 38 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e5fdaa..9cdf736 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,7 +37,7 @@ No changes yet. - One notebook feature-development prompt falsely triggered the skill in the original matrix; the post-rename rerun did not reproduce it. - Manual A/B scoring was not independently blind. -- ShellCheck and rendered LaTeX compilation were unavailable and did not run. Deterministic Shell contracts and static LaTeX contracts passed; the unavailable tools remain disclosed, non-blocking confidence-depth limitations for this experimental release. +- ShellCheck and rendered LaTeX compilation were unavailable in the original local acceptance environment. Deterministic Shell contracts and static LaTeX contracts passed. ShellCheck subsequently ran and passed under the existing policy in remote CI on Ubuntu for Python 3.11 and 3.12; rendered LaTeX compilation remains a disclosed, non-blocking confidence-depth limitation. - Deterministic fixtures and bounded Markdown-only real-notebook evidence do not prove general notebook equivalence or scientific correctness. - Human scientific review remains required; the release remains experimental. diff --git a/README.md b/README.md index bfbccc8..b2da7e5 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ Structural validation proves that a skill can be discovered and parsed; it does `angze-code-style` v0.4.0 changes runtime behavior and completes a clean-break rename from `scientific-code-documenter`. It adds scientific/artifact contract inventory, `evidence-backed` / `observable-only` / `unknown` classification, Tier 0–3 risk grading, structured review findings, conservative notebook policy, comment pruning, and repeated-pass stability. -Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. The initial trigger matrix scored 10/10 positives with one false trigger; the full post-rename rerun scored 10/10 positives and 0/10 negatives. A separately authorized real tutorial-notebook snapshot passed zero-edit review, controlled A/B editing, strict preservation validation, behavior-preservation, and fresh-session idempotence. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable, so Shell syntax/dynamic contracts and static LaTeX contracts were used. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/angze-code-style-v0.4.0.md), [real-notebook acceptance record](docs/evaluations/angze-code-style-v0.4.0-real-notebook.md), and [integrity manifest](docs/evaluations/angze-code-style-v0.4.0.manifest.json). +Controlled synthetic evaluation covered an editable notebook, review-only stale claim, over-commented notebook, generated notebook, and representative Python, Shell, LaTeX, and HTML fixtures. The accepted skill candidates averaged 17.5/18, won all six editing comparisons, passed every behavior and notebook hard gate, and were byte-stable on fresh second passes. The initial trigger matrix scored 10/10 positives with one false trigger; the full post-rename rerun scored 10/10 positives and 0/10 negatives. A separately authorized real tutorial-notebook snapshot passed zero-edit review, controlled A/B editing, strict preservation validation, behavior-preservation, and fresh-session idempotence. Scoring was manual rather than independently blind. ShellCheck and `pdflatex` were unavailable in the original local acceptance environment, so Shell syntax/dynamic contracts and static LaTeX contracts were used. ShellCheck subsequently ran and passed under the existing policy in remote CI on Ubuntu for Python 3.11 and 3.12; rendered `pdflatex` compilation remains unavailable. See the [sanitized v0.4.0 evaluation summary](docs/evaluations/angze-code-style-v0.4.0.md), [real-notebook acceptance record](docs/evaluations/angze-code-style-v0.4.0-real-notebook.md), and [integrity manifest](docs/evaluations/angze-code-style-v0.4.0.manifest.json). Historical v0.1–v0.3 scientific Python and ORCA Shell evidence remains useful predecessor evidence, but it is not accepted as proof for the changed v0.4 runtime. The real-notebook acceptance established bounded Markdown-only preservation, not general notebook equivalence or scientific correctness. The skill remains experimental, and human scientific review remains mandatory. diff --git a/docs/evaluations/angze-code-style-v0.4.0.manifest.json b/docs/evaluations/angze-code-style-v0.4.0.manifest.json index 26c5976..a975ac4 100644 --- a/docs/evaluations/angze-code-style-v0.4.0.manifest.json +++ b/docs/evaluations/angze-code-style-v0.4.0.manifest.json @@ -96,6 +96,33 @@ "skill_score_minimum": 17, "behavior_preservation_score": "3/3 for every accepted editing fixture", "accepted_second_passes_byte_identical": "6/6", + "shell_provenance_refresh": { + "status": "pass", + "model": "gpt-5.6-terra", + "reasoning": "medium", + "scored_fixture_path": "evals/angze-code-style/fixtures/shell/cramped_orca_batch.sh", + "scored_fixture_sha256": "c6558b5065325b51aa4effafa93e8e67849b8ff3369022bf2f3546dca87a29b6", + "scored_fixture_changed_by_sc2181_fix": false, + "auxiliary_fixture_path": "evals/angze-code-style/fixtures/shell/run_campaign.sh", + "auxiliary_fixture_pre_correction_sha256": "e60803dd268b820f139af580f39ffb92aedafef5269ab9304e38e6a1c3f900c5", + "auxiliary_fixture_current_sha256": "2d61b1f3b088b83ccee9f9868e9dccf83c7766100a88e9e38ca9b3f25fc262d5", + "deterministic_contract_path": "evals/angze-code-style/fixtures/contracts/test_shell_fixtures.py", + "deterministic_contract_sha256": "6c9907b81d00431fa4d361a4bec1cf1da45ca863c30d1a5cc632fa30403aaeba", + "baseline_input_sha256": "c6558b5065325b51aa4effafa93e8e67849b8ff3369022bf2f3546dca87a29b6", + "skill_input_sha256": "c6558b5065325b51aa4effafa93e8e67849b8ff3369022bf2f3546dca87a29b6", + "baseline_score": 16, + "skill_score": 17, + "outcome": "skill win", + "behavior_preservation": "3/3 for both candidates", + "accepted_skill_candidate_sha256": "13939830ff6d6463b5a51571f7e8134308eb85c9497ccde4419cbfbb1704effb", + "second_pass_sha256": "13939830ff6d6463b5a51571f7e8134308eb85c9497ccde4419cbfbb1704effb", + "second_pass_changed_bytes": 0, + "local_shellcheck": "unavailable", + "remote_shellcheck": "pass on Ubuntu for Python 3.11 and 3.12", + "historical_scores_reproduced": true, + "unaffected_evaluations_rerun": false, + "runtime_changed": false + }, "post_rename_positive_triggers": "10/10", "post_rename_negative_triggers": "0/10", "post_rename_representative_notebook_validation": "pass", diff --git a/docs/evaluations/angze-code-style-v0.4.0.md b/docs/evaluations/angze-code-style-v0.4.0.md index 636b35d..ede7a19 100644 --- a/docs/evaluations/angze-code-style-v0.4.0.md +++ b/docs/evaluations/angze-code-style-v0.4.0.md @@ -36,6 +36,12 @@ Every accepted skill fixture scored at least 17/18 and behavior preservation was The baseline Python candidate added public annotations and asserted more scientific interpretation than the repository established. The accepted skill candidate retained public signatures and described the threshold neutrally. The skill notebook candidate explicitly kept the scale factor's units and scientific meaning unknown. The pruning candidate consolidated visible `None` filtering and ascending ordering once while deleting syntax narration. Shell, LaTeX, and HTML candidates documented concrete contracts more locally and concisely. +### Targeted Shell provenance confirmation + +The first remote CI run later found ShellCheck `SC2181` in `fixtures/shell/run_campaign.sh`. That auxiliary syntax/lint fixture was corrected without changing behavior (`e60803dd268b820f139af580f39ffb92aedafef5269ab9304e38e6a1c3f900c5` to `2d61b1f3b088b83ccee9f9868e9dccf83c7766100a88e9e38ca9b3f25fc262d5`). The scored Shell A/B input was instead `fixtures/shell/cramped_orca_batch.sh`; its SHA-256 remained `c6558b5065325b51aa4effafa93e8e67849b8ff3369022bf2f3546dca87a29b6` before and after that correction. The original scored provenance was therefore not invalidated. + +For conservative current-fixture confirmation, only the Shell editing pair was rerun with the original prompt, `gpt-5.6-terra`, medium reasoning, fresh isolated sessions, a zero-user-skill baseline, and the frozen runtime. Both arms began from the same `c6558b…` input and passed Bash syntax plus the current deterministic Shell contract. The baseline scored 16/18 and the skill candidate scored 17/18 with behavior preservation at 3/3, reproducing the original skill win. The accepted skill candidate SHA-256 was `13939830ff6d6463b5a51571f7e8134308eb85c9497ccde4419cbfbb1704effb`; fresh repeated skill prompts left it byte-identical and the deterministic contract passed again. No unaffected A/B, trigger, or real-notebook evaluation was rerun, and the frozen runtime did not change. Raw prompts, logs, diffs, excluded attempts, hashes, and validation records remain in the ignored local result bundle. + ## Notebook hard gates | Gate | Result | @@ -76,6 +82,7 @@ For the skill candidate, protected-structure, metadata, output, attachment, exec - One orchestration attempt used incompatible current CLI flags and exited before model launch; no candidate changed. - Initial Python, Shell, and HTML candidates drifted on a second pass. The runtime added an explicit dirty-target preservation boundary, and all affected reruns became byte-identical. - Deletion-only notebook-pruning attempts tied the baseline. The runtime's evidence-class wording was reconciled so visible missing-value and ordering operations are documented neutrally without inventing scientific purpose; the final retained candidate passed and won. +- One targeted Shell-refresh skill attempt was excluded before scoring because its raw log did not record a frozen-runtime read; the accepted retry read the frozen runtime and reproduced the original score and outcome. These attempts remain in the local result bundle and were not removed or scored as accepted outputs. @@ -83,4 +90,4 @@ These attempts remain in the local result bundle and were not removed or scored Historical v0.1–v0.3 scientific Python and ORCA Shell acceptance remains predecessor evidence only because v0.4 changes runtime instructions. -ShellCheck and rendered `pdflatex` compilation were unavailable and did not run. Required Shell syntax and deterministic dynamic contract checks passed, and required LaTeX static contracts passed. The unavailable tools remain disclosed, non-blocking confidence-depth limitations for this experimental release; neither is represented as executed validation. Manual A/B scoring was not independently blind. The deterministic probe and bounded real-notebook acceptance are limited evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. +ShellCheck and rendered `pdflatex` compilation were unavailable in the original local acceptance environment. Required Shell syntax and deterministic dynamic contract checks passed, and required LaTeX static contracts passed. ShellCheck subsequently ran and passed under the repository's existing policy in remote CI on Ubuntu for Python 3.11 and 3.12, including the intentional `SC2086` exclusion for the baseline fixture. Rendered `pdflatex` compilation remains an unavailable, non-blocking confidence-depth limitation. Manual A/B scoring was not independently blind. The deterministic probe and bounded real-notebook acceptance are limited evidence, not general equivalence or scientific validation. Human scientific review remains mandatory. diff --git a/docs/releases/angze-code-style-v0.4.0.md b/docs/releases/angze-code-style-v0.4.0.md index 6651d3c..c0e75d4 100644 --- a/docs/releases/angze-code-style-v0.4.0.md +++ b/docs/releases/angze-code-style-v0.4.0.md @@ -40,7 +40,7 @@ The deterministic probe executes only selected trusted synthetic fixture code in ## Limitations and release status - Scoring was manual side-by-side review, not an independent blind judge. -- ShellCheck and rendered `pdflatex` compilation were unavailable and did not run. Required Shell syntax and deterministic behavior contracts passed, and required LaTeX static contracts passed. The unavailable tools remain disclosed, non-blocking confidence-depth limitations for this experimental release. +- ShellCheck and rendered `pdflatex` compilation were unavailable in the original local acceptance environment. Required Shell syntax and deterministic behavior contracts passed, and required LaTeX static contracts passed. ShellCheck subsequently ran and passed under the existing policy in remote CI on Ubuntu for Python 3.11 and 3.12; rendered `pdflatex` compilation remains a disclosed, non-blocking confidence-depth limitation. - One notebook feature prompt falsely triggered the skill in the original controlled matrix; a post-rename rerun did not reproduce it. - The accepted real-notebook edit was Markdown-only. Unchanged executable sources are bounded preservation evidence, not proof of general notebook equivalence, scientific correctness, output freshness, provenance completeness, or hidden-state safety. - Jupyter is the only newly supported environment; R, MATLAB, Julia, and Fortran remain out of scope.