From d2bfaf7acdc98cdbf7cab2b49644ff0c34d69393 Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Fri, 13 Nov 2026 19:45:31 +0530 Subject: [PATCH 1/7] =?UTF-8?q?P10a=20step=201:=20C-212=20to=20C-215=20and?= =?UTF-8?q?=20C-217=20Decided;=20C-218=20to=20C-221=20(where,=20which=20sc?= =?UTF-8?q?enarios,=20scope,=20lifecycle);=20P8b,=20the=20P8=20gate=20and?= =?UTF-8?q?=20M5=20passed=20with=20limits=20per=20the=20verdict=20on=20PR?= =?UTF-8?q?=20#31=20(Doc=2015=20=C2=A715.3,=20=C2=A725)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/DECISIONS_REGISTER.md | 14 +++++++++----- docs/PHASE_GATES.md | 28 +++++++++++++++++++++++++--- 2 files changed, 34 insertions(+), 8 deletions(-) diff --git a/docs/DECISIONS_REGISTER.md b/docs/DECISIONS_REGISTER.md index 37f5e1d..673e7e4 100644 --- a/docs/DECISIONS_REGISTER.md +++ b/docs/DECISIONS_REGISTER.md @@ -223,9 +223,13 @@ This register records every conflict found between the 15 `CloudSpartanX_*` spec | C-209 | P8a dev run: under C and C′ the open target of C-196 rejects every F8 trajectory at step 1 | Found in the P8a harness check (no label read). C-196 says an open target "still needs a fresh verification result ... a missing result blocks", and its reason is that "without this rule every condition rejects F8 at step 1". In the dev run, on all 28 F8 targets, C and C′ REJECT step 1 with `INSUFFICIENT_EVIDENCE` on `INVARIANT:INV-SEC-001:2`: the middle step does not touch INV-SEC-001's footprint, so impact does not list it, `DailScopePolicy` (unchanged, C-198) owes no work item for it, and a VIOLATED reference cannot be carried forward (C-159), so no fresh result exists and the rule blocks. A and B verify the target on every step and promote these steps (A 84 of 84 F8 transitions, B 63 of 77). So for C and C′ the rule does not reach its stated purpose, and F8 measures only step 1 there. Nothing was changed (no tuning to the dev run). Options for the authors include: let an open target that the candidate does not affect stay VIOLATED with the parent's evidence (a carry of a VIOLATED reference, which the commit path already supports through `InvariantProof.carry_forward`); or make the harness add the open target's item to every condition's work set. The same mechanism also rejects a candidate whose parent holds a non-target invariant VIOLATED that the candidate does not touch (1 transition each under C and C′). **Closed by C-210 (authors, 12 Nov 2026).** | Decided (authors, 12 Nov 2026): see C-210 | | C-210 | P8a-2: DAIL always re-checks every invariant that is not PROTECTED in the parent (closes C-209) | DAIL's verification work for a candidate is the impact work set plus one invariant item for every invariant whose reference in the parent trusted state is not PROTECTED (VIOLATED or UNCERTAIN) and that impact did not already list. This is the paper's required set R(C_t) = G_gate ∪ U_t ∪ I_t ∪ B, where U_t is the set of non-PROTECTED references (C-159 already requires a fresh result for each of them; the work set did not schedule one). It applies to conditions C and C′ (both use `DailScopePolicy`). These items count as verification work. A VIOLATED reference is never carried forward (C-159 unchanged). **Built:** `DailScopePolicy` version `dail-scope-2` adds the U_t items (reason `NOT_PROTECTED_IN_PARENT`, BLOCKING, sorted, never a second item for an invariant impact listed). `VerificationScopePolicy.select` takes the parent state's invariant references as a keyword `parent_refs` (default none): the narrowest input that carries the statuses, which the orchestrator already holds; the impact report does not carry them and a scope policy does not read the database. Impact, verification, identity and the promotion decision are unchanged; with every parent reference PROTECTED the work set is impact's, unchanged. | Decided (authors, 12 Nov 2026) | | C-211 | P8a-2: two reason codes for work a scope policy adds (amends C-138's vocabulary; answers C-207 (1)) | `ReasonCode.CONDITION_SCOPE` (the condition's scope policy requires this check: B's unaffected invariants, A's target when impact did not list it) and `ReasonCode.NOT_PROTECTED_IN_PARENT` (the U_t items of C-210), in a fifth group `POLICY_CODES`. They are valid only on invariant work items, and an impact report that carries one is refused (impact never gives them). The `SCOPE_TYPE_CHANGED` stand-in of C-207 (1) is gone; the experiment records keep `origin = condition` and add the item's reason (`selected_reason`). `tests/unit/test_impact_domain.py` lists the two new codes and the new group explicitly (stricter). `StatefulFullScopePolicy` and `StatelessScopePolicy` are now version 2 (their items' reason changed). | Decided (authors, 12 Nov 2026) | -| C-212 | P8b-1: metric definitions | The metrics of the P8b work package Section 4, exactly, versioned `metrics-draft-1`. They are a draft for the dev pilot; P10 freezes the final definitions before the test split is opened. | Proposed (P8b) | -| C-213 | P8b-2: identity truth for H3 | Identity truth comes from a hand-written table in DAIL, `experiments/scoring/identity_truth.json`: one row per TerraPreserve operator that moves, replaces, deletes or creates a resource, giving the expected change type at the operator's target address, with the sentence of `D4_OPERATORS.md` it comes from. An operator whose identity truth the catalogue does not state gets no row, and its pairs are excluded and counted. The table is written from the catalogue's text, never from DAIL's output. | Proposed (P8b) | -| C-214 | P8b-3: statistics on dev are descriptive only | Counts, proportions with numerator and denominator, Wilson 95% intervals, and per-cluster summaries (cluster = `template_id` for the controlled tier, `repo_lineage_id` for the real-world tier). No hypothesis test, no p-value, no "significant". Inference is for the test split after the protocol freeze (P10). | Proposed (P8b) | -| C-215 | P8b: readings made while building scoring, the extractor and the report | (1) **Evidence-transfer error** (item 10, not defined in the work package): a true replacement that DAIL treats as the same object (neither REPLACED nor UNCERTAIN) in a transition where some required invariant was CARRIED_FORWARD. (2) **Appearing and vanishing resources** for the identity truth are those declared (a top-level `resource` block in the step's `.tf` files) in step i and not in step i−1, or the reverse, restricted to the addresses the operator's mutation rows name; the first version used "every row has no `before`", which wrongly made an existing group that only gains an attribute a CREATE (found on `op_f3_db_ingress_peer_safe` before the report was written; a test now covers it). Scoring reads the step declarations, not DAIL's output. (3) **Excluded operators**: `op_f5_count_each_safe` and the four F1/F4 unsafe operators whose sentence adds an ingress "on the group" without saying whether it is a new resource. (4) **Repair success** is per scenario, so it is reported on all transitions only, not on a common prefix. (5) **MetricRecord** carries Appendix B's fields plus `scope`, `stratum` and `wilson95`; a denominator of 0 gives `value = null`; the Wilson interval is clamped to [0, 1] (floating rounding gave -0.000 for k = 0; the first generated report of run 2 was deleted and regenerated after that fix, before anything referenced it). (6) **The 20-trial re-run** is a new experiment, `p8b-repro-check`, on the same code, compared with run 2 on (scenario, condition, step): a new attempt of a COMPLETED trial is not a transition Doc 13 §7 allows, and no stored record is touched. The 4 extra F8 are those with most steps among the F8 not already chosen, ties by lowest scenario id. (7) **The dev report** `docs/P8_DEV_PILOT.json` (the ExperimentReport, about 1 MB) and its rendering `docs/P8_DEV_PILOT.md` are committed; a test renders the JSON and compares the file. (8) **Agreement** (item 8) uses B's verification results: at S0 the baseline record's results, at every evaluated step the results of every invariant item B ran. | Proposed (P8b) | +| C-212 | P8b-1: metric definitions | The metrics of the P8b work package Section 4, exactly, versioned `metrics-draft-1`. They are a draft for the dev pilot; P10 freezes the final definitions before the test split is opened. | Decided (authors, 13 Nov 2026): accepted as built after external review | +| C-213 | P8b-2: identity truth for H3 | Identity truth comes from a hand-written table in DAIL, `experiments/scoring/identity_truth.json`: one row per TerraPreserve operator that moves, replaces, deletes or creates a resource, giving the expected change type at the operator's target address, with the sentence of `D4_OPERATORS.md` it comes from. An operator whose identity truth the catalogue does not state gets no row, and its pairs are excluded and counted. The table is written from the catalogue's text, never from DAIL's output. | Decided (authors, 13 Nov 2026): accepted as built after external review | +| C-214 | P8b-3: statistics on dev are descriptive only | Counts, proportions with numerator and denominator, Wilson 95% intervals, and per-cluster summaries (cluster = `template_id` for the controlled tier, `repo_lineage_id` for the real-world tier). No hypothesis test, no p-value, no "significant". Inference is for the test split after the protocol freeze (P10). | Decided (authors, 13 Nov 2026): accepted as built after external review | +| C-215 | P8b: readings made while building scoring, the extractor and the report | (1) **Evidence-transfer error** (item 10, not defined in the work package): a true replacement that DAIL treats as the same object (neither REPLACED nor UNCERTAIN) in a transition where some required invariant was CARRIED_FORWARD. (2) **Appearing and vanishing resources** for the identity truth are those declared (a top-level `resource` block in the step's `.tf` files) in step i and not in step i−1, or the reverse, restricted to the addresses the operator's mutation rows name; the first version used "every row has no `before`", which wrongly made an existing group that only gains an attribute a CREATE (found on `op_f3_db_ingress_peer_safe` before the report was written; a test now covers it). Scoring reads the step declarations, not DAIL's output. (3) **Excluded operators**: `op_f5_count_each_safe` and the four F1/F4 unsafe operators whose sentence adds an ingress "on the group" without saying whether it is a new resource. (4) **Repair success** is per scenario, so it is reported on all transitions only, not on a common prefix. (5) **MetricRecord** carries Appendix B's fields plus `scope`, `stratum` and `wilson95`; a denominator of 0 gives `value = null`; the Wilson interval is clamped to [0, 1] (floating rounding gave -0.000 for k = 0; the first generated report of run 2 was deleted and regenerated after that fix, before anything referenced it). (6) **The 20-trial re-run** is a new experiment, `p8b-repro-check`, on the same code, compared with run 2 on (scenario, condition, step): a new attempt of a COMPLETED trial is not a transition Doc 13 §7 allows, and no stored record is touched. The 4 extra F8 are those with most steps among the F8 not already chosen, ties by lowest scenario id. (7) **The dev report** `docs/P8_DEV_PILOT.json` (the ExperimentReport, about 1 MB) and its rendering `docs/P8_DEV_PILOT.md` are committed; a test renders the JSON and compares the file. (8) **Agreement** (item 8) uses B's verification results: at S0 the baseline record's results, at every evaluated step the results of every invariant item B ran. | Decided (authors, 13 Nov 2026): accepted as built after external review | | C-216 | P8b-2: baseline items under registry v1.1 (the version-1 lookup in `core/verification/baseline.py`) | Baseline items (C-148) use the definition that the run's definitions hold for the invariant they belong to (INV-SEC-001 for security groups and rules, INV-SEC-003 for database instances). If more than one version is present, the highest is used. The item's evidence records that id and version, and the §3.3 clause it applies is that definition's registry (version 1 → registry-v1.0, version 2 → registry-v1.1), so a baseline item and its invariant item can never use different rules. If no definition is present, the item stays UNSUPPORTED `invariant.unknown`, as now. Found by the external review of PR #31; it affected every run under registry v1.1. | Decided (authors, 13 Nov 2026) | -| C-217 | P8b-2: readings made in the fix round | (1) **An unproven required check.** DAIL records a required check that is not proven with no proof source (only VERIFIED and CARRIED_FORWARD are sources). The P8b extractor read a fresh result only for VERIFIED checks, so the UNKNOWN and UNSUPPORTED rates (run 2: 0 for every condition) and the new non-PASS table missed every failing or uncertain check, and the guard could not fire on real data; the hand-built record set had marked every check VERIFIED. Found while writing the non-PASS table; the extractor now takes the fresh result of every check that is not CARRIED_FORWARD, the record set records sources as DAIL does, and a mutant that restores the old rule fails 3 tests. (2) **Regenerated report.** The first report of run 3 was written by commit a34e25a's extractor with that rule; it was deleted and regenerated after the fix, before anything referenced it (as for C-215 (5)). The extractor version stays `metrics-draft-1` because the metric definitions (C-212) did not change. (3) **Evidence fields** of a baseline item: `observations.invariant_id`, `invariant_version` and `registry` (C-216). (4) **Guard on real data:** on run 2's records the guard flags 144 baseline items (B 32 groups and 14 rules, C 32 and 14, C' 32 and 20); on run 3's, none. | Proposed (P8b-2) | +| C-217 | P8b-2: readings made in the fix round | (1) **An unproven required check.** DAIL records a required check that is not proven with no proof source (only VERIFIED and CARRIED_FORWARD are sources). The P8b extractor read a fresh result only for VERIFIED checks, so the UNKNOWN and UNSUPPORTED rates (run 2: 0 for every condition) and the new non-PASS table missed every failing or uncertain check, and the guard could not fire on real data; the hand-built record set had marked every check VERIFIED. Found while writing the non-PASS table; the extractor now takes the fresh result of every check that is not CARRIED_FORWARD, the record set records sources as DAIL does, and a mutant that restores the old rule fails 3 tests. (2) **Regenerated report.** The first report of run 3 was written by commit a34e25a's extractor with that rule; it was deleted and regenerated after the fix, before anything referenced it (as for C-215 (5)). The extractor version stays `metrics-draft-1` because the metric definitions (C-212) did not change. (3) **Evidence fields** of a baseline item: `observations.invariant_id`, `invariant_version` and `registry` (C-216). (4) **Guard on real data:** on run 2's records the guard flags 144 baseline items (B 32 groups and 14 rules, C 32 and 14, C' 32 and 20); on run 3's, none. | Decided (authors, 13 Nov 2026): accepted as built after external review | +| C-218 | P10a: where LLM candidates are generated | On the authors' own machine only, with the local Ollama server. The test split's inputs never leave the machine. Model size is limited by what runs on that machine's CPU; the dev pilot (P10a Step 6) measures the time. | Decided (authors, 13 Nov 2026) | +| C-219 | P10a: which scenarios get LLM candidates, and the task | Repair scenarios only (`target.kind = "repair"`: F2, F4, F8). The task given to the model is a fixed template per target invariant: "Repair ``: ``. Keep every other protected invariant satisfied. Make the smallest change." TerraPreserve's `target.description` and `intent` are never given to the model: the change-request descriptions name the damage that makes a scenario unsafe. | Decided (authors, 13 Nov 2026) | +| C-220 | P10a: the scope of an LLM patch (closes P7b verdict limit 7) | For every scenario, `AllowedChangeScope` = every resource address of S0, every supported type, no attribute restriction, and all four operations. It is the same for every condition and is recorded with each candidate. DAIL alone judges safety, so the scope does not filter unsafe edits. | Decided (authors, 13 Nov 2026) | +| C-221 | P10a: a patch that drops the parent's `lifecycle` block (closes P7b verdict limit 4) | A patch that drops the parent's `lifecycle` block is refused: POLICY_VIOLATION before DAIL, counted as an invalid patch. Neither DAIL nor the registry models lifecycle settings. Built in `llm/validation/patch.py`, validator `dail-patch-validator-2`. | Decided (authors, 13 Nov 2026) | diff --git a/docs/PHASE_GATES.md b/docs/PHASE_GATES.md index 44bfcd7..e443e17 100644 --- a/docs/PHASE_GATES.md +++ b/docs/PHASE_GATES.md @@ -555,7 +555,26 @@ Limits: the open-target rule blocked C and C' on F8 (C-209, closed by C-210 in P Carry-forward: C-200 to C-204 and C-207 Proposed; C-209 closed by C-210; P8b scores the stored records. ``` -## P8b — Scoring, metrics and the dev report (pending external review) +## P8b — Scoring, metrics and the dev report — passed with limits (verdict on PR #31) + +```text +P8b: passed with limits (verdict on PR #31). Recorded in P10a (13 Nov 2026). +Limits of the verdict (open items, with the package that closes each): +The verdict's own wording of each limit was not pasted into P10a; the lines below give what the +P10a work package says of each. +1. C-212 to C-215 and C-217 Proposed. [Closed: Decided by the authors on 13 Nov 2026, "accepted as + built after external review", recorded in P10a.] +2. The extractor version (P10a Section 3.1). [Closed in P10a: + EXTRACTOR_VERSION is extractor-2, METRICS_VERSION is metrics-draft-1; every metric record and + report carries both, and a test pins the hash of the extractor modules.] +3. tests/unit/test_e2e_concurrency.py, the named interleavings (P10a Section 3.1). [Closed in P10a: + the "A_done"/"B_done" markers are added; 50 loaded runs are reported in the P10a section.] +4.-8. NOT YET RECORDED: the text of the verdict's limits 4 to 8 was not pasted into P10a; they are + recorded when the user pastes it. +``` + +The P8b author's input to the review, as written before the verdict (the run-2 numbers in it are +superseded by run 3, C-216): ```text P8b mini-gate: NOT DECIDED. Status written by the P8b author as an input to the external review. @@ -575,6 +594,7 @@ Fix round P8b-2 (13 Nov 2026, after the external review of PR #31 at 86a1e5a): b 136 common transitions, 0 differences. An extractor reading found in the fix round (C-217 (1)): unproven required checks now count in the UNKNOWN/UNSUPPORTED rates. The numbers below this paragraph are those of the first P8b report (run 2), kept as written. + [Superseded by run 3 (C-216).] What exists: experiments/scoring (the only label reader: labels with the manifest and scenario hash checks, per-transition truth, the identity truth table identity-truth-1 written from D4_OPERATORS.md, @@ -605,10 +625,12 @@ Limits: metrics-draft-1 is a draft (P10 freezes the definitions before the test Carry-forward: C-212 to C-215 Proposed. ``` -## P8 — Experiment Harness (pending external review) +## P8 — Experiment Harness — passed with limits (verdict on PR #31) ```text -P8 gate (Doc 15 §15.3) and milestone M5 (Doc 15 §25): NOT DECIDED. Each item and what bears on it: +P8 gate (Doc 15 §15.3) and milestone M5 (Doc 15 §25): passed with limits (verdict on PR #31). +Recorded in P10a (13 Nov 2026). The limits are those of the P8b verdict, listed in the P8b section. +The P8b author's map of each item, as written before the verdict: §15.3 "Trials are reproducible" ........ tests.unit.test_experiment_harness.TestReproducibility (records identical across runs, concurrency, corrupted labels); the 20-trial re-run (P8_DEV_PILOT.md section 13); the reviewer's independent re-run of both dev runs (PR #30). From bff1a77b093584c831c1fca97fc7edaa81b192f9 Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Fri, 13 Nov 2026 19:51:24 +0530 Subject: [PATCH 2/7] =?UTF-8?q?P10a=20step=202:=20tests=20first:=20lifecyc?= =?UTF-8?q?le=20rule,=20generation=20blinding,=20scope=20across=20conditio?= =?UTF-8?q?ns,=20digest=20refusal,=20freezing,=20replay=20rebuild=20hash,?= =?UTF-8?q?=20oracle=20export,=20extractor=20versions=20(Doc=2010=20=C2=A7?= =?UTF-8?q?13,=20=C2=A716;=20Doc=2013=20=C2=A726,=20=C2=A747;=20C-219=20to?= =?UTF-8?q?=20C-221)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- tests/contract/test_experiment_blinding.py | 116 ++++++ tests/unit/test_p10a_extractor_version.py | 119 ++++++ tests/unit/test_p10a_generation.py | 445 +++++++++++++++++++++ tests/unit/test_p10a_lifecycle.py | 120 ++++++ 4 files changed, 800 insertions(+) create mode 100644 tests/unit/test_p10a_extractor_version.py create mode 100644 tests/unit/test_p10a_generation.py create mode 100644 tests/unit/test_p10a_lifecycle.py diff --git a/tests/contract/test_experiment_blinding.py b/tests/contract/test_experiment_blinding.py index b46632f..a7bb989 100644 --- a/tests/contract/test_experiment_blinding.py +++ b/tests/contract/test_experiment_blinding.py @@ -16,11 +16,23 @@ not a docstring (a key, a ``get`` or ``getattr`` argument, a set or list member), an attribute, a name or a keyword argument. A self-test feeds sources that read each field, so a checker that can no longer fail is caught. + +P10a (work package Section 3.3): "Extend the blinding contract test: ``experiments/generation/`` +names no label field and no ``description``, ``intent``, ``mutation_record`` or later step, and never +opens ``step_1`` or beyond. Show the test failing on a mutant." (c) Every module of +``experiments/generation/`` is checked for the label fields and for ``description`` (TerraPreserve's +``target.description``); a later step is any string naming ``step_`` with n >= 1, and any use of +a scenario's ``steps`` other than ``steps[0]`` or ``steps[:1]`` (an index, a loop, a slice past S0). +The invariant's own registry statement reaches the task through ``experiments.task_text``, which +reads a DAIL ``Invariant`` only. (d) No generation module imports ``experiments.scoring``, +``.metrics`` or ``.report``. A dynamic check is in ``tests.unit.test_p10a_generation.TestNoLaterStep``: +generation, replay and export run with the later step directories deleted. """ from __future__ import annotations import ast +import re import unittest from pathlib import Path @@ -32,6 +44,10 @@ # P8b: the only package that reads label fields. SCORING_PACKAGE = EXPERIMENTS / "scoring" ANALYSIS_PACKAGES = ("experiments.scoring", "experiments.metrics", "experiments.report") +# P10a: the generation package. +GENERATION_PACKAGE = EXPERIMENTS / "generation" +GENERATION_FORBIDDEN = FORBIDDEN | {"description"} +LATER_STEP = re.compile(r"step_0*[1-9][0-9]*") def _docstrings(tree: ast.AST) -> set[int]: @@ -67,6 +83,61 @@ def label_reads(source: str) -> list[str]: return found +def _first_step_only(node: ast.Subscript) -> bool: + """``steps[0]`` or ``steps[:1]``.""" + index = node.slice + if isinstance(index, ast.Constant): + return index.value == 0 + if isinstance(index, ast.Slice): + return ( + index.lower is None + and index.step is None + and isinstance(index.upper, ast.Constant) + and index.upper.value == 1 + ) + return False + + +def generation_reads(source: str) -> list[str]: + """(c): label fields, ``description``, a later step's name, and any use of ``steps`` other than + ``steps[0]`` or ``steps[:1]``, as 'line: what'.""" + tree = ast.parse(source) + skip = _docstrings(tree) + found: list[str] = [] + allowed_steps: set[int] = set() + for node in ast.walk(tree): + if ( + isinstance(node, ast.Subscript) + and isinstance(node.value, ast.Attribute) + and node.value.attr == "steps" + and _first_step_only(node) + ): + allowed_steps.add(id(node.value)) + for node in ast.walk(tree): + name: str | None = None + if isinstance(node, ast.Constant) and isinstance(node.value, str) and id(node) not in skip: + if node.value in GENERATION_FORBIDDEN: + name = node.value + elif LATER_STEP.search(node.value): + name = f"later step {node.value!r}" + elif isinstance(node, ast.Attribute): + if node.attr in GENERATION_FORBIDDEN: + name = node.attr + elif node.attr == "steps" and id(node) not in allowed_steps: + name = "steps other than steps[0] or steps[:1]" + elif isinstance(node, ast.Name) and node.id in GENERATION_FORBIDDEN: + name = node.id + elif isinstance(node, ast.keyword) and node.arg in GENERATION_FORBIDDEN: + name = node.arg + if name is not None: + found.append(f"{getattr(node, 'lineno', '?')}: {name}") + return found + + +def generation_modules() -> list[Path]: + return sorted(p for p in GENERATION_PACKAGE.rglob("*.py") if "__pycache__" not in p.parts) + + def non_scoring_modules() -> list[Path]: """Every module of experiments/ outside experiments/scoring/ (check (a)).""" return sorted( @@ -129,6 +200,51 @@ def test_execution_never_imports_scoring_metrics_or_report(self) -> None: self.assertEqual(analysis_imports(path.read_text(encoding="utf-8")), []) +class TestGenerationBlinding(unittest.TestCase): + def test_generation_names_no_label_description_or_later_step(self) -> None: + modules = generation_modules() + self.assertIn(GENERATION_PACKAGE / "run.py", modules) + self.assertIn(GENERATION_PACKAGE / "task.py", modules) + for path in modules: + with self.subTest(module=path.relative_to(REPO_ROOT).as_posix()): + self.assertEqual(generation_reads(path.read_text(encoding="utf-8")), []) + + def test_generation_never_imports_scoring_metrics_or_report(self) -> None: + for path in generation_modules(): + with self.subTest(module=path.relative_to(REPO_ROOT).as_posix()): + self.assertEqual(analysis_imports(path.read_text(encoding="utf-8")), []) + + +class TestTheGenerationCheckerCanFail(unittest.TestCase): + def test_each_kind_of_read_is_found(self) -> None: + for source in ( + 'text = doc["target"]["description"]\n', + "text = target.description\n", + 'value = doc.get("intent")\n', + 'value = doc["mutation_record"]\n', + 'value = doc["labels"]\n', + "later = scenario.steps[1]\n", + "last = scenario.steps[-1]\n", + "rest = scenario.steps[1:]\n", + "both = scenario.steps[:2]\n", + "for step in scenario.steps:\n pass\n", + "count = len(scenario.steps)\n", + 'path = base / "step_1"\n', + 'path = base / f"step_{k}" / "step_12"\n', + ): + with self.subTest(source=source.strip()): + self.assertNotEqual(generation_reads(source), []) + + def test_s0_is_allowed(self) -> None: + source = ( + '"""Never the description, nor step_1."""\n' + "first = scenario.steps[0]\n" + "only = scenario.steps[:1]\n" + 'path = base / "step_0"\n' + ) + self.assertEqual(generation_reads(source), []) + + class TestTheCheckerCanFail(unittest.TestCase): def test_each_kind_of_read_is_found(self) -> None: for source in ( diff --git a/tests/unit/test_p10a_extractor_version.py b/tests/unit/test_p10a_extractor_version.py new file mode 100644 index 0000000..5177e1e --- /dev/null +++ b/tests/unit/test_p10a_extractor_version.py @@ -0,0 +1,119 @@ +"""The extractor version, apart from the metric definitions version (P10a work package Section 3.1; +closes the P8b verdict's limit 2). + +Section 3.1: "``EXTRACTOR_VERSION`` becomes its own value (``extractor-2``), separate from the metric +definitions version (``metrics-draft-1``). Every ``MetricRecord`` and report carries both. A test +fails if the extractor's code hash changes without a version change (pin the hash of the extractor +modules in a test)." + +The extractor modules are the code that turns stored records into metric records: +``experiments/metrics/`` and the scoring it reads truth through (``experiments/scoring/``, with its +identity table). Their hash is the SHA-256 of the path-sorted lines ``":"``, so a Windows checkout and CI agree. +A changed module needs a new ``EXTRACTOR_VERSION`` and a new entry in ``PINNED``; an old entry +stays, so the record of what each version was is kept. +""" + +from __future__ import annotations + +import hashlib +import unittest +from pathlib import Path + +from tests.fixture_catalog import REPO_ROOT +from tests.metrics_builders import GATES, scenarios + +PINNED: dict[str, str] = { + "extractor-2": "PIN-ME", +} +EXTRACTOR_PACKAGES = ("experiments/metrics", "experiments/scoring") +CRLF, LF = b"\r\n", b"\n" + + +def extractor_files() -> list[Path]: + found: list[Path] = [] + for package in EXTRACTOR_PACKAGES: + base = REPO_ROOT / package + found += [ + p + for p in base.rglob("*") + if p.is_file() and "__pycache__" not in p.parts and p.suffix in (".py", ".json") + ] + return sorted(found, key=lambda p: p.relative_to(REPO_ROOT).as_posix()) + + +def hash_of(entries: list[tuple[str, bytes]]) -> str: + lines = [ + f"{name}:{hashlib.sha256(data.replace(CRLF, LF)).hexdigest()}" + for name, data in sorted(entries) + ] + return hashlib.sha256("\n".join(lines).encode("utf-8")).hexdigest() + + +def entries() -> list[tuple[str, bytes]]: + return [(p.relative_to(REPO_ROOT).as_posix(), p.read_bytes()) for p in extractor_files()] + + +def extractor_hash() -> str: + return hash_of(entries()) + + +class TestVersions(unittest.TestCase): + def test_the_two_versions_are_separate_values(self) -> None: + from experiments.metrics.extractor import EXTRACTOR_VERSION, METRICS_VERSION + + self.assertEqual(EXTRACTOR_VERSION, "extractor-2") + self.assertEqual(METRICS_VERSION, "metrics-draft-1") + + def test_every_metric_record_and_the_report_carry_both(self) -> None: + from experiments.metrics.extractor import extract + from experiments.metrics.model import Transition + from experiments.report.build import build_report + from experiments.scoring.identity import load_identity_table + + transition = Transition( + "t", "Z", "C", 1, "EVALUATED", "PROMOTE", (), (), GATES, frozenset(), (), None + ) + x = extract( + experiment_id="e", + transitions=[transition], + scored=scenarios(), + baselines={}, + identity=load_identity_table(), + ) + self.assertTrue(x.metrics) + for record in x.metrics: + self.assertEqual(record["metrics_version"], "metrics-draft-1") + self.assertEqual(record["extractor_version"], "extractor-2") + definition = {"experiment_id": "e", "name": "n", "version": "1", "condition_set": ["C"]} + report = build_report(definition=definition, statuses={"t": "COMPLETED"}, extraction=x) + self.assertEqual(report["metrics_version"], "metrics-draft-1") + self.assertEqual(report["extractor_version"], "extractor-2") + + +class TestCodePin(unittest.TestCase): + def test_the_extractor_modules_have_the_hash_pinned_for_their_version(self) -> None: + from experiments.metrics.extractor import EXTRACTOR_VERSION + + self.assertIn(REPO_ROOT / "experiments" / "metrics" / "extractor.py", extractor_files()) + self.assertIn( + EXTRACTOR_VERSION, PINNED, "a new extractor version needs its pinned code hash" + ) + self.assertEqual( + extractor_hash(), + PINNED[EXTRACTOR_VERSION], + "the extractor's code changed: give it a new EXTRACTOR_VERSION and pin its hash", + ) + + def test_the_hash_sees_a_one_byte_change_and_ignores_crlf(self) -> None: + found = entries() + self.assertGreater(len(found), 3) + name, data = found[0] + changed = [(name, data + b" ")] + found[1:] + self.assertNotEqual(hash_of(changed), hash_of(found)) + crlf = [(n, d.replace(CRLF, LF).replace(LF, CRLF)) for n, d in found] + self.assertEqual(hash_of(crlf), hash_of(found)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unit/test_p10a_generation.py b/tests/unit/test_p10a_generation.py new file mode 100644 index 0000000..8686a3d --- /dev/null +++ b/tests/unit/test_p10a_generation.py @@ -0,0 +1,445 @@ +"""LLM candidates for the experiment (P10a work package Section 3.3; decisions 3 to 5, C-218 to C-220). + +Expected values come from the quoted text: + +Decision 4: "repair scenarios only (``target.kind = "repair"``) ... The task given to the model is a +fixed template per target invariant: "Repair ````: ````. Keep every other protected invariant satisfied. Make the smallest change." +TerraPreserve's ``target.description`` and ``intent`` are never given to the model." +Decision 5: "``AllowedChangeScope`` = every resource address of S0, every supported type, no +attribute restriction, and all four operations. It is the same for every condition and is recorded +with each candidate." +Section 3.3 "Model configuration": "The run refuses a model whose installed digest is not the pinned +one." "Freezing: each attempt (valid or not) is stored under ``$DAIL_EXPERIMENT_ROOT/candidates/ +////`` ... A ``candidates_manifest.json`` lists every attempt +with hashes ... Frozen candidates are never regenerated or edited." "Replay: ... The directory is +rebuilt by the validator from the frozen patch in the same call, then deleted ... Each run checks +that the rebuilt tree hash equals the frozen one. Invalid attempts are recorded as generation +failures, never as decisions." "Export for the oracle: for every valid candidate, a +configuration-only directory (the built ``.tf`` files and nothing else) plus an +``oracle_export.json`` with ids and hashes." "``generation_enabled`` is turned on only in the +harness's reviewed configuration copy, the same way as C-207 (3)." + +Everything runs on the mini TerraPreserve tree with the fake provider: no network. +""" + +from __future__ import annotations + +import dataclasses +import hashlib +import json +import shutil +import unittest +from pathlib import Path +from typing import Any + +from core.application.config import load_config +from core.domain.change_scope import ALL_OPERATIONS +from experiments.loader import LoadedSet, load +from llm.adapter.context import SUPPORTED_TYPES +from llm.adapter.fake import FakeProvider +from llm.adapter.provider import ProviderError, ProviderReply +from llm.schemas.records import FailureClass +from tests.experiment_builders import F1, F2, F4, F8, TempDirs, mini_root +from tests.fixture_catalog import REPO_ROOT +from tests.llm_patch_builders import DB_RULE, SG_APP, SG_APP_CLOSED, SSH_RULE, repair + +DIGEST = "b" * 64 +TAG = "fake-coder:1b" +VALID = repair(SG_APP_CLOSED, [SG_APP, SSH_RULE]) +BROKEN = repair('resource "aws_security_group" "app" {\n', [SG_APP]) +CONDITIONS = ("A", "B", "C", "C_PRIME") + + +def ok(text: str) -> ProviderReply: + return ProviderReply(text=text, latency_ms=12.0, token_usage={"input": 100, "output": 20}) + + +def pin(**changes: Any) -> Any: + from experiments.generation.models import ModelPin + + values: dict[str, Any] = { + "key": "fake-coder-1b", + "tag": TAG, + "digest": DIGEST, + "quantization": "Q4_K_M", + "size": "1 GB", + "licence": "Apache-2.0", + "licence_source": "test", + "temperature": 0.0, + "top_p": 1.0, + "seed": 11, + "max_output_tokens": 1024, + "timeout_seconds": 30.0, + "num_ctx": 32768, + } + values.update(changes) + return ModelPin(**values) + + +def installed(digest: str = DIGEST) -> dict[str, str]: + return {"name": TAG, "digest": digest, "quantization": "Q4_K_M"} + + +class Case(unittest.TestCase): + def setUp(self) -> None: + self.dirs = TempDirs() + self.addCleanup(self.dirs.close) + mini = mini_root(self.dirs.new()) + self.loaded: LoadedSet = load(mini.root, manifest_sha256=mini.manifest_sha256) + self.by_id = {s.scenario_id: s for s in self.loaded.scenarios} + self.out = self.dirs.new("dail-p10a-out-") + + def inputs(self, *ids: str) -> list[Any]: + from experiments.generation.task import generation_input + + return [generation_input(self.by_id[sid]) for sid in ids] + + def generate(self, script: list[Any], ids: tuple[str, ...] = (F2,), **options: Any) -> Any: + from experiments.generation.run import GenerationSpec, generate_candidates + + provider = options.pop("provider", None) or FakeProvider(script) + self.provider = provider + settings: dict[str, Any] = { + "spec": GenerationSpec(name="p10a-test", version="1"), + "inputs": self.inputs(*ids), + "models": [(pin(), provider, installed())], + "k": 1, + "root": self.out, + "generation_enabled": True, + "environment": {"dail_commit": "test"}, + } + settings.update(options) + return generate_candidates(**settings) + + def attempt_dir(self, summary: Any, sid: str = F2, k: int = 1) -> Path: + return self.out / "candidates" / summary.generation_id / "fake-coder-1b" / sid / str(k) + + def replay(self, summary: Any, name: str = "replay-1") -> Any: + from experiments.generation.replay import replay_candidates + from experiments.runner import experiment_config + + return replay_candidates( + root=self.out, + generation_id=summary.generation_id, + scenarios=self.by_id, + conditions=CONDITIONS, + config=experiment_config(), + replay_name=name, + ) + + +class TestTask(Case): + def test_only_repair_scenarios_get_candidates(self) -> None: + from experiments.generation.task import GenerationError, generation_input + + with self.assertRaises(GenerationError): + generation_input(self.by_id[F1]) # a change request + for sid in (F2, F4, F8): + self.assertEqual(generation_input(self.by_id[sid]).target_invariant_id, "INV-SEC-001") + + def test_the_input_holds_s0_only_and_the_listed_fields(self) -> None: + from experiments.generation.task import GenerationInput + + fields = {f.name for f in dataclasses.fields(GenerationInput)} + self.assertEqual( + fields, + { + "scenario_id", + "family", + "s0_dir", + "s0_content_hash", + "target_invariant_id", + "source_endpoint", + "destination_endpoint", + "variable_values", + }, + ) + (given,) = self.inputs(F8) + self.assertEqual(given.s0_dir, self.by_id[F8].steps[0].directory) + self.assertEqual(given.s0_content_hash, self.by_id[F8].steps[0].content_hash) + + def test_the_task_is_the_fixed_template(self) -> None: + from datetime import UTC, datetime + + from core.domain.registry_v1_1 import registry_v1_1_invariants + from experiments.generation.task import objective_for + + (definition,) = [ + d + for d in registry_v1_1_invariants(datetime(2026, 1, 1, tzinfo=UTC), source="a.s", destination="b.d") + if d.invariant_id == "INV-SEC-001" + ] + self.assertEqual( + objective_for(definition), + f"Repair INV-SEC-001: {definition.description} Keep every other protected invariant " + "satisfied. Make the smallest change.", + ) + + def test_the_prompt_holds_no_description_or_intent(self) -> None: + summary = self.generate([ok(VALID)]) + (call,) = self.provider.calls + scenario_doc = json.loads( + (self.by_id[F2].steps[0].directory.parent / "scenario.json").read_text("utf-8") + ) + for text in (call.system, call.task): + self.assertNotIn(scenario_doc["target"]["description"], text) + self.assertNotIn("junk", text) # every label and intent field holds "junk" + self.assertIn("Repair INV-SEC-001: ", call.task) + self.assertEqual(summary.attempts[0]["status"], "PATCH") + + +class TestScope(Case): + def test_the_scope_is_recorded_and_identical_across_conditions(self) -> None: + summary = self.generate([ok(VALID)]) + frozen = json.loads((self.attempt_dir(summary) / "attempt.json").read_text("utf-8")) + scope = frozen["scope"] + s0_addresses = frozen["parent"]["resource_addresses"] + self.assertEqual( + scope, + { + "resource_addresses": sorted(s0_addresses), + "resource_types": list(SUPPORTED_TYPES), + "attributes": [], + "operation_types": [op.value for op in ALL_OPERATIONS], + }, + ) + self.assertIn(SSH_RULE, scope["resource_addresses"]) + self.assertIn(DB_RULE, scope["resource_addresses"]) + replayed = self.replay(summary) + self.assertEqual(sorted(r["condition"] for r in replayed.results), sorted(CONDITIONS)) + for result in replayed.results: + self.assertEqual(result["scope"], scope, result["condition"]) + + +class TestDigest(Case): + def test_a_model_whose_installed_digest_is_not_the_pinned_one_is_refused(self) -> None: + from experiments.generation.models import ModelRefused + + provider = FakeProvider([ok(VALID)]) + with self.assertRaises(ModelRefused): + self.generate([], provider=provider, models=[(pin(), provider, installed("c" * 64))]) + self.assertEqual(provider.calls, []) + self.assertFalse((self.out / "candidates").exists() and any( + p.name == "attempt.json" for p in (self.out / "candidates").rglob("*") + )) + + def test_a_missing_model_is_refused(self) -> None: + from experiments.generation.models import ModelRefused, check_installed + + with self.assertRaises(ModelRefused): + check_installed(pin(), {"name": "other:1b", "digest": DIGEST}) + + +class TestFreezing(Case): + def test_each_attempt_and_the_manifest_are_stored_with_hashes(self) -> None: + summary = self.generate([ok(VALID), ok(BROKEN), ok(BROKEN)], ids=(F2, F4)) + manifest_path = self.out / "candidates" / summary.generation_id / "candidates_manifest.json" + raw = manifest_path.read_bytes() + self.assertEqual(summary.manifest_sha256, hashlib.sha256(raw).hexdigest()) + manifest = json.loads(raw) + rows = {(r["scenario_id"], r["k"]): r for r in manifest["attempts"]} + self.assertEqual(set(rows), {(F2, 1), (F4, 1)}) + valid, invalid = rows[(F2, 1)], rows[(F4, 1)] + self.assertEqual(valid["status"], "PATCH") + self.assertIsNone(valid["failure_class"]) + self.assertEqual(invalid["status"], "FAILED") + self.assertEqual(invalid["failure_class"], "INVALID_OUTPUT") + patch = (self.attempt_dir(summary) / "patch.tf").read_bytes() + self.assertEqual(valid["patch_sha256"], hashlib.sha256(patch).hexdigest()) + self.assertIsNone(invalid["patch_sha256"]) + self.assertFalse((self.attempt_dir(summary, F4) / "patch.tf").exists()) + for row in rows.values(): + base = self.out / "candidates" / summary.generation_id + for name, digest in row["files"].items(): + self.assertEqual(hashlib.sha256((base / name).read_bytes()).hexdigest(), digest) + # the request and response records of every attempt, retries included (P7's records) + records = sorted(p.name for p in (self.attempt_dir(summary, F4) / "records").iterdir()) + self.assertEqual(records, ["request-1.json", "request-2.json", "response-1.json", "response-2.json"]) + + def test_a_frozen_generation_is_never_regenerated(self) -> None: + from experiments.generation.run import GenerationError + + summary = self.generate([ok(VALID)]) + before = {p: p.read_bytes() for p in self.attempt_dir(summary).rglob("*") if p.is_file()} + provider = FakeProvider([ok(BROKEN)]) + with self.assertRaises(GenerationError): + self.generate([], provider=provider, models=[(pin(), provider, installed())]) + self.assertEqual(provider.calls, []) + after = {p: p.read_bytes() for p in self.attempt_dir(summary).rglob("*") if p.is_file()} + self.assertEqual(after, before) + + def test_an_interrupted_attempt_is_never_overwritten(self) -> None: + from experiments.generation.run import GenerationError, GenerationSpec + + spec = GenerationSpec(name="p10a-test", version="1") + partial = self.out / "candidates" / spec.generation_id / "fake-coder-1b" / F2 / "1" + partial.mkdir(parents=True) + (partial / "records").mkdir() + with self.assertRaises(GenerationError): + self.generate([ok(VALID)]) + self.assertEqual(list(partial.iterdir()), [partial / "records"]) + + def test_generation_needs_the_switch(self) -> None: + summary = self.generate([ok(VALID)], generation_enabled=False) + self.assertEqual(self.provider.calls, []) + self.assertEqual(summary.attempts[0]["status"], "DISABLED") + + def test_the_switch_is_on_only_in_the_reviewed_copy(self) -> None: + from experiments.generation.run import EXPERIMENT_LLM_FILE, generation_config + + self.assertIs(load_config(REPO_ROOT / "config", "experiment", environ={}).get( + "llm.generation_enabled" + ), False) + self.assertIs(generation_config().get("llm.generation_enabled"), True) + self.assertEqual(EXPERIMENT_LLM_FILE, REPO_ROOT / "experiments" / "config" / "llm.toml") + summary = self.generate([ok(VALID)]) + definition = json.loads( + (self.out / "candidates" / summary.generation_id / "definition.json").read_text("utf-8") + ) + self.assertEqual( + definition["experiment_llm_config_sha256"], + hashlib.sha256(EXPERIMENT_LLM_FILE.read_bytes()).hexdigest(), + ) + + +class TestReplay(Case): + def test_a_valid_candidate_is_rebuilt_checked_evaluated_and_deleted(self) -> None: + summary = self.generate([ok(VALID)]) + replayed = self.replay(summary) + self.assertEqual(len(replayed.results), 4) + frozen = json.loads((self.attempt_dir(summary) / "attempt.json").read_text("utf-8")) + for result in replayed.results: + self.assertTrue(result["tree_hash_matches"]) + self.assertEqual(result["rebuilt_tree_hash"], frozen["candidate_tree_hash"]) + self.assertIn(result["decision"], ("PROMOTE", "REJECT")) + self.assertFalse(Path(result["build_dir"]).exists()) # deleted after the call + by_condition = {r["condition"]: r["decision"] for r in replayed.results} + self.assertEqual(by_condition["C"], "PROMOTE") # SSH closed, the path kept + + def test_an_edited_frozen_patch_is_refused_without_a_decision(self) -> None: + summary = self.generate([ok(VALID)]) + responses = self.attempt_dir(summary) / "records" / "response-1.json" + record = json.loads(responses.read_text("utf-8")) + record["raw_response"] = record["raw_response"].replace('name = \\"app\\"', 'name = \\"app2\\"') + responses.chmod(0o666) + responses.write_text(json.dumps(record), "utf-8") + replayed = self.replay(summary) + for result in replayed.results: + self.assertFalse(result["tree_hash_matches"]) + self.assertIsNone(result["decision"]) + + def test_invalid_attempts_are_generation_failures_not_decisions(self) -> None: + summary = self.generate([ok(BROKEN), ok(BROKEN)]) + replayed = self.replay(summary) + self.assertEqual(replayed.results, []) + self.assertEqual( + [(f["scenario_id"], f["failure_class"]) for f in replayed.generation_failures], + [(F2, "INVALID_OUTPUT")], + ) + + def test_a_replay_is_written_once(self) -> None: + from experiments.generation.run import GenerationError + + summary = self.generate([ok(VALID)]) + self.replay(summary) + with self.assertRaises(GenerationError): + self.replay(summary) + + +class TestExport(Case): + def test_the_export_holds_only_the_built_tf_files_and_its_index(self) -> None: + from experiments.generation.export import export_for_oracle + + summary = self.generate([ok(VALID), ok(BROKEN), ok(BROKEN)], ids=(F2, F4)) + index_path = export_for_oracle( + root=self.out, generation_id=summary.generation_id, inputs=self.inputs(F2, F4) + ) + index = json.loads(index_path.read_bytes()) + self.assertEqual([c["scenario_id"] for c in index["candidates"]], [F2]) # valid only + (entry,) = index["candidates"] + directory = index_path.parent / entry["directory"] + names = sorted(p.name for p in directory.rglob("*") if p.is_file()) + self.assertTrue(names) + self.assertTrue(all(n.endswith(".tf") for n in names), names) + self.assertEqual(sorted(entry["files"]), names) + self.assertEqual(entry["generation_id"], summary.generation_id) + self.assertEqual(entry["tree_hash"], json.loads( + (self.attempt_dir(summary) / "attempt.json").read_text("utf-8") + )["candidate_tree_hash"]) + main = (directory / "main.tf").read_text("utf-8") + self.assertNotIn("app_ssh_public", main) + + def test_the_configuration_only_copy_drops_every_other_file(self) -> None: + from experiments.generation.parent import config_only_copy + + source = self.dirs.new() + for name in ("main.tf", "variables.tf", "x.tf.json", "README.md", ".terraform.lock.hcl"): + (source / name).write_text("#\n", "utf-8") + (source / ".terraform").mkdir() + (source / ".terraform" / "a.tf").write_text("#\n", "utf-8") + target = self.dirs.new() / "parent" + config_only_copy(source, target) + self.assertEqual( + sorted(p.name for p in target.rglob("*")), ["main.tf", "variables.tf", "x.tf.json"] + ) + + +class TestNoLaterStep(Case): + def test_generation_replay_and_export_work_with_the_later_steps_gone(self) -> None: + from experiments.generation.export import export_for_oracle + + given = self.inputs(F8) + for step in self.by_id[F8].steps[1:]: + shutil.rmtree(step.directory) + summary = self.generate([ok(VALID)], ids=(F8,), inputs=given) + self.assertEqual(summary.attempts[0]["status"], "PATCH") + self.assertEqual(len(self.replay(summary).results), 4) + export_for_oracle(root=self.out, generation_id=summary.generation_id, inputs=given) + + +class TestOllamaContextWindow(unittest.TestCase): + """Section 3.3: "``num_ctx`` if the model needs it" (P7a verdict limit 5: the model's real + context window).""" + + def test_num_ctx_is_sent_when_pinned_and_recorded(self) -> None: + from llm.adapter.ollama import OllamaProvider + from tests.unit.test_llm_ollama import FakeOllama, always, call, chat_ok, reply + + base = call() + pinned = dataclasses.replace(base, config=dataclasses.replace(base.config, num_ctx=8192)) + with FakeOllama(always(200, chat_ok(reply()))) as server: + OllamaProvider(base_url=server.url).generate(pinned) + body = server.requests[-1][2] + assert body is not None + self.assertEqual(body["options"]["num_ctx"], 8192) + self.assertEqual(pinned.config.to_dict()["num_ctx"], 8192) + self.assertNotIn("num_ctx", base.config.to_dict()) # unchanged records without it + + def test_a_pin_gives_the_budget_its_context_window(self) -> None: + from experiments.generation.models import budget_for + + limits = budget_for(pin(num_ctx=8192, max_output_tokens=1024)) + self.assertEqual(limits.max_context_tokens, 8192) + self.assertEqual(limits.reserved_output_tokens, 1024) + + +class TestProviderFailures(Case): + def test_an_unsupported_provider_answer_is_kept_apart_from_model_failures(self) -> None: + summary = self.generate([ProviderError(FailureClass.UNSUPPORTED, "HTTP 404")]) + (row,) = summary.attempts + self.assertEqual(row["failure_class"], "UNSUPPORTED") + self.assertEqual(row["failure_source"], "provider") + summary2 = self.generate( + [ok(repair('resource "aws_lambda_function" "f" {\n}\n', ["aws_lambda_function.f"]))], + spec=__import__("experiments.generation.run", fromlist=["GenerationSpec"]).GenerationSpec( + name="p10a-test", version="2" + ), + ) + (row2,) = summary2.attempts + self.assertEqual(row2["failure_class"], "UNSUPPORTED") + self.assertEqual(row2["failure_source"], "model") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/unit/test_p10a_lifecycle.py b/tests/unit/test_p10a_lifecycle.py new file mode 100644 index 0000000..964d801 --- /dev/null +++ b/tests/unit/test_p10a_lifecycle.py @@ -0,0 +1,120 @@ +"""The lifecycle rule of the patch validator (P10a work package Section 3.2; decision 6, C-221). + +Section 3.2: "a MODIFY block for an address whose parent block has a ``lifecycle`` block must +contain that lifecycle block, compared as now. Otherwise POLICY_VIOLATION. Validator version -> +``dail-patch-validator-2``. Tests: dropped -> refused; kept unchanged -> passes; parent without +lifecycle -> unchanged behaviour." +Decision 6: "A patch that drops the parent's ``lifecycle`` block is refused: POLICY_VIOLATION before +DAIL, counted as an invalid patch." + +The parent with a lifecycle block is the canonical baseline with one ``lifecycle`` block added to +``aws_security_group.app`` (in a temporary copy; the fixture is never written). +""" + +from __future__ import annotations + +import shutil +import tempfile +import unittest +from pathlib import Path + +from core.terraform_model.normalization import normalize_directory +from llm.schemas.records import FailureClass +from llm.validation.parser import parse_response +from llm.validation.patch import VALIDATOR_VERSION, PatchValidator, ValidationOutcome +from tests.identity_builders import NOW +from tests.impact_builders import registry +from tests.llm_patch_builders import ( + BASELINE, + PARENT_NAME, + PATCH_ID, + REQUEST_ID, + SG_APP, + SG_APP_CLOSED, + WorkRoot, + repair, + scope, +) +from tests.promotion_pipeline import trusted_from + +F = FailureClass +SG_APP_PARENT = ( + 'resource "aws_security_group" "app" {\n' + ' name = "app"\n' + ' description = "Application security group"\n' + " vpc_id = aws_vpc.main.id\n" + "}\n" +) +LIFECYCLE = " lifecycle {\n create_before_destroy = true\n }\n" +SG_APP_WITH_LIFECYCLE = SG_APP_PARENT.replace("}\n", LIFECYCLE + "}\n") +# the repair keeps the parent's lifecycle block unchanged (whitespace differs: compared as now) +CLOSED_KEEPS = SG_APP_CLOSED.replace( + "}\n", " lifecycle {\n create_before_destroy = true\n }\n}\n" +) +CLOSED_CHANGES = SG_APP_CLOSED.replace( + "}\n", " lifecycle {\n create_before_destroy = false\n }\n}\n" +) + + +class Case(unittest.TestCase): + def setUp(self) -> None: + self.work = WorkRoot() + self.addCleanup(self.work.close) + holder = tempfile.TemporaryDirectory(prefix="dail-p10a-lifecycle-") + self.addCleanup(holder.cleanup) + self.parent = Path(holder.name) / "parent" + shutil.copytree(BASELINE, self.parent) + main = self.parent / "main.tf" + text = main.read_bytes().decode("utf-8").replace("\r\n", "\n") + self.assertEqual(text.count(SG_APP_PARENT), 1) + main.write_bytes(text.replace(SG_APP_PARENT, SG_APP_WITH_LIFECYCLE).encode("utf-8")) + + def validator(self, parent: Path) -> PatchValidator: + outcome = normalize_directory(parent, None, now=NOW) + definitions = list(registry("aws_instance.app", "aws_db_instance.database").values()) + trusted = trusted_from(PARENT_NAME, outcome.resources, definitions) + return PatchValidator( + parent_dir=parent, + trusted=trusted, + scope=scope(), + work_root=self.work.path, + variables=None, + now=NOW, + ) + + def run_patch(self, parent: Path, patch: str) -> ValidationOutcome: + parsed = parse_response(repair(patch, [SG_APP])) + assert parsed.response is not None, parsed.errors + return self.validator(parent).validate( + parsed.response, request_id=REQUEST_ID, patch_id=PATCH_ID + ) + + +class TestLifecycleRule(Case): + def test_the_version_is_2(self) -> None: + self.assertEqual(VALIDATOR_VERSION, "dail-patch-validator-2") + + def test_a_modify_that_drops_the_parents_lifecycle_block_is_refused(self) -> None: + outcome = self.run_patch(self.parent, SG_APP_CLOSED) + self.assertIs(outcome.failure_class, F.POLICY_VIOLATION, outcome.errors) + self.assertIsNone(outcome.build) + self.assertIn("lifecycle", " ".join(outcome.errors)) + + def test_a_modify_that_keeps_it_unchanged_passes(self) -> None: + outcome = self.run_patch(self.parent, CLOSED_KEEPS) + self.assertIsNone(outcome.failure_class, outcome.errors) + self.assertIsNotNone(outcome.build) + + def test_a_modify_that_changes_it_is_still_refused(self) -> None: + outcome = self.run_patch(self.parent, CLOSED_CHANGES) + self.assertIs(outcome.failure_class, F.POLICY_VIOLATION, outcome.errors) + + def test_a_parent_without_lifecycle_is_unchanged(self) -> None: + # as before: no lifecycle block passes, an added one is refused + self.assertIsNone(self.run_patch(BASELINE, SG_APP_CLOSED).failure_class) + added = SG_APP_CLOSED.replace("}\n", LIFECYCLE + "}\n") + self.assertIs(self.run_patch(BASELINE, added).failure_class, F.POLICY_VIOLATION) + + +if __name__ == "__main__": + unittest.main() From 4f1156f1bbff45f4aefdf48cdb23e00344428761 Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Fri, 13 Nov 2026 19:53:15 +0530 Subject: [PATCH 3/7] =?UTF-8?q?P10a=20step=203:=20EXTRACTOR=5FVERSION=20ex?= =?UTF-8?q?tractor-2=20apart=20from=20METRICS=5FVERSION=20metrics-draft-1,?= =?UTF-8?q?=20both=20on=20every=20record=20and=20report,=20the=20extractor?= =?UTF-8?q?=20code=20pinned=20by=20hash;=20the=20named=20interleavings=20w?= =?UTF-8?q?ait=20for=20A=5Fdone/B=5Fdone=20(Doc=2013=20=C2=A732;=20P8b=20v?= =?UTF-8?q?erdict=20limits=202,=203)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- experiments/metrics/extractor.py | 16 +++++++++++++--- experiments/report/build.py | 8 +++++++- experiments/report/render.py | 7 ++++++- tests/unit/test_e2e_concurrency.py | 5 +++-- tests/unit/test_p10a_extractor_version.py | 2 +- tests/unit/test_p8b_metrics.py | 8 ++++++-- 6 files changed, 36 insertions(+), 10 deletions(-) diff --git a/experiments/metrics/extractor.py b/experiments/metrics/extractor.py index fb04153..70960b2 100644 --- a/experiments/metrics/extractor.py +++ b/experiments/metrics/extractor.py @@ -1,5 +1,12 @@ -"""The metric extractor, version ``metrics-draft-1`` (P8b work package Section 4; Doc 13 §25-§35, -Appendix B; C-212 to C-214). +"""The metric extractor ``extractor-2``, for the metric definitions ``metrics-draft-1`` (P8b work +package Section 4; Doc 13 §25-§35, Appendix B; C-212 to C-214). + +Versions (P10a Section 3.1): ``METRICS_VERSION`` names the metric definitions (C-212) and +``EXTRACTOR_VERSION`` names this code; every metric record and report carries both. The code of +``experiments/metrics`` and ``experiments/scoring`` is pinned by hash per extractor version in +``tests.unit.test_p10a_extractor_version``: a code change needs a new ``EXTRACTOR_VERSION``. The +first extractor was version ``metrics-draft-1`` (one value for both, P8b); ``extractor-2`` is the same +code with the two versions apart and the generation metrics of ``experiments.metrics.generation``. Input: the transitions of the stored trial records (``experiments.metrics.model``), the scenarios' truth from ``experiments.scoring``, DAIL's S0 results under B, and the identity truth table. Output: @@ -51,7 +58,8 @@ from experiments.scoring.labels import ScenarioLabels from experiments.scoring.truth import TransitionTruth, transition_truth -EXTRACTOR_VERSION: Final = "metrics-draft-1" +METRICS_VERSION: Final = "metrics-draft-1" # the metric definitions (C-212) +EXTRACTOR_VERSION: Final = "extractor-2" # this code (P10a Section 3.1) NAMESPACE: Final = uuid.uuid5(uuid.NAMESPACE_URL, "dail/experiments/metrics") Z95: Final = 1.959963984540054 ORDER: Final = ("A", "B", "C", "C_PRIME") @@ -130,6 +138,7 @@ def add( "value": value, "unit": unit, "wilson95": interval, + "metrics_version": METRICS_VERSION, "extractor_version": EXTRACTOR_VERSION, } ) @@ -670,6 +679,7 @@ def baseline_guard(report: Mapping[str, Any]) -> list[dict[str, Any]]: def to_json(x: Extraction) -> dict[str, Any]: return { + "metrics_version": METRICS_VERSION, "extractor_version": EXTRACTOR_VERSION, "experiment_id": x.experiment_id, "metrics": x.metrics, diff --git a/experiments/report/build.py b/experiments/report/build.py index 0cd6c5c..a661602 100644 --- a/experiments/report/build.py +++ b/experiments/report/build.py @@ -13,7 +13,12 @@ from collections.abc import Mapping from typing import Any, Final -from experiments.metrics.extractor import EXTRACTOR_VERSION, Extraction, to_json +from experiments.metrics.extractor import ( + EXTRACTOR_VERSION, + METRICS_VERSION, + Extraction, + to_json, +) REPORT_KEYS: Final = ( "experiment_id", @@ -73,6 +78,7 @@ def build_report( "experiment_id": definition.get("experiment_id"), "experiment_name": definition.get("name"), "experiment_version": definition.get("version"), + "metrics_version": METRICS_VERSION, "extractor_version": EXTRACTOR_VERSION, "trial_count": len(statuses), "included_count": len(statuses) - len(failures), diff --git a/experiments/report/render.py b/experiments/report/render.py index f32a76e..7b95970 100644 --- a/experiments/report/render.py +++ b/experiments/report/render.py @@ -122,7 +122,12 @@ def render_markdown(report: Mapping[str, Any]) -> str: "", f"- Experiment: `{report['experiment_name']}` version {report['experiment_version']}, " f"id `{report['experiment_id']}`", - f"- Extractor: `{report['extractor_version']}`", + f"- Extractor: `{report['extractor_version']}`" + + ( + f"; metric definitions: `{report['metrics_version']}`" + if "metrics_version" in report # P10a; a P8b report has one version + else "" + ), f"- Trials: {report['trial_count']} (included {report['included_count']}, " f"excluded {report['excluded_count']})", f"- Common prefix of all conditions: {prefix['common_prefix_all']} transitions; " diff --git a/tests/unit/test_e2e_concurrency.py b/tests/unit/test_e2e_concurrency.py index ea423e0..5e64ca9 100644 --- a/tests/unit/test_e2e_concurrency.py +++ b/tests/unit/test_e2e_concurrency.py @@ -138,7 +138,8 @@ def assert_one_winner(self, a: EvaluationResult, b: EvaluationResult) -> None: class TestNamedInterleavings(RaceCase): def test_one_runs_to_the_end_before_the_other_starts(self) -> None: - a, b = self.race(["A"] * STAGE_POINTS + ["B"] * STAGE_POINTS) + # "A_done": B may not pass its first checkpoint while A is still committing (P10a 3.1) + a, b = self.race(["A"] * STAGE_POINTS + ["A_done"] + ["B"] * STAGE_POINTS) self.assert_one_winner(a, b) self.assertIs(a.kind, ResultKind.PROMOTED) # the late one never asked for a commit: it was rejected before any request @@ -150,7 +151,7 @@ def test_one_runs_to_the_end_before_the_other_starts(self) -> None: self.assertNotIn("STATE_PROMOTED", types) def test_the_other_way_round(self) -> None: - a, b = self.race(["B"] * STAGE_POINTS + ["A"] * STAGE_POINTS) + a, b = self.race(["B"] * STAGE_POINTS + ["B_done"] + ["A"] * STAGE_POINTS) self.assert_one_winner(a, b) self.assertIs(b.kind, ResultKind.PROMOTED) diff --git a/tests/unit/test_p10a_extractor_version.py b/tests/unit/test_p10a_extractor_version.py index 5177e1e..6d12f72 100644 --- a/tests/unit/test_p10a_extractor_version.py +++ b/tests/unit/test_p10a_extractor_version.py @@ -24,7 +24,7 @@ from tests.metrics_builders import GATES, scenarios PINNED: dict[str, str] = { - "extractor-2": "PIN-ME", + "extractor-2": "7a0fec10865e69672727ac801751520d835097ae078388c553f325457b9a1fc9", } EXTRACTOR_PACKAGES = ("experiments/metrics", "experiments/scoring") CRLF, LF = b"\r\n", b"\n" diff --git a/tests/unit/test_p8b_metrics.py b/tests/unit/test_p8b_metrics.py index 5ffb097..3239684 100644 --- a/tests/unit/test_p8b_metrics.py +++ b/tests/unit/test_p8b_metrics.py @@ -232,11 +232,15 @@ def test_metric_records_have_the_fields_of_appendix_b(self) -> None: "denominator", "value", "unit", + "metrics_version", "extractor_version", ): self.assertIn(key, record) - self.assertEqual(record["extractor_version"], "metrics-draft-1") - self.assertEqual(EXTRACTOR_VERSION, "metrics-draft-1") + # P10a Section 3.1: the definitions stay metrics-draft-1; the extractor has its own + # version (it was the same value in P8b) + self.assertEqual(record["metrics_version"], "metrics-draft-1") + self.assertEqual(record["extractor_version"], EXTRACTOR_VERSION) + self.assertEqual(EXTRACTOR_VERSION, "extractor-2") def test_the_extractor_is_deterministic(self) -> None: from experiments.metrics.extractor import dumps_metrics From c8e241f29c5523b9c8077707907b0495163eb2d6 Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Fri, 13 Nov 2026 20:16:42 +0530 Subject: [PATCH 4/7] =?UTF-8?q?P10a=20step=204:=20validator=202=20refuses?= =?UTF-8?q?=20a=20MODIFY=20that=20drops=20the=20parent's=20lifecycle=20blo?= =?UTF-8?q?ck;=20experiments/generation=20(S0-only=20input,=20the=20fixed?= =?UTF-8?q?=20task,=20the=20decision-5=20scope,=20pinned=20models=20with?= =?UTF-8?q?=20the=20digest=20check=20and=20num=5Fctx,=20generate=20and=20f?= =?UTF-8?q?reeze,=20replay=20with=20the=20rebuild=20hash=20check,=20config?= =?UTF-8?q?uration-only=20oracle=20export);=20generation=20metrics=20and?= =?UTF-8?q?=20the=20pilot=20report=20(Doc=2010=20=C2=A713,=20=C2=A716,=20?= =?UTF-8?q?=C2=A724;=20Doc=2013=20=C2=A726,=20=C2=A747;=20C-219=20to=20C-2?= =?UTF-8?q?21)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- experiments/README.md | 66 +++ experiments/config/llm.toml | 7 + experiments/config/models.toml | 20 + experiments/generation/__init__.py | 8 + experiments/generation/__main__.py | 184 +++++++++ experiments/generation/export.py | 136 +++++++ experiments/generation/models.py | 124 ++++++ experiments/generation/parent.py | 114 ++++++ experiments/generation/replay.py | 275 +++++++++++++ experiments/generation/run.py | 451 +++++++++++++++++++++ experiments/generation/task.py | 124 ++++++ experiments/metrics/extractor.py | 5 +- experiments/metrics/generation.py | 140 +++++++ experiments/report/__main__.py | 41 +- experiments/report/pilot.py | 312 ++++++++++++++ experiments/task_text.py | 31 ++ llm/adapter/ollama.py | 5 +- llm/schemas/records.py | 11 +- llm/validation/patch.py | 13 +- tests/contract/test_experiment_blinding.py | 9 +- tests/unit/test_p10a_extractor_version.py | 10 +- tests/unit/test_p10a_generation.py | 56 ++- tests/unit/test_p10a_generation_metrics.py | 145 +++++++ 23 files changed, 2244 insertions(+), 43 deletions(-) create mode 100644 experiments/config/llm.toml create mode 100644 experiments/config/models.toml create mode 100644 experiments/generation/__init__.py create mode 100644 experiments/generation/__main__.py create mode 100644 experiments/generation/export.py create mode 100644 experiments/generation/models.py create mode 100644 experiments/generation/parent.py create mode 100644 experiments/generation/replay.py create mode 100644 experiments/generation/run.py create mode 100644 experiments/generation/task.py create mode 100644 experiments/metrics/generation.py create mode 100644 experiments/report/pilot.py create mode 100644 experiments/task_text.py create mode 100644 tests/unit/test_p10a_generation_metrics.py diff --git a/experiments/README.md b/experiments/README.md index de6923c..210ea4c 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -78,3 +78,69 @@ $DAIL_EXPERIMENT_ROOT/experiment// for scoring, which is P8b. - **Test split:** the loader refuses `split = "test"`. The test split, `$TERRAPRESERVE_PRIVATE` and the test ids are not read before P10, after the protocol is frozen. + +## LLM candidates (P10a) + +`experiments/generation/` generates repair candidates with free open-weight models on the local +Ollama server, freezes every attempt, replays the valid ones through DAIL under A, B, C and C′, and +exports them for TerraPreserve's oracle. Everything stays on this machine (C-71, C-218). No label, +`description`, `intent`, `mutation_record` or step after S0 is read +(`tests/contract/test_experiment_blinding.py`, checks (c) and (d)). + +```bash +python3.12 -m experiments.generation generate --name p10a-pilot --models KEY,... --per-family 4 --k 1 +python3.12 -m experiments.generation replay --name p10a-pilot --replay-name replay-1 +python3.12 -m experiments.generation export --name p10a-pilot +python3.12 -m experiments.report pilot --name p10a-pilot --replay-name replay-1 \ + --licence-checks checks.json --doc docs/P10A_MODEL_PILOT +``` + +- **Scenarios and task (C-219):** repair scenarios only (F2, F4, F8). The model is given S0, the + fixed task "Repair ``: ``. Keep every other protected invariant + satisfied. Make the smallest change." (`experiments/task_text.py`), and P7's context + (`build_context`, focus = the target invariant's scope resources). +- **Scope (C-220):** every resource address of S0, every supported type, no attribute restriction, + all four operations; recorded with every candidate and every replay. +- **Parent:** S0 through the baseline flow with the registry-v1.1 definitions, from a + configuration-only copy (`*.tf`, `*.tf.json`) of the step directory. +- **Generation:** `llm/adapter/generate.py` only (prompt `dail-repair-prompt-2`, parser 2, validator + `dail-patch-validator-2`, which refuses a MODIFY that drops the parent's `lifecycle` block, C-221). + Retry policy `p10a-retry-2`: at most two calls, again only after TRANSIENT_ERROR or + INVALID_OUTPUT. The seed of repetition k is the pinned seed + k − 1. +- **Switch:** `llm.generation_enabled` is on only in the harness's copy of the configuration, by the + reviewed `experiments/config/llm.toml` (as C-207 (3) does for promotion); `config/` keeps it off. +- **Models:** pinned in the reviewed `experiments/config/models.toml` by tag, digest, temperature, + top_p, seed, max_output_tokens, timeout and num_ctx (the budget's context window). A model whose + installed digest is not the pinned one is refused before anything is written. + +```text +$DAIL_EXPERIMENT_ROOT/candidates// + definition.json what was asked (write once) + /model.json the pin and the installed facts + /// + records/request-.json P7's LLMRequest of each call + records/response-.json P7's LLMResponse of each call (raw reply redacted) + patch.tf a valid candidate's patch only + timing.json wall-clock time + attempt.json status, failure class and source, parent, scope, hashes + candidates_manifest.json every attempt with the SHA-256 of each file + replay// one store per candidate and condition; replay.json + oracle_export/ the built .tf files per valid candidate; oracle_export.json + pilot_report// the pilot report (python -m experiments.report pilot) +``` + +- **Freezing:** every file is written once. A generation whose manifest exists is frozen and never + regenerated or edited; an attempt directory without `attempt.json` (an interrupted call) is never + completed or overwritten. A REGENERATE (Doc 13 §47) is a new name or version, so a new + `generation_id`. +- **Replay:** each valid candidate is evaluated as a one-step candidate from S0 under every condition + through `LLMEvaluationRequest`. In the same call the validator rebuilds the directory from the + frozen reply, the rebuilt tree hash is compared with the frozen one, and the build is deleted. A + mismatch gets no decision. Invalid attempts are listed as generation failures, never as + decisions. +- **Export:** for every valid candidate, the built `.tf` files and nothing else, with ids and hashes + (tree hash and the D4-09 content hash) in `oracle_export.json`. TerraPreserve labels them in its + own package; DAIL never does. +- **Failure source:** a failed attempt's class is the provider's when there was no reply (a + connection failure, a timeout, Ollama's 400/404 as UNSUPPORTED), the budget's for + CONTEXT_INSUFFICIENT, and the model's otherwise (P7a verdict limit 4). diff --git a/experiments/config/llm.toml b/experiments/config/llm.toml new file mode 100644 index 0000000..82a38a0 --- /dev/null +++ b/experiments/config/llm.toml @@ -0,0 +1,7 @@ +# The harness's reviewed LLM file (P10a Section 3.3; the same way as C-207 (3) for promotion). +# experiments.generation.run.generation_config() copies config/ and puts this file in place of +# config/base/llm.toml for candidate generation only; config/ keeps generation off everywhere. +# Every generation definition pins this file's SHA-256 (experiment_llm_config_sha256). +generation_enabled = true +# the local Ollama server only (C-71, C-182, C-218) +provider = "ollama" diff --git a/experiments/config/models.toml b/experiments/config/models.toml new file mode 100644 index 0000000..858d301 --- /dev/null +++ b/experiments/config/models.toml @@ -0,0 +1,20 @@ +# Reviewed model pins for LLM candidate generation (P10a Section 3.3 "Model configuration"; C-182, +# C-218). Free open-weight models only, served by the local Ollama server only (C-71). A run refuses +# a model whose installed digest is not the one pinned here. Licence: as checked on the day on the +# model's Ollama library page and with `ollama show --license` (licence_source says where). +# Every experiment pin sets num_ctx: the budget uses it as the context window (P7a verdict limit 5). + +# The P7 wiring model: wiring only, never results (C-186). +[model.qwen2_5-coder-1_5b] +tag = "qwen2.5-coder:1.5b" +digest = "d7372fd828518a4d38b1eb196c673c31a85f2ed302b3d1e406c4c2d1b64a0668" +quantization = "Q4_K_M" +size = "986 MB (1.5B parameters)" +licence = "Apache-2.0" +licence_source = "ollama.com/library/qwen2.5-coder:1.5b, licence blob 832dd9e00a68, and ollama show --license (P7a, 11 Nov 2026)" +temperature = 0.0 +top_p = 1.0 +seed = 20261113 +max_output_tokens = 2048 +timeout_seconds = 900.0 +num_ctx = 16384 diff --git a/experiments/generation/__init__.py b/experiments/generation/__init__.py new file mode 100644 index 0000000..9be20ed --- /dev/null +++ b/experiments/generation/__init__.py @@ -0,0 +1,8 @@ +"""LLM candidates for the experiment (P10a; C-218 to C-221). + +``task`` (what generation is given: S0 only, the fixed task, the scope of decision 5), ``models`` +(pins, digest check, budget), ``parent`` (S0 through the baseline flow, configuration only), +``run`` (generate and freeze), ``replay`` (rebuild, check, evaluate under A, B, C and C'), +``export`` (configuration-only candidates for the oracle). No module here reads a label, a +description, an intent or a later step (``tests/contract/test_experiment_blinding.py``). +""" diff --git a/experiments/generation/__main__.py b/experiments/generation/__main__.py new file mode 100644 index 0000000..66b496a --- /dev/null +++ b/experiments/generation/__main__.py @@ -0,0 +1,184 @@ +"""The candidate generation command line (P10a; C-218 to C-221). + + python -m experiments.generation generate --name NAME [--version V] --models KEY,... + [--k K] (--only ID,... | --per-family N [--families F2,F4,F8]) + python -m experiments.generation replay --name NAME [--version V] --replay-name R + [--conditions A,B,C,C_PRIME] + python -m experiments.generation export --name NAME [--version V] + +``TERRAPRESERVE_ROOT`` and ``DAIL_EXPERIMENT_ROOT`` as for ``python -m experiments`` (C-204): the +checkout at ``terrapreserve-v1.0``, the dev split only (the loader refuses the test split), the +output outside both repositories. Models are the pins of the reviewed +``experiments/config/models.toml``; each is served by the local Ollama server only (C-71, C-218) +and refused unless its installed digest is the pinned one. ``--per-family N`` takes the N lowest +scenario ids of each repair family (F2, F4, F8). Generation is switched on by the reviewed +``experiments/config/llm.toml`` only (``generation_config``). +""" + +from __future__ import annotations + +import argparse +import json +import sys +from collections.abc import Sequence +from typing import Any + +from experiments.__main__ import CommandError, _environment, _roots +from experiments.definition import Condition +from experiments.generation.export import export_for_oracle +from experiments.generation.models import load_pins +from experiments.generation.replay import replay_candidates +from experiments.generation.run import ( + GenerationSpec, + generate_candidates, + generation_config, +) +from experiments.generation.task import REPAIR, GenerationInput, generation_input +from experiments.loader import ScenarioInput, load +from experiments.runner import experiment_config +from llm.adapter.ollama import OllamaProvider + +REPAIR_FAMILIES = ("F2", "F4", "F8") + + +def choose( + scenarios: Sequence[ScenarioInput], + *, + only: Sequence[str] | None, + per_family: int | None, + families: Sequence[str], +) -> list[GenerationInput]: + repairs = [s for s in scenarios if s.target_kind == REPAIR] + if only: + by_id = {s.scenario_id: s for s in repairs} + missing = [sid for sid in only if sid not in by_id] + if missing: + raise CommandError(f"not repair scenarios of the dev split: {missing}") + return [generation_input(by_id[sid]) for sid in sorted(only)] + if per_family is None: + raise CommandError("give --only or --per-family") + chosen: list[GenerationInput] = [] + for family in families: + mine = sorted((s for s in repairs if s.family == family), key=lambda s: s.scenario_id) + chosen += [generation_input(s) for s in mine[:per_family]] + return chosen + + +def _spec(args: argparse.Namespace) -> GenerationSpec: + return GenerationSpec(name=args.name, version=args.version) + + +def cmd_generate(args: argparse.Namespace) -> int: + terrapreserve, output = _roots() + environment = _environment(terrapreserve) + loaded = load(terrapreserve) + inputs = choose( + loaded.scenarios, + only=args.only.split(",") if args.only else None, + per_family=args.per_family, + families=args.families.split(","), + ) + pins = load_pins() + models = [] + for key in args.models.split(","): + if key not in pins: + raise CommandError(f"no pin {key!r} in experiments/config/models.toml") + pin = pins[key] + provider = OllamaProvider(expected_digest=pin.digest) + installed = provider.model_facts(pin.tag) + models.append((pin, provider, installed)) + environment = {**environment, "ollama_version": models[0][1].server_version()} + switch = generation_config().get("llm.generation_enabled") + summary = generate_candidates( + spec=_spec(args), + inputs=inputs, + models=models, + k=args.k, + root=output, + generation_enabled=switch is True, + environment=environment, + ) + counts: dict[str, dict[str, int]] = {} + for row in summary.attempts: + mine = counts.setdefault(row["model_key"], {}) + key = row["status"] if row["failure_class"] is None else row["failure_class"] + mine[key] = mine.get(key, 0) + 1 + print( + json.dumps( + { + "generation_id": summary.generation_id, + "candidates_manifest": str(summary.manifest_path), + "candidates_manifest_sha256": summary.manifest_sha256, + "attempts": counts, + }, + indent=1, + ) + ) + return 0 + + +def cmd_replay(args: argparse.Namespace) -> int: + terrapreserve, output = _roots() + loaded = load(terrapreserve) + summary = replay_candidates( + root=output, + generation_id=_spec(args).generation_id, + scenarios={s.scenario_id: s for s in loaded.scenarios}, + conditions=[Condition(c).value for c in args.conditions.split(",")], + config=experiment_config(), + replay_name=args.replay_name, + ) + decisions: dict[str, dict[str, int]] = {} + for result in summary.results: + mine = decisions.setdefault(result["condition"], {}) + key = result["decision"] or "NO DECISION" + mine[key] = mine.get(key, 0) + 1 + print( + json.dumps( + { + "replay": str(summary.path), + "decisions": decisions, + "generation_failures": len(summary.generation_failures), + }, + indent=1, + ) + ) + return 0 + + +def cmd_export(args: argparse.Namespace) -> int: + terrapreserve, output = _roots() + loaded = load(terrapreserve) + inputs = [generation_input(s) for s in loaded.scenarios if s.target_kind == REPAIR] + path = export_for_oracle(root=output, generation_id=_spec(args).generation_id, inputs=inputs) + print(str(path)) + return 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="python -m experiments.generation") + sub = parser.add_subparsers(dest="command", required=True) + for name in ("generate", "replay", "export"): + command = sub.add_parser(name) + command.add_argument("--name", required=True) + command.add_argument("--version", default="1") + if name == "generate": + command.add_argument("--models", required=True) + command.add_argument("--k", type=int, default=1) + command.add_argument("--only") + command.add_argument("--per-family", type=int) + command.add_argument("--families", default=",".join(REPAIR_FAMILIES)) + if name == "replay": + command.add_argument("--replay-name", required=True) + command.add_argument("--conditions", default="A,B,C,C_PRIME") + args = parser.parse_args(argv) + handler: Any = {"generate": cmd_generate, "replay": cmd_replay, "export": cmd_export} + try: + return int(handler[args.command](args)) + except CommandError as error: + print(f"error: {error}", file=sys.stderr) + return 2 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/generation/export.py b/experiments/generation/export.py new file mode 100644 index 0000000..36add0e --- /dev/null +++ b/experiments/generation/export.py @@ -0,0 +1,136 @@ +"""Export valid LLM candidates for the oracle (P10a Section 3.3 "Export for the oracle"; P7b verdict +limit 6). + +"For every valid candidate, a configuration-only directory (the built ``.tf`` files and nothing +else) plus an ``oracle_export.json`` with ids and hashes. TerraPreserve labels these in its own +package; DAIL never labels them." + +Each valid frozen candidate is rebuilt from the frozen reply by the patch validator on the +configuration-only copy of S0 (the parent is built again with the same name-based ids, so it is the +parent the candidate was generated against: its state id is checked), the rebuilt tree hash is +checked against the frozen one, the ``.tf`` files of the build are copied, and the build is deleted. +A candidate whose rebuild differs, or whose build holds a file that is not ``.tf``, is listed under +``refused`` and not exported. ``content_hash`` of an exported directory is computed as +TerraPreserve's step hash (D4-09, ``experiments.loader.step_content_hash``). + +Output, written once: ``/candidates//oracle_export/oracle_export.json`` and one +directory per candidate, ``///``, beside it. +""" + +from __future__ import annotations + +import shutil +import tempfile +from collections.abc import Sequence +from pathlib import Path +from typing import Any + +from experiments.artifacts import read_record, write_record +from experiments.generation.parent import s0_parent +from experiments.generation.run import ( + MANIFEST, + attempt_dir, + frozen_response_text, + generation_dir, + read_manifest, + sha256_file, +) +from experiments.generation.task import GenerationError, GenerationInput, change_scope +from experiments.loader import step_content_hash +from experiments.trial import DAIL_CLOCK_BASE +from llm.validation.parser import parse_response +from llm.validation.patch import PatchValidator + +EXPORT_INDEX = "oracle_export.json" + + +def _rebuild( + given: GenerationInput, frozen: dict[str, Any], reply: str, request_id: str, work: Path +) -> tuple[Path | None, str | None, str | None]: + """(the built directory, its tree hash, why it was refused).""" + parent = s0_parent(given, work / "parent", "generation") + if parent.state.state_id != frozen["parent"]["state_id"]: + return None, None, "the rebuilt S0 is not the frozen parent" + parsed = parse_response(reply) + if parsed.response is None or parsed.failure_class is not None: + return None, None, "the frozen reply does not parse" + outcome = PatchValidator( + parent_dir=parent.directory, + trusted=parent.state, + scope=change_scope(parent.state.resources), + work_root=work / "builds", + variables=dict(given.variable_values), + now=DAIL_CLOCK_BASE, + ).validate(parsed.response, request_id=request_id, patch_id=frozen["patch_id"]) + if outcome.build is None: + return None, None, f"the validator refused the rebuild: {list(outcome.errors)}" + return outcome.build.candidate_dir, outcome.build.tree_hash, None + + +def export_for_oracle(*, root: Path, generation_id: str, inputs: Sequence[GenerationInput]) -> Path: + manifest = read_manifest(root, generation_id) + base = generation_dir(root, generation_id) + out = base / "oracle_export" + if (out / EXPORT_INDEX).exists(): + raise GenerationError(f"{out / EXPORT_INDEX} exists; an export is written once") + by_id = {g.scenario_id: g for g in inputs} + exported: list[dict[str, Any]] = [] + refused: list[dict[str, Any]] = [] + for row in manifest["attempts"]: + if row["status"] != "PATCH": + continue + given = by_id.get(row["scenario_id"]) + if given is None: + raise GenerationError(f"{row['scenario_id']}: no generation input given") + k = int(row["k"]) + folder = attempt_dir(base, row["model_key"], row["scenario_id"], k) + frozen = read_record(folder / "attempt.json") + reply, request_id = frozen_response_text(folder, frozen) + ident = {"model_key": row["model_key"], "scenario_id": row["scenario_id"], "k": k} + with tempfile.TemporaryDirectory(prefix="dail-p10a-export-") as tmp: + built, digest, why = _rebuild(given, frozen, reply, request_id, Path(tmp)) + if built is not None and digest != frozen["candidate_tree_hash"]: + why = "the rebuilt tree hash is not the frozen one" + files = [] if built is None else sorted(p for p in built.rglob("*") if p.is_file()) + others = [p.name for p in files if not p.name.endswith(".tf")] + if built is not None and others: + why = f"the build holds files that are not .tf: {others}" + if why is not None: + refused.append({**ident, "reason": why}) + continue + assert built is not None + target = out / row["model_key"] / row["scenario_id"] / str(k) + target.mkdir(parents=True) + for path in files: + if path.parent != built: + raise GenerationError(f"{path}: a built file outside the top level") + shutil.copyfile(path, target / path.name) + shutil.rmtree(built.parent, ignore_errors=True) + exported.append( + { + "candidate_id": f"{generation_id}/{row['model_key']}/{row['scenario_id']}/{k}", + "generation_id": generation_id, + **ident, + "family": row["family"], + "model_tag": frozen["model_tag"], + "model_digest": frozen["model_digest"], + "s0_content_hash": frozen["parent"]["s0_content_hash"], + "parent_state_id": frozen["parent"]["state_id"], + "patch_sha256": frozen["patch_sha256"], + "patch_id": frozen["patch_id"], + "tree_hash": digest, + "content_hash": step_content_hash(target), + "directory": target.relative_to(out).as_posix(), + "files": {p.name: sha256_file(p) for p in sorted(target.iterdir())}, + } + ) + return write_record( + out / EXPORT_INDEX, + { + "record": "oracle_export", + "generation_id": generation_id, + "candidates_manifest_sha256": sha256_file(base / MANIFEST), + "candidates": exported, + "refused": refused, + }, + ) diff --git a/experiments/generation/models.py b/experiments/generation/models.py new file mode 100644 index 0000000..172b986 --- /dev/null +++ b/experiments/generation/models.py @@ -0,0 +1,124 @@ +"""Model pins for candidate generation (P10a Section 3.3 "Model configuration"; C-182, C-218). + +"Model configuration: pinned per model by tag and digest, ``temperature``, ``seed``, ``top_p``, +``max_output_tokens``, ``timeout``, and ``num_ctx`` if the model needs it. The run refuses a model +whose installed digest is not the pinned one." + +A ``ModelPin`` also records what the pilot reports of the model (quantization, size, licence and +where the licence was checked). Pins live in the reviewed file ``experiments/config/models.toml``; +``load_pins`` reads it (standard library ``tomllib``). ``check_installed`` compares the local +server's facts with the pin before any generation; ``OllamaProvider(expected_digest=...)`` checks +the digest again before its first call (C-182). + +The generation settings that are not per model are fixed here, for every model: the retry policy +(``RETRY_POLICY``: two attempts, another one only after TRANSIENT_ERROR or INVALID_OUTPUT; with +temperature 0 and a fixed seed a NO_SOLUTION would come back unchanged) and the budget (the model's +context window ``num_ctx`` with ``max_output_tokens`` reserved for the reply and a safety margin, so +an oversized prompt is CONTEXT_INSUFFICIENT before any call instead of being cut by the server). +The seed of repetition ``k`` is ``seed + k - 1`` (``seed_for``), so repetitions differ only by seed. +""" + +from __future__ import annotations + +import tomllib +from collections.abc import Mapping +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any, Final + +from llm.prompts.templates import PROMPT_VERSION +from llm.schemas.records import BudgetConfig, FailureClass, LLMConfig, RetryPolicy +from llm.schemas.repair_response import RESPONSE_SCHEMA_VERSION + +REPO_ROOT: Final = Path(__file__).resolve().parents[2] +MODELS_FILE: Final = REPO_ROOT / "experiments" / "config" / "models.toml" +RETRY_POLICY: Final = RetryPolicy( + policy_id="p10a-retry-2", + max_attempts=2, + retry_on=(FailureClass.TRANSIENT_ERROR, FailureClass.INVALID_OUTPUT), + backoff_seconds=0.0, + prompt_adjustment_policy="none", +) +SAFETY_MARGIN: Final = 256 + + +class ModelRefused(Exception): + """The installed model is not the pinned one (or is not installed): nothing was generated.""" + + +@dataclass(frozen=True) +class ModelPin: + key: str # a file-system-safe name for the records + tag: str + digest: str + quantization: str + size: str + licence: str + licence_source: str + temperature: float + top_p: float + seed: int + max_output_tokens: int + timeout_seconds: float + num_ctx: int | None = None + + def __post_init__(self) -> None: + if not self.key or any(c in self.key for c in '\\/:*?"<>| '): + raise ValueError(f"model key {self.key!r} must be a plain file name") + if len(self.digest) != 64 or any(c not in "0123456789abcdef" for c in self.digest): + raise ValueError(f"{self.key}: the digest must be 64 lowercase hex characters") + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +def seed_for(pin: ModelPin, k: int) -> int: + return pin.seed + k - 1 + + +def llm_config(pin: ModelPin, provider_name: str, k: int = 1) -> LLMConfig: + return LLMConfig( + provider=provider_name, + model_name=pin.tag, + model_version=pin.digest, + temperature=pin.temperature, + top_p=pin.top_p, + seed=seed_for(pin, k), + max_output_tokens=pin.max_output_tokens, + timeout_seconds=pin.timeout_seconds, + retry_policy_id=RETRY_POLICY.policy_id, + prompt_version=PROMPT_VERSION, + response_schema_version=RESPONSE_SCHEMA_VERSION, + num_ctx=pin.num_ctx, + ) + + +def budget_for(pin: ModelPin) -> BudgetConfig: + if pin.num_ctx is None: + # the budget must know the window the server will use; DAIL does not assume a default + raise ValueError(f"{pin.key}: an experiment pin sets num_ctx") + return BudgetConfig( + max_context_tokens=pin.num_ctx, + reserved_output_tokens=pin.max_output_tokens, + safety_margin=SAFETY_MARGIN, + ) + + +def check_installed(pin: ModelPin, facts: Mapping[str, Any]) -> None: + """Refuse unless the local server's model is the pinned tag and digest.""" + if facts.get("name") != pin.tag: + raise ModelRefused(f"{pin.tag} is not installed (the server named {facts.get('name')!r})") + if facts.get("digest") != pin.digest: + raise ModelRefused( + f"{pin.tag}: the installed digest {facts.get('digest')} is not the pinned " + f"{pin.digest} (C-182)" + ) + + +def load_pins(path: Path = MODELS_FILE) -> dict[str, ModelPin]: + """The pins of the reviewed file, by key.""" + data = tomllib.loads(Path(path).read_text(encoding="utf-8")) + pins: dict[str, ModelPin] = {} + for key, values in (data.get("model") or {}).items(): + pins[key] = ModelPin(key=key, **values) + return pins diff --git a/experiments/generation/parent.py b/experiments/generation/parent.py new file mode 100644 index 0000000..dfd62a3 --- /dev/null +++ b/experiments/generation/parent.py @@ -0,0 +1,114 @@ +"""The parent of every LLM candidate: S0 (P10a Section 3.3 "Parent"; P7b verdict limit 6). + +"Parent: the S0 trusted state built by the existing baseline flow with the registry v1.1 definitions +(the same S0 every condition starts from)." ``s0_parent`` copies S0's configuration files into a +directory of its own (``config_only_copy``: ``*.tf`` and ``*.tf.json`` at the top level, nothing +else, so a candidate built from it holds configuration only, P7b verdict limit 6), and establishes +the baseline there through ``core.application.baseline.establish_baseline`` in a fresh store, with +the four registry-v1.1 definitions (earlier versions registered first, Doc 06 §4.2) and the +INV-FUNC-001 binding of the scenario's endpoints, as condition C's lineage does (C-198, C-203). + +Ids are name-based and the clock is the harness's logical clock (C-207), so the same S0 gives the +same parent state id and hash every time it is built (for generation and again for the export). +""" + +from __future__ import annotations + +import shutil +from dataclasses import dataclass +from pathlib import Path +from typing import Final + +from core.application.baseline import BaselineRequest, establish_baseline +from core.dependency.invariants import SemanticBinding +from core.domain.invariant import Invariant +from core.domain.registry_v1 import registry_v1_invariants +from core.domain.registry_v1_1 import REGISTRY_VERSION_V1_1, registry_v1_1_invariants +from core.domain.repositories import UnitOfWork +from core.domain.state import TrustedState +from core.persistence.schema import initialize_database +from core.persistence.sqlite import SqliteUnitOfWork +from core.promotion.commit import PromotionContext +from experiments.conditions import FUNC_BINDING +from experiments.definition import name_id +from experiments.generation.task import GenerationInput +from experiments.trial import DAIL_CLOCK_BASE, LogicalClock + +CONFIG_SUFFIXES: Final = (".tf", ".tf.json") + + +def config_only_copy(source: Path, target: Path) -> Path: + """Copy the configuration files at the top level of ``source`` into a new ``target``.""" + target.mkdir(parents=True) + for path in sorted(Path(source).iterdir()): + if path.is_file() and not path.is_symlink() and path.name.endswith(CONFIG_SUFFIXES): + shutil.copyfile(path, target / path.name) + return target + + +@dataclass(frozen=True) +class Parent: + state: TrustedState + directory: Path # the configuration-only copy of S0 + definitions: tuple[Invariant, ...] + bindings: tuple[SemanticBinding, ...] + + +def definitions_for(given: GenerationInput) -> tuple[Invariant, ...]: + return registry_v1_1_invariants( + DAIL_CLOCK_BASE, source=given.source_endpoint, destination=given.destination_endpoint + ) + + +def s0_parent(given: GenerationInput, work: Path, purpose: str) -> Parent: + """Build S0 under ``work`` (which must not exist); ``purpose`` names the ids.""" + work = Path(work) + directory = config_only_copy(given.s0_dir, work / "s0") + definitions = definitions_for(given) + func, protocol, port = FUNC_BINDING + bindings = ( + SemanticBinding(func, given.source_endpoint, given.destination_endpoint, protocol, port), + ) + held = {d.invariant_id: d.version for d in definitions} + history = tuple( + d + for d in registry_v1_invariants( + DAIL_CLOCK_BASE, source=given.source_endpoint, destination=given.destination_endpoint + ) + if d.version < held[d.invariant_id] + ) + store = work / "store" / "dail.sqlite" + store.parent.mkdir(parents=True) + initialize_database(store) + + def factory() -> UnitOfWork: + return SqliteUnitOfWork(store) + + with factory() as uow: # earlier versions first: no gaps (Doc 06 §4.2) + for definition in history: + uow.invariants.register_definition(definition) + clock = LogicalClock() + + def nid(*parts: object) -> str: + return name_id("p10a-parent", purpose, given.scenario_id, given.s0_content_hash, *parts) + + request = BaselineRequest( + baseline_id=nid("baseline"), + lineage_id=nid("lineage"), + baseline_dir=directory, + variables=dict(given.variable_values), + definitions=definitions, + bindings=bindings, + context=PromotionContext( + run_id=nid("run"), + correlation_id=nid("correlation"), + operation_id=nid("operation"), + attempt_id=nid("attempt"), + actor_id=None, + ), + now=clock(), + eligible=True, + registry_version=REGISTRY_VERSION_V1_1, + ) + made = establish_baseline(factory, request) + return Parent(made.state, directory, definitions, bindings) diff --git a/experiments/generation/replay.py b/experiments/generation/replay.py new file mode 100644 index 0000000..abf6c0d --- /dev/null +++ b/experiments/generation/replay.py @@ -0,0 +1,275 @@ +"""Replay frozen LLM candidates under A, B, C and C' (P10a Section 3.3 "Replay"; P7b verdict limit +5). + +"A frozen valid patch is evaluated as a one-step candidate from S0 under A, B, C and C' through +``LLMEvaluationRequest``. The directory is rebuilt by the validator from the frozen patch in the +same call, then deleted (P7b limit 5). Each run checks that the rebuilt tree hash equals the frozen +one. Invalid attempts are recorded as generation failures, never as decisions." + +For one candidate and one condition: a fresh store and lineage, S0 through the harness's baseline +flow for that condition (``experiments.trial``, C-198, C-202, C-203) from the scenario's first step +only; the frozen reply is parsed again (P7's parser) and the patch validator rebuilds the candidate +from the configuration-only copy of S0 with the decision-5 scope of that condition's S0; the rebuilt +tree hash is compared with the frozen one. Only when they are equal is the candidate evaluated, by +the condition's ``EvaluationOrchestrator`` with an ``LLMEvaluationRequest`` (no open target: one +step). The build is deleted in the same call, whatever happened. A mismatch, or a rebuild the +validator refuses, is recorded with no decision. + +Output, written once under ``/candidates//replay//``: +``////`` with the attempt's store and records, and +``replay.json`` listing every result and every generation failure. +""" + +from __future__ import annotations + +import dataclasses +import shutil +import tempfile +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from core.application.orchestrator import EvaluationOrchestrator, LLMEvaluationRequest +from core.domain.repositories import UnitOfWork +from core.persistence.schema import initialize_database +from core.persistence.sqlite import SqliteUnitOfWork +from experiments import records +from experiments.artifacts import ATTEMPT_DIRS, read_record, write_record +from experiments.conditions import setup_for +from experiments.definition import Condition, name_id +from experiments.generation.parent import config_only_copy +from experiments.generation.run import ( + attempt_dir, + frozen_response_text, + generation_dir, + read_manifest, +) +from experiments.generation.task import GenerationError, change_scope, scope_record +from experiments.loader import D6_VALIDATE_EVIDENCE, ScenarioInput +from experiments.trial import DAIL_CLOCK_BASE, STORE_NAME, TrialInputs, _Attempt +from llm.validation.parser import parse_response +from llm.validation.patch import PatchValidator + + +@dataclass(frozen=True) +class ReplaySummary: + path: Path + results: list[dict[str, Any]] + generation_failures: list[dict[str, Any]] + + +def s0_only(scenario: ScenarioInput) -> ScenarioInput: + """The scenario with its first step only: replay never opens a later step.""" + return dataclasses.replace(scenario, steps=scenario.steps[:1]) + + +def _replay_one( + *, + generation_id: str, + replay_name: str, + row: Mapping[str, Any], + frozen: Mapping[str, Any], + reply: str, + request_id: str, + scenario: ScenarioInput, + condition: Condition, + folder: Path, + config: Any, +) -> dict[str, Any]: + trial_id = name_id( + "p10a-replay", generation_id, replay_name, row["model_key"], row["scenario_id"], + row["k"], condition.value, + ) # fmt: skip + inputs = TrialInputs( + trial_id=trial_id, + experiment_id=generation_id, + scenario=scenario, + condition=condition, + repetition=int(row["k"]), + attempt=1, + admission_evidence=D6_VALIDATE_EVIDENCE, + ) + attempt = _Attempt(inputs, folder, config, None) + setup = setup_for(condition, scenario, DAIL_CLOCK_BASE) + for name in ATTEMPT_DIRS: + (folder / name).mkdir(parents=True, exist_ok=True) + store = folder / "store" / STORE_NAME + store.parent.mkdir(parents=True, exist_ok=True) + initialize_database(store) + + def factory() -> UnitOfWork: + return SqliteUnitOfWork(store) + + trial: dict[str, Any] = {} + parent = attempt._baseline(factory, setup, trial) + result: dict[str, Any] = { + "record": "llm_candidate_replay", + "generation_id": generation_id, + "replay_name": replay_name, + "trial_id": trial_id, + "condition": condition.value, + "model_key": row["model_key"], + "scenario_id": row["scenario_id"], + "family": row["family"], + "k": row["k"], + "versions": setup.versions(), + "baseline": trial.get("baseline"), + "frozen_tree_hash": frozen["candidate_tree_hash"], + "rebuilt_tree_hash": None, + "tree_hash_matches": False, + "scope": None, + "scope_matches_frozen": False, + "rebuild_failure": None, + "decision": None, + "result_kind": None, + "build_dir": None, + "transition": None, + } + if parent is None: + result["rebuild_failure"] = "the condition's S0 baseline was refused" + return result + scope = change_scope(parent.resources) + result["scope"] = scope_record(scope) + result["scope_matches_frozen"] = result["scope"] == frozen["scope"] + parsed = parse_response(reply) + if parsed.response is None or parsed.failure_class is not None: + result["rebuild_failure"] = f"the frozen reply does not parse: {list(parsed.errors)}" + return result + with tempfile.TemporaryDirectory(prefix="dail-p10a-replay-") as tmp: + work = Path(tmp) + validator = PatchValidator( + parent_dir=config_only_copy(scenario.steps[0].directory, work / "s0"), + trusted=parent, + scope=scope, + work_root=work / "builds", + variables=dict(scenario.variable_values), + now=attempt.clock(), + ) + outcome = validator.validate( + parsed.response, request_id=request_id, patch_id=attempt.id("patch", 1) + ) + build = outcome.build + if build is None: + result["rebuild_failure"] = ( + f"{None if outcome.failure_class is None else outcome.failure_class.value}: " + f"{list(outcome.errors)}" + ) + return result + result["build_dir"] = str(build.candidate_dir) + result["rebuilt_tree_hash"] = build.tree_hash + result["tree_hash_matches"] = build.tree_hash == frozen["candidate_tree_hash"] + try: + if not result["tree_hash_matches"] or not result["scope_matches_frozen"]: + return result + orchestrator = EvaluationOrchestrator( + factory, + config=config, + scope_policy=setup.scope_policy, + clock=attempt.clock, + policy_version=setup.policy_version, + policy_factory=setup.policy_factory, + identity_strategy=setup.identity_strategy, + ) + request = LLMEvaluationRequest( + lineage_id=attempt.lineage_id, + candidate_id=attempt.id("candidate", 1), + candidate_sequence=1, + attempt_number=1, + patch=build.patch, + candidate_dir=build.candidate_dir, + variables=dict(scenario.variable_values), + definitions=setup.definitions, + bindings=setup.bindings, + context=attempt.context(1), + built_tree_hash=build.tree_hash, + ) + evaluated = orchestrator.evaluate(request) + finally: + shutil.rmtree(build.candidate_dir.parent, ignore_errors=True) + admission = { + "step": 1, + "llm_candidate": f"{row['model_key']}/{row['scenario_id']}/{row['k']}", + "patch_sha256": frozen["patch_sha256"], + "tree_hash": result["rebuilt_tree_hash"], + "tree_hash_matches_frozen": True, + } + transition = records.transition_record( + base=attempt.base, + step=1, + admission=admission, + parent=parent, + open_targets=(), + result=evaluated, + ) + result["transition"] = transition + result["result_kind"] = evaluated.kind.value + result["decision"] = None if evaluated.decision is None else evaluated.decision.decision.value + return result + + +def replay_candidates( + *, + root: Path, + generation_id: str, + scenarios: Mapping[str, ScenarioInput], + conditions: Sequence[str], + config: Any, + replay_name: str, +) -> ReplaySummary: + """Replay every valid frozen candidate of ``generation_id`` (see the module docstring).""" + manifest = read_manifest(root, generation_id) + base = generation_dir(root, generation_id) + out = base / "replay" / replay_name + if out.exists(): + raise GenerationError(f"{out} exists; a replay is written once") + out.mkdir(parents=True) + chosen = [Condition(c) for c in conditions] + results: list[dict[str, Any]] = [] + failures: list[dict[str, Any]] = [] + for row in manifest["attempts"]: + if row["status"] != "PATCH": + failures.append( + { + key: row[key] + for key in ("model_key", "scenario_id", "k", "status", "failure_class") + } + | {"failure_source": row["failure_source"]} + ) + continue + scenario = scenarios.get(row["scenario_id"]) + if scenario is None: + raise GenerationError(f"{row['scenario_id']}: not in the loaded scenarios") + folder = attempt_dir(base, row["model_key"], row["scenario_id"], int(row["k"])) + frozen = read_record(folder / "attempt.json") + reply, request_id = frozen_response_text(folder, frozen) + for condition in chosen: + target = out / row["model_key"] / row["scenario_id"] / str(row["k"]) / condition.value + result = _replay_one( + generation_id=generation_id, + replay_name=replay_name, + row=row, + frozen=frozen, + reply=reply, + request_id=request_id, + scenario=s0_only(scenario), + condition=condition, + folder=target, + config=config, + ) + write_record(target / "result.json", result) + results.append(result) + path = write_record( + out / "replay.json", + { + "record": "llm_candidate_replay_set", + "generation_id": generation_id, + "replay_name": replay_name, + "conditions": [c.value for c in chosen], + "results": [ + {k: v for k, v in r.items() if k not in ("transition", "baseline")} for r in results + ], + "generation_failures": failures, + }, + ) + return ReplaySummary(path, results, failures) diff --git a/experiments/generation/run.py b/experiments/generation/run.py new file mode 100644 index 0000000..12e121d --- /dev/null +++ b/experiments/generation/run.py @@ -0,0 +1,451 @@ +"""Generate and freeze LLM candidates (P10a Section 3.3 "Generation" and "Freezing"; C-218 to +C-220). + +For each scenario (a ``GenerationInput``: S0 only) the parent is built once (``parent.s0_parent``); +then, for each pinned model and each repetition ``k``, one ``Generator.generate`` call through +``llm/adapter/generate.py`` (prompt ``dail-repair-prompt-2``, parser 2, validator 2), so every +request and response record, retry and failure class is P7's. The context is P7's ``build_context`` +with the target invariant as the repair target, the focus of ``task.focus_for`` and the scope of +decision 5; the objective is the fixed task of decision 4. + +**Switch.** ``generation_enabled`` is the caller's; the command line takes it from +``generation_config()``, the repository configuration with the reviewed +``experiments/config/llm.toml`` in place of ``config/base/llm.toml``, loaded for the ``experiment`` +environment with no environment variable (as C-207 (3) does for promotion). ``config/`` keeps it +off. + +**Model check.** Before anything is written, every model's installed facts are checked against its +pin (``models.check_installed``): one refusal stops the run. + +**Freezing (layout)**, under ``/candidates//``:: + + definition.json what was asked (write once; a changed request is refused) + /model.json the pin and the installed facts + /// + records/request-.json P7's LLMRequest of attempt n + records/response-.json P7's LLMResponse of attempt n (raw reply redacted, §33) + patch.tf the patch text of a valid candidate (PATCH) only + timing.json wall-clock time of the call + attempt.json status, failure class and its source, validator outcome, + parent, scope, context and prompt hashes, written last + candidates_manifest.json every attempt with the SHA-256 of each of its files + +Every file is written once (``experiments.artifacts.write_record``); an attempt directory without +its ``attempt.json`` (an interrupted call) is never overwritten or completed, and a generation whose +manifest exists is frozen: it is never regenerated or edited. A REGENERATE (Doc 13 §47) is a new +``generation_id`` (a new name or version). + +**Failure source.** A failed attempt's class comes from the provider when the provider gave no reply +(``raw_response`` is null: a connection failure, a timeout, Ollama's 400/404 as UNSUPPORTED), from +the budget for CONTEXT_INSUFFICIENT (no call was made), and from the model otherwise (the parser or +the validator refused its reply). P7a verdict limit 4: provider UNSUPPORTED is kept apart. +""" + +from __future__ import annotations + +import hashlib +import shutil +import tempfile +import time +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from datetime import UTC, datetime +from pathlib import Path +from typing import Any, Final + +from core.application.config import ResolvedConfig, load_config +from experiments.artifacts import ArtifactError, read_record, write_or_same, write_record +from experiments.definition import name_id +from experiments.generation.models import ( + RETRY_POLICY, + ModelPin, + budget_for, + check_installed, + llm_config, + seed_for, +) +from experiments.generation.parent import Parent, s0_parent +from experiments.generation.task import ( + TASK_TEMPLATE, + GenerationError, + GenerationInput, + change_scope, + config_address, + focus_for, + objective_for, + scope_record, +) +from experiments.trial import DAIL_CLOCK_BASE +from llm.adapter.context import CONTEXT_VERSION, ContextInputs, LLMContext, build_context +from llm.adapter.generate import Generator +from llm.adapter.provider import LLMProvider +from llm.prompts.templates import PROMPT_VERSION, template_hash +from llm.schemas.records import FailureClass, GenerationResult, GenerationStatus +from llm.validation.parser import PARSER_VERSION +from llm.validation.patch import VALIDATOR_VERSION, PatchValidator + +REPO_ROOT: Final = Path(__file__).resolve().parents[2] +EXPERIMENT_LLM_FILE: Final = REPO_ROOT / "experiments" / "config" / "llm.toml" +MANIFEST: Final = "candidates_manifest.json" +SCOPE_RULE: Final = ( + "decision 5 (C-220): every resource address of S0, every supported type, no attribute " + "restriction, all four operations; the same for every condition" +) +ModelEntry = tuple[ModelPin, LLMProvider, Mapping[str, Any]] + +__all__ = [ + "EXPERIMENT_LLM_FILE", + "GenerationError", + "GenerationSpec", + "GenerationSummary", + "generate_candidates", + "generation_config", +] + + +def sha256_file(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def generation_config() -> ResolvedConfig: + """The repository configuration with the reviewed ``experiments/config/llm.toml`` in place of + ``config/base/llm.toml``; refuses to start if generation is not on there.""" + with tempfile.TemporaryDirectory(prefix="dail-generation-config-") as tmp: + root = Path(tmp) / "config" + shutil.copytree(REPO_ROOT / "config", root) + shutil.copyfile(EXPERIMENT_LLM_FILE, root / "base" / "llm.toml") + config = load_config(root, "experiment", environ={}) + if config.get("llm.generation_enabled") is not True: + raise GenerationError("the generation configuration must enable llm.generation_enabled") + return config + + +@dataclass(frozen=True) +class GenerationSpec: + name: str + version: str + + @property + def generation_id(self) -> str: + return name_id("generation", self.name, self.version) + + +@dataclass(frozen=True) +class GenerationSummary: + generation_id: str + manifest_path: Path + manifest_sha256: str + attempts: list[dict[str, Any]] + + +def generation_dir(root: Path, generation_id: str) -> Path: + return Path(root) / "candidates" / generation_id + + +def attempt_dir(base: Path, model_key: str, scenario_id: str, k: int) -> Path: + return base / model_key / scenario_id / str(k) + + +def _wall() -> datetime: + return datetime.now(UTC) + + +def failure_source(result: GenerationResult) -> str | None: + if result.failure_class is None or result.status is GenerationStatus.DISABLED: + return None + if result.status is GenerationStatus.CONTEXT_INSUFFICIENT or not result.attempts: + return "budget" + if result.attempts[-1].response.raw_response is None: + return "provider" + return "model" + + +def _context(given: GenerationInput, parent: Parent) -> tuple[LLMContext, dict[str, Any]]: + state = parent.state + (target,) = (d for d in parent.definitions if d.invariant_id == given.target_invariant_id) + scope = change_scope(state.resources) + focus = focus_for(target, state.resources) + objective = objective_for(target) + context = build_context( + ContextInputs( + parent_state_id=state.state_id, + trusted_state_hash=state.state_hash, + resources=tuple(state.resources), + invariants=parent.definitions, + statuses={r.invariant_id: r.status.value for r in state.invariant_refs}, + objective=objective, + repair_targets=(target.invariant_id,), + scope=scope, + focus=focus, + ) + ) + facts = { + "parent": { + "state_id": state.state_id, + "state_hash": state.state_hash, + "resource_set_hash": state.resource_set_hash(), + "resource_addresses": sorted({config_address(r.address) for r in state.resources}), + "s0_content_hash": given.s0_content_hash, + "configuration_files": sorted(p.name for p in parent.directory.iterdir()), + "invariants": { + r.invariant_id: {"version": r.invariant_version, "status": r.status.value} + for r in state.invariant_refs + }, + }, + "scope": scope_record(scope), + "objective": objective, + "focus": list(focus), + "target_invariant_id": target.invariant_id, + } + return context, facts + + +def _definition( + spec: GenerationSpec, + inputs: Sequence[GenerationInput], + models: Sequence[ModelEntry], + k: int, + environment: Mapping[str, str], +) -> dict[str, Any]: + return { + "record": "generation_definition", + "generation_id": spec.generation_id, + "name": spec.name, + "version": spec.version, + "task_template": TASK_TEMPLATE, + "scope_rule": SCOPE_RULE, + "context_version": CONTEXT_VERSION, + "prompt_version": PROMPT_VERSION, + "template_hash": template_hash(PROMPT_VERSION), + "parser_version": PARSER_VERSION, + "validator_version": VALIDATOR_VERSION, + "retry_policy": RETRY_POLICY.to_dict(), + "repetitions": k, + "seed_rule": "seed + k - 1", + "scenarios": [g.to_dict() for g in inputs], + "models": [pin.to_dict() for pin, _, _ in models], + "experiment_llm_config_sha256": sha256_file(EXPERIMENT_LLM_FILE), + "environment": dict(environment), + } + + +def _attempt_record( + *, + spec: GenerationSpec, + pin: ModelPin, + given: GenerationInput, + k: int, + result: GenerationResult, + facts: Mapping[str, Any], + context: LLMContext, +) -> dict[str, Any]: + final = result.attempts[-1].response if result.attempts else None + build = result.build + return { + "record": "llm_candidate_attempt", + "generation_id": spec.generation_id, + "model_key": pin.key, + "model_tag": pin.tag, + "model_digest": pin.digest, + "scenario_id": given.scenario_id, + "family": given.family, + "k": k, + "seed": seed_for(pin, k), + "status": result.status.value, + "failure_class": None if result.failure_class is None else result.failure_class.value, + "failure_source": failure_source(result), + "detail": result.detail, + **facts, + "context_version": context.context_version, + "context_hash": context.context_hash, + "observability": result.observability(), + "config": result.config.to_dict(), + "parsed_status": None if final is None else final.parsed_status, + "parse_errors": [] if final is None else list(final.parse_errors), + "validation_errors": [] if final is None else list(final.validation_errors), + "patch_sha256": ( + None if result.patch is None else hashlib.sha256(result.patch.encode()).hexdigest() + ), + "patch_id": None if final is None else final.patch_id, + "final_request_id": None if not result.attempts else result.attempts[-1].request.request_id, + "candidate_tree_hash": None if build is None else build.tree_hash, + "candidate_resource_set_hash": None if build is None else build.resource_set_hash, + } + + +def _one( + *, + spec: GenerationSpec, + entry: ModelEntry, + given: GenerationInput, + parent: Parent, + context: LLMContext, + facts: Mapping[str, Any], + k: int, + folder: Path, + work: Path, + generation_enabled: bool, +) -> dict[str, Any]: + pin, provider, _ = entry + validator = PatchValidator( + parent_dir=parent.directory, + trusted=parent.state, + scope=change_scope(parent.state.resources), + work_root=work / "builds", + variables=dict(given.variable_values), + now=DAIL_CLOCK_BASE, + ) + generator = Generator( + provider=provider, + config=llm_config(pin, provider.name, k), + retry_policy=RETRY_POLICY, + budget=budget_for(pin), + generation_enabled=generation_enabled, + clock=_wall, + validator=validator, + ) + folder.mkdir(parents=True) + started_at, started = _wall(), time.perf_counter() + result = generator.generate(context) + seconds = time.perf_counter() - started + try: + for attempt in result.attempts: + n = attempt.request.attempt_number + write_record(folder / "records" / f"request-{n}.json", attempt.request.to_dict()) + write_record(folder / "records" / f"response-{n}.json", attempt.response.to_dict()) + if result.status is GenerationStatus.PATCH and result.patch is not None: + path = folder / "patch.tf" + with path.open("xb") as handle: + handle.write(result.patch.encode("utf-8")) + write_record( + folder / "timing.json", + { + "started_at": started_at.isoformat(), + "completed_at": _wall().isoformat(), + "wall_seconds": seconds, + }, + ) + record = _attempt_record( + spec=spec, pin=pin, given=given, k=k, result=result, facts=facts, context=context + ) + write_record(folder / "attempt.json", record) + finally: + if result.build is not None: # the build is not kept: replay and export rebuild it + shutil.rmtree(result.build.candidate_dir.parent, ignore_errors=True) + return record + + +def _manifest_row(base: Path, folder: Path, record: Mapping[str, Any]) -> dict[str, Any]: + files = { + p.relative_to(base).as_posix(): sha256_file(p) + for p in sorted(folder.rglob("*")) + if p.is_file() + } + return { + "model_key": record["model_key"], + "scenario_id": record["scenario_id"], + "family": record["family"], + "k": record["k"], + "status": record["status"], + "failure_class": record["failure_class"], + "failure_source": record["failure_source"], + "patch_sha256": record["patch_sha256"], + "candidate_tree_hash": record["candidate_tree_hash"], + "files": files, + } + + +def generate_candidates( + *, + spec: GenerationSpec, + inputs: Sequence[GenerationInput], + models: Sequence[ModelEntry], + k: int, + root: Path, + generation_enabled: bool, + environment: Mapping[str, str], +) -> GenerationSummary: + """Generate, freeze and list every attempt (see the module docstring).""" + if k < 1 or not inputs or not models: + raise GenerationError("generation needs k >= 1, at least one scenario and one model") + base = generation_dir(root, spec.generation_id) + if (base / MANIFEST).exists(): + raise GenerationError( + f"generation {spec.generation_id} is frozen; a REGENERATE is a new generation_id " + "(Doc 13 §47)" + ) + for pin, _, installed in models: + check_installed(pin, installed) + planned = [(given, entry, n) for given in inputs for entry in models for n in range(1, k + 1)] + for given, (pin, _, _), n in planned: + folder = attempt_dir(base, pin.key, given.scenario_id, n) + if folder.exists() and not (folder / "attempt.json").exists(): + raise GenerationError( + f"{folder} is an interrupted attempt; it is never overwritten (a REGENERATE is a " + "new generation_id)" + ) + try: + write_or_same(base / "definition.json", _definition(spec, inputs, models, k, environment)) + for pin, provider, installed in models: + write_or_same( + base / pin.key / "model.json", + {"pin": pin.to_dict(), "installed": dict(installed), "provider": provider.name}, + ) + except ArtifactError as changed: + raise GenerationError(f"the generation request changed: {changed}") from None + rows: list[dict[str, Any]] = [] + for given in inputs: + with tempfile.TemporaryDirectory(prefix="dail-p10a-generation-") as tmp: + work = Path(tmp) + parent = s0_parent(given, work / "parent", "generation") + context, facts = _context(given, parent) + for entry in models: + pin = entry[0] + for n in range(1, k + 1): + folder = attempt_dir(base, pin.key, given.scenario_id, n) + if (folder / "attempt.json").exists(): + record = read_record(folder / "attempt.json") + else: + record = _one( + spec=spec, + entry=entry, + given=given, + parent=parent, + context=context, + facts=facts, + k=n, + folder=folder, + work=work, + generation_enabled=generation_enabled, + ) + rows.append(_manifest_row(base, folder, record)) + manifest = { + "record": "candidates_manifest", + "generation_id": spec.generation_id, + "definition_sha256": sha256_file(base / "definition.json"), + "attempts": rows, + } + path = write_record(base / MANIFEST, manifest) + return GenerationSummary(spec.generation_id, path, sha256_file(path), rows) + + +def read_manifest(root: Path, generation_id: str) -> dict[str, Any]: + path = generation_dir(root, generation_id) / MANIFEST + if not path.exists(): + raise GenerationError(f"generation {generation_id} has no manifest: it is not frozen") + data = read_record(path) + assert isinstance(data, dict) + return data + + +def frozen_response_text(folder: Path, record: Mapping[str, Any]) -> tuple[str, str]: + """(the reply text of the final attempt, its request id), as frozen.""" + n = len(record["observability"]["request_ids"]) + response = read_record(folder / "records" / f"response-{n}.json") + if not isinstance(response, dict) or not isinstance(response.get("raw_response"), str): + raise GenerationError(f"{folder}: no frozen reply text") + return response["raw_response"], response["request_id"] + + +def failure_classes() -> tuple[str, ...]: + return tuple(c.value for c in FailureClass) diff --git a/experiments/generation/task.py b/experiments/generation/task.py new file mode 100644 index 0000000..f3533cc --- /dev/null +++ b/experiments/generation/task.py @@ -0,0 +1,124 @@ +"""What generation is given per scenario (P10a Section 3.3 "Input per scenario"; decisions 4 and 5, +C-219, C-220). + +``GenerationInput`` is everything generation may know of a scenario: "the S0 directory and its +hashes (through the existing loader), ``target.invariant_id``, the endpoints, the primary variable +values. Nothing else." (plus the scenario id and family, which name the records). It is projected +from the loader's ``ScenarioInput`` by ``generation_input``, which takes the first step only and +refuses a scenario that is not a repair (decision 4: "repair scenarios only"). + +``change_scope`` is decision 5: every resource address of S0, every supported type, no attribute +restriction and all four operations; ``scope_record`` is its JSON form, recorded with each +candidate and with each replay under every condition. + +``focus_for`` is the context's focus (Section 3.3 "Context": "focus = the target invariant's scope +resources"): the S0 resources the target definition's ``scope`` names, by address +(``scope.resources``) or by type (``scope.resource_types``). Nothing is walked from them; the +dependency walk is the impact engine's, and S0 has no impact report. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Mapping +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from core.domain.change_scope import ALL_OPERATIONS, AllowedChangeScope +from core.domain.invariant import Invariant +from core.domain.resource import Resource +from experiments.loader import ScenarioInput +from experiments.task_text import TASK_TEMPLATE, objective_for +from llm.adapter.context import SUPPORTED_TYPES + +REPAIR = "repair" + +__all__ = [ + "TASK_TEMPLATE", + "GenerationError", + "GenerationInput", + "change_scope", + "focus_for", + "generation_input", + "objective_for", + "scope_record", +] + + +class GenerationError(Exception): + """Generation cannot run as asked (a scenario that is not a repair, a frozen generation).""" + + +@dataclass(frozen=True) +class GenerationInput: + scenario_id: str + family: str + s0_dir: Path + s0_content_hash: str + target_invariant_id: str + source_endpoint: str + destination_endpoint: str + variable_values: Mapping[str, Any] + + def to_dict(self) -> dict[str, Any]: + return { + "scenario_id": self.scenario_id, + "family": self.family, + "s0_content_hash": self.s0_content_hash, + "target_invariant_id": self.target_invariant_id, + "source_endpoint": self.source_endpoint, + "destination_endpoint": self.destination_endpoint, + "variable_values": dict(self.variable_values), + } + + +def generation_input(scenario: ScenarioInput) -> GenerationInput: + if scenario.target_kind != REPAIR or scenario.target_invariant_id is None: + raise GenerationError( + f"{scenario.scenario_id}: only repair scenarios get LLM candidates (C-219)" + ) + first = scenario.steps[0] + if first.index != 0: + raise GenerationError(f"{scenario.scenario_id}: the first step is not S0") + return GenerationInput( + scenario_id=scenario.scenario_id, + family=scenario.family, + s0_dir=first.directory, + s0_content_hash=first.content_hash, + target_invariant_id=scenario.target_invariant_id, + source_endpoint=scenario.source_endpoint, + destination_endpoint=scenario.destination_endpoint, + variable_values=dict(scenario.variable_values), + ) + + +def config_address(address: str) -> str: + """``type.name`` of a resource address (an instance key is dropped, C-189).""" + return address.partition("[")[0] + + +def change_scope(resources: Iterable[Resource]) -> AllowedChangeScope: + """Decision 5 (C-220): "every resource address of S0" as configuration addresses.""" + return AllowedChangeScope( + resource_addresses=tuple(sorted({config_address(r.address) for r in resources})), + resource_types=SUPPORTED_TYPES, + attributes=(), + operation_types=ALL_OPERATIONS, + ) + + +def scope_record(scope: AllowedChangeScope) -> dict[str, Any]: + return { + "resource_addresses": list(scope.resource_addresses), + "resource_types": list(scope.resource_types), + "attributes": list(scope.attributes), + "operation_types": [op.value for op in scope.operation_types], + } + + +def focus_for(definition: Invariant, resources: Iterable[Resource]) -> tuple[str, ...]: + named = set(definition.scope.resources) + types = set(definition.scope.resource_types) + return tuple( + sorted(r.address for r in resources if r.address in named or r.resource_type in types) + ) diff --git a/experiments/metrics/extractor.py b/experiments/metrics/extractor.py index 70960b2..47de96e 100644 --- a/experiments/metrics/extractor.py +++ b/experiments/metrics/extractor.py @@ -5,8 +5,9 @@ ``EXTRACTOR_VERSION`` names this code; every metric record and report carries both. The code of ``experiments/metrics`` and ``experiments/scoring`` is pinned by hash per extractor version in ``tests.unit.test_p10a_extractor_version``: a code change needs a new ``EXTRACTOR_VERSION``. The -first extractor was version ``metrics-draft-1`` (one value for both, P8b); ``extractor-2`` is the same -code with the two versions apart and the generation metrics of ``experiments.metrics.generation``. +first extractor was version ``metrics-draft-1`` (one value for both, P8b); ``extractor-2`` is the +same code with the two versions apart and the generation metrics of +``experiments.metrics.generation``. Input: the transitions of the stored trial records (``experiments.metrics.model``), the scenarios' truth from ``experiments.scoring``, DAIL's S0 results under B, and the identity truth table. Output: diff --git a/experiments/metrics/generation.py b/experiments/metrics/generation.py new file mode 100644 index 0000000..79a2347 --- /dev/null +++ b/experiments/metrics/generation.py @@ -0,0 +1,140 @@ +"""Generation metrics of frozen LLM candidates (P10a Section 3.3 "Generation metrics"; Doc 13 §26; +P7a verdict limits 4 and 5; metric definitions ``metrics-draft-1``, extractor ``extractor-2``). + +"Generation metrics (in the extractor, under ``metrics-draft-1``): valid patch rate per model +(Doc 13 §26), failure classes per model (with Ollama UNSUPPORTED kept apart from model failures, +P7a limit 4), retries, latency, and token counts (marked as estimates where they come from +``ceil(characters / 4)``, P7a limit 5). Repair metrics that need truth wait for the oracle labels." + +Input: the frozen attempt records (``attempt.json``), their response records and timings, and the +replay results; nothing else, and no label. Per model: + +* ``valid_patch_rate``: attempts whose status is PATCH (parsed and passed validator 2, so a + candidate was built) / attempts (one attempt = one scenario and one ``k``), with Wilson 95%; +* failure classes by source: ``provider`` (no reply: connection, timeout, Ollama 400/404 as + UNSUPPORTED), ``budget`` (CONTEXT_INSUFFICIENT, no call), ``model`` (the reply was refused by the + parser or the validator); +* calls (provider requests, retries included) and retries (calls - attempts that made a call); +* latency per call in ms (mean, max) from the response records (calls with a reply); +* tokens: ``provider`` counts as the local server reported them (``prompt_eval_count``, + ``eval_count``), and the budget's ``estimate`` of the input (``ceil(characters / 4)``), labelled + as such; +* wall time: the sum of the frozen ``timing.json`` durations (generation, validation and build of + every attempt), its mean per attempt, and the projection mean x scenarios x k; +* DAIL decisions per condition from the replay (PROMOTE, REJECT, ..., or NO DECISION when the + rebuild did not match). +""" + +from __future__ import annotations + +import statistics +from collections import Counter +from collections.abc import Iterable, Mapping, Sequence +from typing import Any, Final + +from experiments.metrics.extractor import EXTRACTOR_VERSION, METRICS_VERSION, wilson + +ESTIMATE_RULE: Final = "ceil(characters / 4)" +PROJECTION_SCENARIOS: Final = 36 +PROJECTION_K: Final = (1, 3, 5) +NO_DECISION: Final = "NO DECISION" + + +def _mean(values: Sequence[float]) -> float | None: + return statistics.fmean(values) if values else None + + +def model_metrics( + attempts: Sequence[Mapping[str, Any]], + responses: Mapping[tuple[str, str, int], Sequence[Mapping[str, Any]]], + timings: Mapping[tuple[str, str, int], Mapping[str, Any]], + replays: Iterable[Mapping[str, Any]], +) -> dict[str, dict[str, Any]]: + """Per model key (see the module docstring). ``responses`` and ``timings`` are keyed by + (model key, scenario id, k).""" + out: dict[str, dict[str, Any]] = {} + keys = sorted({a["model_key"] for a in attempts}) + replay_list = list(replays) + for key in keys: + mine = [a for a in attempts if a["model_key"] == key] + n = len(mine) + valid = sum(a["status"] == "PATCH" for a in mine) + sources: dict[str, Counter[str]] = { + "provider": Counter(), + "budget": Counter(), + "model": Counter(), + } + statuses: Counter[str] = Counter(a["status"] for a in mine) + for a in mine: + if a["failure_class"] is not None: + sources[a["failure_source"]][a["failure_class"]] += 1 + calls: list[Mapping[str, Any]] = [] + for a in mine: + calls += list(responses.get((key, a["scenario_id"], int(a["k"])), ())) + made_a_call = sum(1 for a in mine if responses.get((key, a["scenario_id"], int(a["k"])))) + latencies = [float(r["latency_ms"]) for r in calls if r.get("latency_ms") is not None] + tokens_in = [int(r["token_usage"]["input"]) for r in calls if "input" in r["token_usage"]] + tokens_out = [ + int(r["token_usage"]["output"]) for r in calls if "output" in r["token_usage"] + ] + estimates = [ + int(a["observability"]["budget"]["estimated_input_tokens"]) + for a in mine + if a["observability"]["budget"].get("estimated_input_tokens") is not None + ] + walls = [ + float(t["wall_seconds"]) + for a in mine + if (t := timings.get((key, a["scenario_id"], int(a["k"])))) is not None + ] + per_attempt = _mean(walls) + decisions: dict[str, dict[str, int]] = {} + for r in replay_list: + if r["model_key"] == key: + cell = decisions.setdefault(r["condition"], {}) + label = r["decision"] or NO_DECISION + cell[label] = cell.get(label, 0) + 1 + low, high = wilson(valid, n) + out[key] = { + "metrics_version": METRICS_VERSION, + "extractor_version": EXTRACTOR_VERSION, + "attempts": n, + "statuses": dict(sorted(statuses.items())), + "valid_patch_rate": { + "numerator": valid, + "denominator": n, + "value": valid / n if n else None, + "wilson95": [low, high], + }, + "failure_classes": {s: dict(sorted(c.items())) for s, c in sources.items()}, + "calls": len(calls), + "retries": len(calls) - made_a_call, + "latency_ms": { + "calls_with_a_reply": len(latencies), + "mean": _mean(latencies), + "max": max(latencies) if latencies else None, + }, + "tokens": { + "provider_input_total": sum(tokens_in), + "provider_output_total": sum(tokens_out), + "provider_input_mean_per_call": _mean(tokens_in), + "provider_output_mean_per_call": _mean(tokens_out), + "provider_source": "the local server's prompt_eval_count and eval_count", + "estimated_input_mean": _mean(estimates), + "estimated_input_max": max(estimates) if estimates else None, + "estimate_rule": f"estimate: {ESTIMATE_RULE} of the rendered prompt", + }, + "wall_seconds": { + "total": sum(walls), + "mean_per_attempt": per_attempt, + "max_per_attempt": max(walls) if walls else None, + }, + "projection_seconds": { + f"{PROJECTION_SCENARIOS} scenarios x k={k}": ( + None if per_attempt is None else per_attempt * PROJECTION_SCENARIOS * k + ) + for k in PROJECTION_K + }, + "decisions": dict(sorted(decisions.items())), + } + return out diff --git a/experiments/report/__main__.py b/experiments/report/__main__.py index c69ef65..e670a3d 100644 --- a/experiments/report/__main__.py +++ b/experiments/report/__main__.py @@ -3,6 +3,8 @@ python -m experiments.report extract --name NAME [--version V] [--doc docs/P8_DEV_PILOT] python -m experiments.report select --name NAME [--version V] python -m experiments.report compare --name NAME --other OTHER [--version V] + python -m experiments.report pilot --name GENERATION [--version V] --replay-name R + --licence-checks FILE.json [--doc docs/P10A_MODEL_PILOT] ``extract`` reads the stored records of the experiment under ``$DAIL_EXPERIMENT_ROOT`` and the dev labels under ``$TERRAPRESERVE_ROOT`` (through ``experiments.scoring``), runs the extractor and @@ -10,7 +12,9 @@ it checks that what it would write is byte-identical to what is stored (Doc 13 §30) and fails if not. With ``--doc`` it also writes ``.json`` and ``.md``. ``select`` prints the scenario ids of the reproducibility re-run. ``compare`` compares the decisions and invariant results of two -experiments on their common (scenario, condition, step) and prints the counts. +experiments on their common (scenario, condition, step) and prints the counts. ``pilot`` (P10a +Step 6) reads one frozen generation and its replay and writes the model pilot report once, under +``candidates//pilot_report//`` (and with ``--doc`` beside the docs). """ from __future__ import annotations @@ -23,10 +27,11 @@ from typing import Any from experiments.artifacts import ArtifactError, dumps, read_record, write_or_same -from experiments.definition import experiment_id_for +from experiments.definition import experiment_id_for, name_id from experiments.metrics.extractor import EXTRACTOR_VERSION, dumps_metrics, extract from experiments.metrics.model import Transition, transitions_from_result, trial_outcome from experiments.report.build import build_report +from experiments.report.pilot import build_pilot, collect_pilot, render_pilot from experiments.report.render import render_markdown from experiments.runner import Layout, read_events, read_result from experiments.scoring.identity import load_identity_table @@ -195,10 +200,29 @@ def cmd_compare(args: argparse.Namespace) -> int: return 0 if not decision_diff and not result_diff else 1 +def cmd_pilot(args: argparse.Namespace) -> int: + root = _env("DAIL_EXPERIMENT_ROOT") + generation_id = name_id("generation", args.name, args.version) + collected = collect_pilot(root, generation_id, args.replay_name) + checks = json.loads(Path(args.licence_checks).read_text(encoding="utf-8")) + report = build_pilot(collected, checks) + out = root / "candidates" / generation_id / "pilot_report" / args.replay_name + try: + write_or_same(out / "pilot.json", report) + except ArtifactError as changed: + raise SystemExit(str(changed)) from None + (out / "pilot.md").write_text(render_pilot(report), encoding="utf-8", newline="\n") + if args.doc: + Path(f"{args.doc}.json").write_bytes(dumps(report)) + Path(f"{args.doc}.md").write_text(render_pilot(report), encoding="utf-8", newline="\n") + print(json.dumps({"generation_id": generation_id, "pilot_report": str(out)})) + return 0 + + def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(prog="python -m experiments.report") commands = parser.add_subparsers(dest="command", required=True) - for name in ("extract", "select", "compare"): + for name in ("extract", "select", "compare", "pilot"): command = commands.add_parser(name) command.add_argument("--name", required=True) command.add_argument("--version", default="1") @@ -208,8 +232,17 @@ def main(argv: list[str] | None = None) -> int: command.add_argument("--preface", default="", help="a text file shown at the top") if name == "compare": command.add_argument("--other", required=True) + if name == "pilot": + command.add_argument("--replay-name", required=True) + command.add_argument("--licence-checks", required=True) + command.add_argument("--doc", default="") args = parser.parse_args(argv) - handler = {"extract": cmd_extract, "select": cmd_select, "compare": cmd_compare}[args.command] + handler = { + "extract": cmd_extract, + "select": cmd_select, + "compare": cmd_compare, + "pilot": cmd_pilot, + }[args.command] return handler(args) diff --git a/experiments/report/pilot.py b/experiments/report/pilot.py new file mode 100644 index 0000000..0e75091 --- /dev/null +++ b/experiments/report/pilot.py @@ -0,0 +1,312 @@ +"""The dev model pilot report (P10a Step 6; ``docs/P10A_MODEL_PILOT.md``). + +``collect_pilot`` reads the frozen records of one generation and one replay under +``$DAIL_EXPERIMENT_ROOT/candidates//`` (the manifest, every ``attempt.json`` with its +response records and ``timing.json``, each model's ``model.json``, and +``replay//replay.json``) and nothing else: no label, no oracle. ``build_pilot`` adds the +generation metrics of ``experiments.metrics.generation`` and the licence checks, and +``render_pilot`` writes the Markdown. The JSON is the record; the Markdown is its rendering (a test +renders the committed JSON and compares). + +Step 6: "Do not rank the models' safety. There are no labels yet, and the pilot only sizes the main +run." The report says so, and orders models by their key only. +""" + +from __future__ import annotations + +import hashlib +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + +from experiments.artifacts import read_record +from experiments.metrics.extractor import EXTRACTOR_VERSION, METRICS_VERSION +from experiments.metrics.generation import PROJECTION_K, PROJECTION_SCENARIOS, model_metrics + + +def _sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def collect_pilot(root: Path, generation_id: str, replay_name: str) -> dict[str, Any]: + base = Path(root) / "candidates" / generation_id + manifest_path = base / "candidates_manifest.json" + manifest = read_record(manifest_path) + definition = read_record(base / "definition.json") + attempts: list[dict[str, Any]] = [] + responses: dict[tuple[str, str, int], list[dict[str, Any]]] = {} + timings: dict[tuple[str, str, int], dict[str, Any]] = {} + for row in manifest["attempts"]: + key = (row["model_key"], row["scenario_id"], int(row["k"])) + folder = base / row["model_key"] / row["scenario_id"] / str(row["k"]) + attempts.append(read_record(folder / "attempt.json")) + records = folder / "records" + responses[key] = ( + [ + read_record(records / f"response-{n}.json") + for n in range(1, len(list(records.glob("response-*.json"))) + 1) + ] + if records.exists() + else [] + ) + timings[key] = read_record(folder / "timing.json") + models = {p.parent.name: read_record(p) for p in sorted(base.glob("*/model.json"))} + replay_path = base / "replay" / replay_name / "replay.json" + replay = read_record(replay_path) + return { + "generation_id": generation_id, + "definition": definition, + "manifest_sha256": _sha(manifest_path), + "replay_name": replay_name, + "replay_sha256": _sha(replay_path), + "attempts": attempts, + "responses": responses, + "timings": timings, + "models": models, + "replay": replay, + } + + +def build_pilot( + collected: Mapping[str, Any], licence_checks: Mapping[str, Mapping[str, str]] +) -> dict[str, Any]: + """The pilot report as JSON. ``licence_checks`` holds, per model key, what the author checked on + the day (the Ollama library page and ``ollama show --license``).""" + metrics = model_metrics( + collected["attempts"], + collected["responses"], + collected["timings"], + collected["replay"]["results"], + ) + definition = collected["definition"] + models = [] + for key in sorted(metrics): + stored = collected["models"][key] + models.append( + { + "key": key, + "pin": stored["pin"], + "installed": stored["installed"], + "licence_check": dict(licence_checks.get(key, {})), + "metrics": metrics[key], + } + ) + scenarios = definition["scenarios"] + return { + "record": "p10a_model_pilot", + "metrics_version": METRICS_VERSION, + "extractor_version": EXTRACTOR_VERSION, + "generation_id": collected["generation_id"], + "generation_name": definition["name"], + "generation_version": definition["version"], + "candidates_manifest_sha256": collected["manifest_sha256"], + "replay_name": collected["replay_name"], + "replay_sha256": collected["replay_sha256"], + "environment": definition["environment"], + "prompt_version": definition["prompt_version"], + "parser_version": definition["parser_version"], + "validator_version": definition["validator_version"], + "retry_policy": definition["retry_policy"], + "repetitions": definition["repetitions"], + "scenarios": [ + {"scenario_id": s["scenario_id"], "family": s["family"], + "target_invariant_id": s["target_invariant_id"]} + for s in scenarios + ], # fmt: skip + "conditions": collected["replay"]["conditions"], + "models": models, + } + + +def _n(value: Any, digits: int = 1) -> str: + if value is None: + return "n/a" + if isinstance(value, float): + return f"{value:.{digits}f}" + return str(value) + + +def _classes(cells: Mapping[str, Mapping[str, int]], source: str) -> str: + found = cells.get(source) or {} + return ", ".join(f"{k} {v}" for k, v in found.items()) or "none" + + +def _table(head: Sequence[str], rows: Sequence[Sequence[str]]) -> list[str]: + lines = ["| " + " | ".join(head) + " |", "|" + "---|" * len(head)] + lines += ["| " + " | ".join(row) + " |" for row in rows] + return lines + + +def render_pilot(report: Mapping[str, Any]) -> str: + env = report["environment"] + models = report["models"] + lines = [ + "# P10a dev model pilot", + "", + "Generated by `python -m experiments.report pilot` from the frozen candidate records and", + "the replay; do not edit by hand. Dev split only. It sizes the main run and is not a", + "result: there are no oracle labels for these candidates yet, so no model's safety is", + "judged or ranked (P10a Step 6). The authors choose the models and k at the protocol", + "freeze (P10b).", + "", + f"- Generation: `{report['generation_name']}` version {report['generation_version']}, id " + f"`{report['generation_id']}`; candidates manifest sha256 " + f"`{report['candidates_manifest_sha256']}`", + f"- Replay: `{report['replay_name']}` (sha256 `{report['replay_sha256']}`), conditions " + + ", ".join(report["conditions"]), + f"- DAIL `{env.get('dail_commit')}` (clean tree: {env.get('dail_worktree_clean')}); " + f"TerraPreserve `{env.get('terrapreserve_tag')}`; Ollama {env.get('ollama_version')}", + f"- Prompt `{report['prompt_version']}`, parser `{report['parser_version']}`, validator " + f"`{report['validator_version']}`; retry policy `{report['retry_policy']['policy_id']}` " + f"(at most {report['retry_policy']['max_attempts']} calls, again only after " + + ", ".join(report["retry_policy"]["retry_on"]) + + ")", + f"- Metric definitions `{report['metrics_version']}`, extractor " + f"`{report['extractor_version']}`", + f"- Scenarios: {len(report['scenarios'])} dev repair scenarios, k = " + f"{report['repetitions']}: " + + ", ".join( + f"{s['scenario_id']} ({s['target_invariant_id']})" for s in report["scenarios"] + ), + "", + "## 1. Models", + "", + ] + lines += _table( + ["Model", "Tag", "Digest", "Quantization", "Size", "Licence", "Licence checked"], + [ + [ + m["key"], + f"`{m['pin']['tag']}`", + f"`{m['pin']['digest']}`", + m["pin"]["quantization"], + m["pin"]["size"], + m["pin"]["licence"], + m["pin"]["licence_source"], + ] + for m in models + ], + ) + lines += [ + "", + "Settings: " + + "; ".join( + f"{m['key']}: temperature {m['pin']['temperature']}, top_p {m['pin']['top_p']}, seed " + f"{m['pin']['seed']}, max_output_tokens {m['pin']['max_output_tokens']}, num_ctx " + f"{m['pin']['num_ctx']}, timeout {m['pin']['timeout_seconds']} s" + for m in models + ), + "", + "## 2. Validity, failures and retries", + "", + ] + lines += _table( + ["Model", "Valid patches", "Failures: model", "Failures: provider", "Failures: budget", + "Calls", "Retries"], + [ + [ + m["key"], + f"{m['metrics']['valid_patch_rate']['numerator']} / " + f"{m['metrics']['valid_patch_rate']['denominator']}", + _classes(m["metrics"]["failure_classes"], "model"), + _classes(m["metrics"]["failure_classes"], "provider"), + _classes(m["metrics"]["failure_classes"], "budget"), + str(m["metrics"]["calls"]), + str(m["metrics"]["retries"]), + ] + for m in models + ], + ) # fmt: skip + lines += [ + "", + "Provider failures (no reply: connection, timeout, Ollama 400/404 as UNSUPPORTED) are kept", + "apart from model failures (a reply the parser or validator 2 refused) (P7a limit 4).", + "", + "## 3. Latency, tokens and wall time", + "", + ] + lines += _table( + ["Model", "Latency per call, mean (s)", "max (s)", "Tokens in / out per call (server)", + "Input estimate, mean / max", "Wall time total (s)", "per attempt, mean (s)"], + [ + [ + m["key"], + _n(None if m["metrics"]["latency_ms"]["mean"] is None + else m["metrics"]["latency_ms"]["mean"] / 1000), + _n(None if m["metrics"]["latency_ms"]["max"] is None + else m["metrics"]["latency_ms"]["max"] / 1000), + f"{_n(m['metrics']['tokens']['provider_input_mean_per_call'], 0)} / " + f"{_n(m['metrics']['tokens']['provider_output_mean_per_call'], 0)}", + f"{_n(m['metrics']['tokens']['estimated_input_mean'], 0)} / " + f"{_n(m['metrics']['tokens']['estimated_input_max'])}", + _n(m["metrics"]["wall_seconds"]["total"]), + _n(m["metrics"]["wall_seconds"]["mean_per_attempt"]), + ] + for m in models + ], + ) # fmt: skip + lines += [ + "", + "Token counts per call are the local server's (`prompt_eval_count`, `eval_count`). The", + "input estimate is the budget's estimate, ceil(characters / 4) of the rendered prompt", + "(P7a limit 5), not a tokenizer count. Wall time is the frozen duration of each attempt", + "(calls, validation and build).", + "", + "## 4. DAIL decisions on the valid candidates", + "", + ] + conditions = report["conditions"] + lines += _table( + ["Model", *conditions], + [ + [ + m["key"], + *[ + ", ".join(f"{k} {v}" for k, v in m["metrics"]["decisions"].get(c, {}).items()) + or "none" + for c in conditions + ], + ] + for m in models + ], + ) + lines += [ + "", + "Each valid candidate is evaluated once per condition as a one-step candidate from S0.", + "These are DAIL's decisions, not judgements of the candidates: without oracle labels a", + "PROMOTE is", + "not known to be safe, and a REJECT is not known to be right.", + "", + f"## 5. Projection for {PROJECTION_SCENARIOS} test repair scenarios", + "", + ] + lines += _table( + ["Model", *[f"k = {k} (hours)" for k in PROJECTION_K]], + [ + [ + m["key"], + *[ + _n(None if v is None else v / 3600, 2) + for v in m["metrics"]["projection_seconds"].values() + ], + ] + for m in models + ], + ) + lines += [ + "", + f"Projection = mean wall time per attempt here x {PROJECTION_SCENARIOS} x k, one model", + "at a time on this machine; it assumes the test scenarios are like these dev scenarios", + "in size.", + "", + "## 6. Licence checks", + "", + ] + for m in models: + check = m["licence_check"] + lines.append( + f"- {m['key']}: {m['pin']['licence']}. " + + "; ".join(f"{k}: {v}" for k, v in sorted(check.items())) + ) + lines.append("") + return "\n".join(lines) diff --git a/experiments/task_text.py b/experiments/task_text.py new file mode 100644 index 0000000..308e33a --- /dev/null +++ b/experiments/task_text.py @@ -0,0 +1,31 @@ +"""The repair task given to the model (P10a decision 4, C-219). + +"The task given to the model is a fixed template per target invariant: "Repair ````: +````. Keep every other protected invariant +satisfied. Make the smallest change."" The statement is the ``description`` of DAIL's own registry +definition (``core.domain.registry_v1_1``), read here from a DAIL ``Invariant`` and nothing else. +TerraPreserve's ``target.description`` and ``intent`` are never read (C-219). + +This module sits outside ``experiments/generation/`` because the generation package may not name +``description`` at all (the blinding contract, check (c)): the name there could only mean +TerraPreserve's field. +""" + +from __future__ import annotations + +from typing import Final + +from core.domain.invariant import Invariant + +TASK_TEMPLATE: Final = ( + "Repair {invariant_id}: {statement} Keep every other protected invariant satisfied. " + "Make the smallest change." +) + + +def objective_for(definition: Invariant) -> str: + if not isinstance(definition, Invariant): + raise TypeError("objective_for needs a DAIL Invariant definition") + return TASK_TEMPLATE.format( + invariant_id=definition.invariant_id, statement=definition.description + ) diff --git a/llm/adapter/ollama.py b/llm/adapter/ollama.py index 92fc7ad..a046d8d 100644 --- a/llm/adapter/ollama.py +++ b/llm/adapter/ollama.py @@ -10,7 +10,8 @@ * ``POST /api/chat`` with ``model``, ``messages`` (system, user), ``stream: false``, ``format`` (the JSON schema of ``LLMRepairResponse``: "Format can be ``json`` or a JSON schema") and ``options`` - (``temperature``, ``top_p``, ``seed``, ``num_predict``). The reply is one object with + (``temperature``, ``top_p``, ``seed``, ``num_predict``, and ``num_ctx`` when the configuration + sets it, P10a). The reply is one object with ``message.content``, ``done``, ``prompt_eval_count`` and ``eval_count``. * ``GET /api/tags`` (``name``, ``digest``, ``details.quantization_level``) and ``GET /api/version``. * Errors are ``{"error": ""}`` with 400, 404, 429, 500 or 502. @@ -145,6 +146,8 @@ def generate(self, call: ProviderCall) -> ProviderReply: } if config.seed is not None: options["seed"] = config.seed + if config.num_ctx is not None: + options["num_ctx"] = config.num_ctx body = { "model": config.model_name, "messages": [ diff --git a/llm/schemas/records.py b/llm/schemas/records.py index a9a9974..e8a5949 100644 --- a/llm/schemas/records.py +++ b/llm/schemas/records.py @@ -1,8 +1,10 @@ """Configuration, policy and record types of the LLM channel (Doc 10 §16-§18, §20-§24, §32, §34). * ``FailureClass`` is §24; ``RETRYABLE`` is the ``retry_on`` set §23 allows. -* ``LLMConfig`` is §16 plus ``seed`` (C-182 pins every experiment model by temperature and seed). - All of it is recorded on every request (§17). +* ``LLMConfig`` is §16 plus ``seed`` (C-182 pins every experiment model by temperature and seed) + and, optionally, ``num_ctx``: the model's context window as the local server is asked to use it + (P10a; P7a verdict limit 5). All of it is recorded on every request (§17); ``num_ctx`` only when + set, so the records of a configuration without it are unchanged. * ``RetryPolicy`` is §23; ``max_attempts`` is bounded (1 to 10): "Never retry indefinitely." * ``BudgetConfig`` is §18. * ``LLMRequest`` is §20 plus the attempt number, the trusted-state hash (§34 "Persist fixture/state @@ -96,8 +98,11 @@ class LLMConfig: retry_policy_id: str prompt_version: str response_schema_version: str + num_ctx: int | None = None # P10a: the context window, when the model needs it set def __post_init__(self) -> None: + if self.num_ctx is not None: + _integer(self.num_ctx, "LLMConfig.num_ctx", 1) for name in ( "provider", "model_name", @@ -121,7 +126,9 @@ def __post_init__(self) -> None: ) def to_dict(self) -> dict[str, Any]: + extra = {} if self.num_ctx is None else {"num_ctx": self.num_ctx} return { + **extra, "provider": self.provider, "model_name": self.model_name, "model_version": self.model_version, diff --git a/llm/validation/patch.py b/llm/validation/patch.py index 9e2d1a7..6bbbfe1 100644 --- a/llm/validation/patch.py +++ b/llm/validation/patch.py @@ -12,7 +12,8 @@ (a limit of the parser, C-93) UNSUPPORTED 3 only ``resource`` blocks at the top level (with two labels) POLICY_VIOLATION 4 ``provisioner``, ``connection`` inside a block; a ``lifecycle`` that is not POLICY_VIOLATION - the parent's (whitespace-normalized); ``dynamic`` UNSUPPORTED + the parent's (whitespace-normalized), including a MODIFY block that drops + the parent's ``lifecycle`` (C-221, validator 2); ``dynamic`` UNSUPPORTED 5 boundaries (C-188, C-189): blocks and ``changed_addresses`` agree, no INVALID_OUTPUT block twice, no DELETE of an address the parent does not have 6 scope (C-190): type outside the supported model UNSUPPORTED @@ -77,7 +78,7 @@ from llm.schemas.records import FailureClass from llm.schemas.repair_response import LLMRepairResponse, RepairStatus -VALIDATOR_VERSION: Final = "dail-patch-validator-1" +VALIDATOR_VERSION: Final = "dail-patch-validator-2" # 2: the lifecycle rule (C-221) PATCH_FILE: Final = "dail_llm_patch.tf" F = FailureClass OP = OperationType @@ -318,10 +319,14 @@ def _constructs(self, blocks: Sequence[Block], data: bytes) -> None: for i in block.body.items if isinstance(i, Block) and i.type == "lifecycle" ] - if lifecycles and lifecycles != self._parent_lifecycles(_address(block)): + # compared as in validator 1; since validator 2 (C-221) a block for an address whose + # parent block has a lifecycle block must also contain it unchanged + parent = self._parent_lifecycles(_address(block)) + if (lifecycles or parent) and lifecycles != parent: raise _Refuse( F.POLICY_VIOLATION, - f"{_address(block)}: a lifecycle block is allowed only as the parent's", + f"{_address(block)}: a lifecycle block is allowed only as the parent's, and a " + "MODIFY must keep the parent's lifecycle block", ) def _parent_decl(self, address: str) -> ResourceDecl | None: diff --git a/tests/contract/test_experiment_blinding.py b/tests/contract/test_experiment_blinding.py index a7bb989..2e11374 100644 --- a/tests/contract/test_experiment_blinding.py +++ b/tests/contract/test_experiment_blinding.py @@ -18,15 +18,16 @@ longer fail is caught. P10a (work package Section 3.3): "Extend the blinding contract test: ``experiments/generation/`` -names no label field and no ``description``, ``intent``, ``mutation_record`` or later step, and never -opens ``step_1`` or beyond. Show the test failing on a mutant." (c) Every module of +names no label field and no ``description``, ``intent``, ``mutation_record`` or later step, and +never opens ``step_1`` or beyond. Show the test failing on a mutant." (c) Every module of ``experiments/generation/`` is checked for the label fields and for ``description`` (TerraPreserve's ``target.description``); a later step is any string naming ``step_`` with n >= 1, and any use of a scenario's ``steps`` other than ``steps[0]`` or ``steps[:1]`` (an index, a loop, a slice past S0). The invariant's own registry statement reaches the task through ``experiments.task_text``, which reads a DAIL ``Invariant`` only. (d) No generation module imports ``experiments.scoring``, -``.metrics`` or ``.report``. A dynamic check is in ``tests.unit.test_p10a_generation.TestNoLaterStep``: -generation, replay and export run with the later step directories deleted. +``.metrics`` or ``.report``. A dynamic check is in +``tests.unit.test_p10a_generation.TestNoLaterStep``: generation, replay and export run with the +later step directories deleted. """ from __future__ import annotations diff --git a/tests/unit/test_p10a_extractor_version.py b/tests/unit/test_p10a_extractor_version.py index 6d12f72..72510e9 100644 --- a/tests/unit/test_p10a_extractor_version.py +++ b/tests/unit/test_p10a_extractor_version.py @@ -1,10 +1,10 @@ """The extractor version, apart from the metric definitions version (P10a work package Section 3.1; closes the P8b verdict's limit 2). -Section 3.1: "``EXTRACTOR_VERSION`` becomes its own value (``extractor-2``), separate from the metric -definitions version (``metrics-draft-1``). Every ``MetricRecord`` and report carries both. A test -fails if the extractor's code hash changes without a version change (pin the hash of the extractor -modules in a test)." +Section 3.1: "``EXTRACTOR_VERSION`` becomes its own value (``extractor-2``), separate from the +metric definitions version (``metrics-draft-1``). Every ``MetricRecord`` and report carries both. A +test fails if the extractor's code hash changes without a version change (pin the hash of the +extractor modules in a test)." The extractor modules are the code that turns stored records into metric records: ``experiments/metrics/`` and the scoring it reads truth through (``experiments/scoring/``, with its @@ -24,7 +24,7 @@ from tests.metrics_builders import GATES, scenarios PINNED: dict[str, str] = { - "extractor-2": "7a0fec10865e69672727ac801751520d835097ae078388c553f325457b9a1fc9", + "extractor-2": "5b4a598940e59fa3d2b3fe1b48360135d174db65124df7b7830757800e80a25b", } EXTRACTOR_PACKAGES = ("experiments/metrics", "experiments/scoring") CRLF, LF = b"\r\n", b"\n" diff --git a/tests/unit/test_p10a_generation.py b/tests/unit/test_p10a_generation.py index 8686a3d..1fc32b4 100644 --- a/tests/unit/test_p10a_generation.py +++ b/tests/unit/test_p10a_generation.py @@ -1,11 +1,12 @@ -"""LLM candidates for the experiment (P10a work package Section 3.3; decisions 3 to 5, C-218 to C-220). +"""LLM candidates for the experiment (P10a work package Section 3.3; decisions 3 to 5, C-218 to +C-220). Expected values come from the quoted text: Decision 4: "repair scenarios only (``target.kind = "repair"``) ... The task given to the model is a -fixed template per target invariant: "Repair ````: ````. Keep every other protected invariant satisfied. Make the smallest change." -TerraPreserve's ``target.description`` and ``intent`` are never given to the model." +fixed template per target invariant: "Repair ````: ````. Keep every other protected invariant satisfied. Make the smallest +change." TerraPreserve's ``target.description`` and ``intent`` are never given to the model." Decision 5: "``AllowedChangeScope`` = every resource address of S0, every supported type, no attribute restriction, and all four operations. It is the same for every condition and is recorded with each candidate." @@ -165,11 +166,13 @@ def test_the_task_is_the_fixed_template(self) -> None: from core.domain.registry_v1_1 import registry_v1_1_invariants from experiments.generation.task import objective_for - (definition,) = [ + (definition,) = ( d - for d in registry_v1_1_invariants(datetime(2026, 1, 1, tzinfo=UTC), source="a.s", destination="b.d") + for d in registry_v1_1_invariants( + datetime(2026, 1, 1, tzinfo=UTC), source="a.s", destination="b.d" + ) if d.invariant_id == "INV-SEC-001" - ] + ) self.assertEqual( objective_for(definition), f"Repair INV-SEC-001: {definition.description} Keep every other protected invariant " @@ -220,9 +223,10 @@ def test_a_model_whose_installed_digest_is_not_the_pinned_one_is_refused(self) - with self.assertRaises(ModelRefused): self.generate([], provider=provider, models=[(pin(), provider, installed("c" * 64))]) self.assertEqual(provider.calls, []) - self.assertFalse((self.out / "candidates").exists() and any( - p.name == "attempt.json" for p in (self.out / "candidates").rglob("*") - )) + self.assertFalse( + (self.out / "candidates").exists() + and any(p.name == "attempt.json" for p in (self.out / "candidates").rglob("*")) + ) def test_a_missing_model_is_refused(self) -> None: from experiments.generation.models import ModelRefused, check_installed @@ -255,7 +259,9 @@ def test_each_attempt_and_the_manifest_are_stored_with_hashes(self) -> None: self.assertEqual(hashlib.sha256((base / name).read_bytes()).hexdigest(), digest) # the request and response records of every attempt, retries included (P7's records) records = sorted(p.name for p in (self.attempt_dir(summary, F4) / "records").iterdir()) - self.assertEqual(records, ["request-1.json", "request-2.json", "response-1.json", "response-2.json"]) + self.assertEqual( + records, ["request-1.json", "request-2.json", "response-1.json", "response-2.json"] + ) def test_a_frozen_generation_is_never_regenerated(self) -> None: from experiments.generation.run import GenerationError @@ -288,9 +294,12 @@ def test_generation_needs_the_switch(self) -> None: def test_the_switch_is_on_only_in_the_reviewed_copy(self) -> None: from experiments.generation.run import EXPERIMENT_LLM_FILE, generation_config - self.assertIs(load_config(REPO_ROOT / "config", "experiment", environ={}).get( - "llm.generation_enabled" - ), False) + self.assertIs( + load_config(REPO_ROOT / "config", "experiment", environ={}).get( + "llm.generation_enabled" + ), + False, + ) self.assertIs(generation_config().get("llm.generation_enabled"), True) self.assertEqual(EXPERIMENT_LLM_FILE, REPO_ROOT / "experiments" / "config" / "llm.toml") summary = self.generate([ok(VALID)]) @@ -321,7 +330,9 @@ def test_an_edited_frozen_patch_is_refused_without_a_decision(self) -> None: summary = self.generate([ok(VALID)]) responses = self.attempt_dir(summary) / "records" / "response-1.json" record = json.loads(responses.read_text("utf-8")) - record["raw_response"] = record["raw_response"].replace('name = \\"app\\"', 'name = \\"app2\\"') + record["raw_response"] = record["raw_response"].replace( + 'name = \\"app\\"', 'name = \\"app2\\"' + ) responses.chmod(0o666) responses.write_text(json.dumps(record), "utf-8") replayed = self.replay(summary) @@ -364,9 +375,12 @@ def test_the_export_holds_only_the_built_tf_files_and_its_index(self) -> None: self.assertTrue(all(n.endswith(".tf") for n in names), names) self.assertEqual(sorted(entry["files"]), names) self.assertEqual(entry["generation_id"], summary.generation_id) - self.assertEqual(entry["tree_hash"], json.loads( - (self.attempt_dir(summary) / "attempt.json").read_text("utf-8") - )["candidate_tree_hash"]) + self.assertEqual( + entry["tree_hash"], + json.loads((self.attempt_dir(summary) / "attempt.json").read_text("utf-8"))[ + "candidate_tree_hash" + ], + ) main = (directory / "main.tf").read_text("utf-8") self.assertNotIn("app_ssh_public", main) @@ -432,9 +446,9 @@ def test_an_unsupported_provider_answer_is_kept_apart_from_model_failures(self) self.assertEqual(row["failure_source"], "provider") summary2 = self.generate( [ok(repair('resource "aws_lambda_function" "f" {\n}\n', ["aws_lambda_function.f"]))], - spec=__import__("experiments.generation.run", fromlist=["GenerationSpec"]).GenerationSpec( - name="p10a-test", version="2" - ), + spec=__import__( + "experiments.generation.run", fromlist=["GenerationSpec"] + ).GenerationSpec(name="p10a-test", version="2"), ) (row2,) = summary2.attempts self.assertEqual(row2["failure_class"], "UNSUPPORTED") diff --git a/tests/unit/test_p10a_generation_metrics.py b/tests/unit/test_p10a_generation_metrics.py new file mode 100644 index 0000000..e08f138 --- /dev/null +++ b/tests/unit/test_p10a_generation_metrics.py @@ -0,0 +1,145 @@ +"""Generation metrics and the pilot report (P10a Section 3.3 "Generation metrics", Step 6, Step 7). + +Section 3.3: "valid patch rate per model (Doc 13 §26), failure classes per model (with Ollama +UNSUPPORTED kept apart from model failures, P7a limit 4), retries, latency, and token counts (marked +as estimates where they come from ``ceil(characters / 4)``, P7a limit 5)." Step 6 "Report per +model: latency per call (mean, max); valid patch rate and failure classes; DAIL decisions under C; +total wall time; a projection of the time for 36 test repair scenarios x k in {1, 3, 5}." + +The record set below is built by hand; every expected value is computed by hand in the comments. +""" + +from __future__ import annotations + +import json +import unittest +from typing import Any + +from experiments.metrics.generation import model_metrics +from experiments.report.pilot import render_pilot +from tests.fixture_catalog import REPO_ROOT + + +def attempt(sid: str, status: str, failure: str | None, source: str | None, estimate: int) -> Any: + return { + "model_key": "m", + "scenario_id": sid, + "k": 1, + "status": status, + "failure_class": failure, + "failure_source": source, + "observability": {"budget": {"estimated_input_tokens": estimate}}, + } + + +def response(latency: float | None, tokens: dict[str, int]) -> dict[str, Any]: + return {"latency_ms": latency, "token_usage": tokens} + + +ATTEMPTS = [ + attempt("s1", "PATCH", None, None, 900), + attempt("s2", "FAILED", "INVALID_OUTPUT", "model", 1000), + attempt("s3", "FAILED", "UNSUPPORTED", "provider", 800), + attempt("s4", "CONTEXT_INSUFFICIENT", "CONTEXT_INSUFFICIENT", "budget", 5000), + {**attempt("s1", "PATCH", None, None, 1), "model_key": "other"}, +] +RESPONSES = { + ("m", "s1", 1): [response(10_000.0, {"input": 1000, "output": 100})], + ("m", "s2", 1): [ + response(20_000.0, {"input": 2000, "output": 200}), + response(30_000.0, {"input": 2100, "output": 210}), + ], + ("m", "s3", 1): [response(None, {})], + ("m", "s4", 1): [], + ("other", "s1", 1): [response(1.0, {"input": 1, "output": 1})], +} +TIMINGS = { + ("m", "s1", 1): {"wall_seconds": 12.0}, + ("m", "s2", 1): {"wall_seconds": 55.0}, + ("m", "s3", 1): {"wall_seconds": 0.5}, + ("m", "s4", 1): {"wall_seconds": 0.1}, + ("other", "s1", 1): {"wall_seconds": 1.0}, +} +REPLAYS = [ + {"model_key": "m", "condition": "C", "decision": "PROMOTE"}, + {"model_key": "m", "condition": "C", "decision": None}, + {"model_key": "m", "condition": "A", "decision": "PROMOTE"}, + {"model_key": "other", "condition": "C", "decision": "REJECT"}, +] + + +class TestModelMetrics(unittest.TestCase): + def setUp(self) -> None: + self.m = model_metrics(ATTEMPTS, RESPONSES, TIMINGS, REPLAYS)["m"] + + def test_valid_patch_rate(self) -> None: + rate = self.m["valid_patch_rate"] + self.assertEqual((rate["numerator"], rate["denominator"]), (1, 4)) # s1 only + self.assertEqual(rate["value"], 0.25) + self.assertEqual(self.m["attempts"], 4) + + def test_failure_classes_by_source(self) -> None: + self.assertEqual( + self.m["failure_classes"], + { + "provider": {"UNSUPPORTED": 1}, # Ollama's 404: not the model's failure + "budget": {"CONTEXT_INSUFFICIENT": 1}, + "model": {"INVALID_OUTPUT": 1}, + }, + ) + + def test_calls_and_retries(self) -> None: + self.assertEqual(self.m["calls"], 4) # 1 + 2 + 1 + 0 + self.assertEqual(self.m["retries"], 1) # 4 calls - 3 attempts that made a call + + def test_latency_per_call(self) -> None: + # calls with a reply: 10 000, 20 000, 30 000 ms + self.assertEqual(self.m["latency_ms"]["mean"], 20_000.0) + self.assertEqual(self.m["latency_ms"]["max"], 30_000.0) + self.assertEqual(self.m["latency_ms"]["calls_with_a_reply"], 3) + + def test_tokens_server_counts_and_the_estimate_apart(self) -> None: + tokens = self.m["tokens"] + self.assertEqual(tokens["provider_input_total"], 5100) # 1000 + 2000 + 2100 + self.assertEqual(tokens["provider_output_total"], 510) + self.assertEqual(tokens["provider_input_mean_per_call"], 1700.0) + self.assertEqual(tokens["estimated_input_mean"], 1925.0) # (900+1000+800+5000)/4 + self.assertEqual(tokens["estimated_input_max"], 5000) + self.assertIn("estimate: ceil(characters / 4)", tokens["estimate_rule"]) + + def test_wall_time_and_projection(self) -> None: + wall = self.m["wall_seconds"] + self.assertAlmostEqual(wall["total"], 67.6) # 12 + 55 + 0.5 + 0.1 + self.assertAlmostEqual(wall["mean_per_attempt"], 16.9) + projection = self.m["projection_seconds"] + self.assertAlmostEqual(projection["36 scenarios x k=1"], 16.9 * 36) + self.assertAlmostEqual(projection["36 scenarios x k=3"], 16.9 * 36 * 3) + self.assertAlmostEqual(projection["36 scenarios x k=5"], 16.9 * 36 * 5) + + def test_decisions_per_condition(self) -> None: + self.assertEqual( + self.m["decisions"], {"A": {"PROMOTE": 1}, "C": {"PROMOTE": 1, "NO DECISION": 1}} + ) + + def test_versions(self) -> None: + self.assertEqual(self.m["metrics_version"], "metrics-draft-1") + self.assertEqual(self.m["extractor_version"], "extractor-2") + + +class TestPilotDocument(unittest.TestCase): + def test_the_document_is_the_rendering_of_its_json(self) -> None: + doc = REPO_ROOT / "docs" / "P10A_MODEL_PILOT.md" + data = REPO_ROOT / "docs" / "P10A_MODEL_PILOT.json" + report = json.loads(data.read_text(encoding="utf-8")) + self.assertEqual( + doc.read_text(encoding="utf-8").replace("\r\n", "\n"), render_pilot(report) + ) + self.assertIn("no model's safety is", render_pilot(report)) + self.assertGreaterEqual(len(report["models"]), 1) + for model in report["models"]: + self.assertTrue(model["licence_check"], model["key"]) + self.assertIn("C", model["metrics"]["decisions"]) + + +if __name__ == "__main__": + unittest.main() From 34f0153e09c403bc69f247d852b5be6582c72981 Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Fri, 13 Nov 2026 20:51:14 +0530 Subject: [PATCH 5/7] P10a step 5-6: the opt-in wiring run test; pins of the three pilot models (qwen2.5-coder:7b, granite3.3:8b, phi4-mini:3.8b) with digests and licence checks (C-182, C-218) Co-Authored-By: Claude Opus 5.5 --- experiments/config/models.toml | 44 ++++++++++++ tests/unit/test_p10a_wiring.py | 121 +++++++++++++++++++++++++++++++++ 2 files changed, 165 insertions(+) create mode 100644 tests/unit/test_p10a_wiring.py diff --git a/experiments/config/models.toml b/experiments/config/models.toml index 858d301..c3da7d3 100644 --- a/experiments/config/models.toml +++ b/experiments/config/models.toml @@ -18,3 +18,47 @@ seed = 20261113 max_output_tokens = 2048 timeout_seconds = 900.0 num_ctx = 16384 + +# The dev model pilot (P10a Step 6), checked on 13 Nov 2026. Three publishers; each licence is the +# model's own "license" layer on its Ollama library page and the text `ollama show --license` printed. +[model.qwen2_5-coder-7b] +tag = "qwen2.5-coder:7b" +digest = "dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364" +quantization = "Q4_K_M" +size = "4.7 GB (7.6B parameters; context length 32768)" +licence = "Apache-2.0" +licence_source = "ollama.com/library/qwen2.5-coder:7b, license blob 832dd9e00a68 (Apache License 2.0), and ollama show --license (13 Nov 2026)" +temperature = 0.0 +top_p = 1.0 +seed = 20261113 +max_output_tokens = 2048 +timeout_seconds = 1800.0 +num_ctx = 16384 + +[model.granite3_3-8b] +tag = "granite3.3:8b" +digest = "fd429f23b90980ed1bef53b990894e7b0199331f6ae90c5650240a7d5b70f1f7" +quantization = "Q4_K_M" +size = "4.9 GB (8.2B parameters; context length 131072)" +licence = "Apache-2.0" +licence_source = "ollama.com/library/granite3.3:8b, license blob 4a99a6dd617d (Apache License 2.0), and ollama show --license (13 Nov 2026)" +temperature = 0.0 +top_p = 1.0 +seed = 20261113 +max_output_tokens = 2048 +timeout_seconds = 1800.0 +num_ctx = 16384 + +[model.phi4-mini-3_8b] +tag = "phi4-mini:3.8b" +digest = "78fad5d182a7c33065e153a5f8ba210754207ba9d91973f57dffa7f487363753" +quantization = "Q4_K_M" +size = "2.5 GB (3.8B parameters; context length 131072)" +licence = "MIT" +licence_source = "ollama.com/library/phi4-mini:3.8b, license blob fa8235e5b48f (MIT License, Microsoft Corporation), and ollama show --license (13 Nov 2026)" +temperature = 0.0 +top_p = 1.0 +seed = 20261113 +max_output_tokens = 2048 +timeout_seconds = 1800.0 +num_ctx = 16384 diff --git a/tests/unit/test_p10a_wiring.py b/tests/unit/test_p10a_wiring.py new file mode 100644 index 0000000..961f824 --- /dev/null +++ b/tests/unit/test_p10a_wiring.py @@ -0,0 +1,121 @@ +"""The P10a wiring run (work package Step 5): opt-in, the author's machine only. + +"Wiring run (opt-in test): one dev repair scenario, the P7 wiring model, k = 1, end to end +(generate, freeze, replay under four conditions, export)." + +Environment: ``DAIL_OLLAMA_TEST=1``, ``TERRAPRESERVE_ROOT`` (a checkout at ``terrapreserve-v1.0``, +dev split only) and the local Ollama server with the pinned wiring model +(``experiments/config/models.toml``, key ``qwen2_5-coder-1_5b``). Output goes to a temporary +directory. The model is not judged: an invalid patch is a valid outcome of a wiring run, and then +replay has no decision to make (the test says which happened). +""" + +from __future__ import annotations + +import json +import os +import tempfile +import unittest +from pathlib import Path + +WIRING_MODEL = "qwen2_5-coder-1_5b" + + +@unittest.skipUnless( + os.environ.get("DAIL_OLLAMA_TEST") == "1" and os.environ.get("TERRAPRESERVE_ROOT"), + "opt-in: DAIL_OLLAMA_TEST=1 and TERRAPRESERVE_ROOT", +) +class TestWiringRun(unittest.TestCase): + def test_generate_freeze_replay_export(self) -> None: + from experiments.generation.__main__ import choose + from experiments.generation.export import export_for_oracle + from experiments.generation.models import load_pins + from experiments.generation.replay import replay_candidates + from experiments.generation.run import ( + GenerationSpec, + generate_candidates, + generation_config, + ) + from experiments.loader import load + from experiments.runner import experiment_config + from llm.adapter.ollama import OllamaProvider + + loaded = load(Path(os.environ["TERRAPRESERVE_ROOT"])) + (given,) = choose(loaded.scenarios, only=None, per_family=1, families=("F2",)) + pin = load_pins()[WIRING_MODEL] + provider = OllamaProvider(expected_digest=pin.digest) + with tempfile.TemporaryDirectory(prefix="dail-p10a-wiring-") as tmp: + root = Path(tmp) + summary = generate_candidates( + spec=GenerationSpec(name="p10a-wiring", version="1"), + inputs=[given], + models=[(pin, provider, provider.model_facts(pin.tag))], + k=1, + root=root, + generation_enabled=generation_config().get("llm.generation_enabled") is True, + environment={"ollama_version": provider.server_version()}, + ) + replay = replay_candidates( + root=root, + generation_id=summary.generation_id, + scenarios={s.scenario_id: s for s in loaded.scenarios}, + conditions=("A", "B", "C", "C_PRIME"), + config=experiment_config(), + replay_name="wiring", + ) + export = json.loads( + export_for_oracle( + root=root, generation_id=summary.generation_id, inputs=[given] + ).read_bytes() + ) + (row,) = summary.attempts + attempt = json.loads( + ( + root / "candidates" / summary.generation_id / pin.key / given.scenario_id + / "1" / "attempt.json" + ).read_bytes() + ) # fmt: skip + facts = { + "scenario_id": given.scenario_id, + "target": given.target_invariant_id, + "generation_id": summary.generation_id, + "manifest_sha256": summary.manifest_sha256, + "status": row["status"], + "failure_class": row["failure_class"], + "failure_source": row["failure_source"], + "parse_errors": attempt["parse_errors"], + "validation_errors": attempt["validation_errors"], + "latency_ms": attempt["observability"]["latency_ms"], + "token_usage": attempt["observability"]["token_usage"], + "estimated_input_tokens": attempt["observability"]["budget"].get( + "estimated_input_tokens" + ), + "replay": [ + { + "condition": r["condition"], + "tree_hash_matches": r["tree_hash_matches"], + "decision": r["decision"], + "build_deleted": r["build_dir"] is None + or not Path(r["build_dir"]).exists(), + } + for r in replay.results + ], + "generation_failures": replay.generation_failures, + "exported": [ + {"candidate_id": c["candidate_id"], "files": sorted(c["files"])} + for c in export["candidates"] + ], + "export_refused": export["refused"], + } + print("\nWIRING-RUN-P10A " + json.dumps(facts, sort_keys=True)) + if row["status"] == "PATCH": + self.assertEqual(len(replay.results), 4) + self.assertTrue(all(r["tree_hash_matches"] for r in replay.results)) + self.assertEqual(len(export["candidates"]), 1) + else: + self.assertEqual(replay.results, []) + self.assertEqual(export["candidates"], []) + + +if __name__ == "__main__": + unittest.main() From 29007c0770073656b4fba67f81c1a0f4e02e5151 Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Sat, 14 Nov 2026 10:35:10 +0530 Subject: [PATCH 6/7] =?UTF-8?q?P10a=20step=206-7:=20the=20dev=20model=20pi?= =?UTF-8?q?lot=20report=20docs/P10A=5FMODEL=5FPILOT=20(generated=20from=20?= =?UTF-8?q?the=20frozen=20records),=20licence=20checks,=20PHASE=5FGATES=20?= =?UTF-8?q?P10a=20section,=20BUILD=5FSEQUENCE=20row=2024,=20C-222;=20the?= =?UTF-8?q?=20pilot=20renderer=20sorts=20decision=20counts=20(Doc=2013=20?= =?UTF-8?q?=C2=A726,=20=C2=A733;=20C-218=20to=20C-222)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- docs/BUILD_SEQUENCE.md | 3 +- docs/DECISIONS_REGISTER.md | 1 + docs/P10A_LICENCE_CHECKS.json | 14 ++ docs/P10A_MODEL_PILOT.json | 387 ++++++++++++++++++++++++++++++++++ docs/P10A_MODEL_PILOT.md | 79 +++++++ docs/PHASE_GATES.md | 50 +++++ experiments/report/pilot.py | 4 +- 7 files changed, 536 insertions(+), 2 deletions(-) create mode 100644 docs/P10A_LICENCE_CHECKS.json create mode 100644 docs/P10A_MODEL_PILOT.json create mode 100644 docs/P10A_MODEL_PILOT.md diff --git a/docs/BUILD_SEQUENCE.md b/docs/BUILD_SEQUENCE.md index 37fe9f0..fa43e56 100644 --- a/docs/BUILD_SEQUENCE.md +++ b/docs/BUILD_SEQUENCE.md @@ -33,7 +33,8 @@ The build is split into 18 work packages (N-06 in `DECISIONS_REGISTER.md`). Ever | 21 | — | ~~P9 AWS integration (free route) → M6~~ removed by C-71 (advisor, 11 Nov 2026): AWS stays fully offline, no deployment, no AWS account, no live calibration; milestone M6 (AWS Demonstration) is removed | | 22 | Dec 12–14 (built 12 Nov 2026) | P8a experiment harness: conditions A, B, C, C′ (C-198), open targets (C-196), the TerraPreserve loader with hash pins and label blinding, trials, records and artifacts, the dev-split harness check (C-196 to C-208) | | 23 | Dec 15–18 (built 13 Nov 2026) | P8b metrics and reports: scoring (the only label reader), the extractor metrics-draft-1, the ExperimentReport, docs/P8_DEV_PILOT.md (C-212 to C-215) → P8 gate, M5 | -| 24–28 | Dec 19–Jan 2 | P10 hardening, protocol freeze + pilot, LLM patch generation, main run, QA report | +| 24 | (built 13 Nov 2026) | P10a LLM candidates for the experiment: experiments/generation (generate, freeze, replay, oracle export), validator 2 (lifecycle, C-221), extractor-2, the dev model pilot docs/P10A_MODEL_PILOT.md (C-218 to C-222) | +| 25–28 | Dec 19–Jan 2 | P10b protocol freeze; P10c main run on the test split; P10d statistics, QA report, release gates | | 29 | Jan 3–4 | TerraPreserve v1.1 (real LLM patches) | Critical path: P1a → P6b. diff --git a/docs/DECISIONS_REGISTER.md b/docs/DECISIONS_REGISTER.md index 673e7e4..a15f257 100644 --- a/docs/DECISIONS_REGISTER.md +++ b/docs/DECISIONS_REGISTER.md @@ -233,3 +233,4 @@ This register records every conflict found between the 15 `CloudSpartanX_*` spec | C-219 | P10a: which scenarios get LLM candidates, and the task | Repair scenarios only (`target.kind = "repair"`: F2, F4, F8). The task given to the model is a fixed template per target invariant: "Repair ``: ``. Keep every other protected invariant satisfied. Make the smallest change." TerraPreserve's `target.description` and `intent` are never given to the model: the change-request descriptions name the damage that makes a scenario unsafe. | Decided (authors, 13 Nov 2026) | | C-220 | P10a: the scope of an LLM patch (closes P7b verdict limit 7) | For every scenario, `AllowedChangeScope` = every resource address of S0, every supported type, no attribute restriction, and all four operations. It is the same for every condition and is recorded with each candidate. DAIL alone judges safety, so the scope does not filter unsafe edits. | Decided (authors, 13 Nov 2026) | | C-221 | P10a: a patch that drops the parent's `lifecycle` block (closes P7b verdict limit 4) | A patch that drops the parent's `lifecycle` block is refused: POLICY_VIOLATION before DAIL, counted as an invalid patch. Neither DAIL nor the registry models lifecycle settings. Built in `llm/validation/patch.py`, validator `dail-patch-validator-2`. | Decided (authors, 13 Nov 2026) | +| C-222 | P10a: readings made while building generation | (1) **Focus** (Section 3.3 "focus = the target invariant's scope resources"): the S0 resources the target definition's `scope` names by address or by type; nothing is walked from them (S0 has no impact report). (2) **Scope addresses** are configuration addresses (`type.name`, an instance key dropped), as `AllowedChangeScope` requires (C-189). (3) **Configuration-only parent**: the top-level `*.tf` and `*.tf.json` files of S0; the export holds the built `.tf` files and refuses a candidate whose build holds any other file. (4) **Retry policy** `p10a-retry-2`: at most two calls, again only after TRANSIENT_ERROR or INVALID_OUTPUT (at temperature 0 with a fixed seed a NO_SOLUTION would come back unchanged). The seed of repetition k is the pinned seed + k - 1. P10b may change both. (5) **num_ctx**: `LLMConfig.num_ctx` (optional, recorded only when set, so P7 records are unchanged) is sent as Ollama's `num_ctx` option; every experiment pin sets it, and the budget uses it as `max_context_tokens` with `max_output_tokens` reserved and a margin of 256, so an oversized prompt is CONTEXT_INSUFFICIENT before any call (P7a verdict limit 5). (6) **Failure source**: provider when there was no reply, budget for CONTEXT_INSUFFICIENT, model otherwise (P7a verdict limit 4). (7) **Replay** reuses the harness's trial attempt (`experiments.trial._Attempt`) for the condition's S0 baseline, with the scenario cut to its first step; a scope that differs from the frozen one also gets no decision. (8) **The registry statement** reaches the task through `experiments/task_text.py`, outside the generation package, because the blinding check forbids the name `description` there. (9) **Extractor pin**: the hash covers `experiments/metrics` and `experiments/scoring` (`.py` and `.json`, CRLF read as LF). | Proposed (P10a) | diff --git a/docs/P10A_LICENCE_CHECKS.json b/docs/P10A_LICENCE_CHECKS.json new file mode 100644 index 0000000..39291a8 --- /dev/null +++ b/docs/P10A_LICENCE_CHECKS.json @@ -0,0 +1,14 @@ +{ + "granite3_3-8b": { + "library page": "ollama.com/library/granite3.3:8b: publisher IBM Research; licence Apache 2.0; license layer 4a99a6dd617d (Apache License 2.0)", + "ollama show --license": "Apache License, Version 2.0, January 2004 (13 Nov 2026)" + }, + "phi4-mini-3_8b": { + "library page": "ollama.com/library/phi4-mini:3.8b: references Microsoft Phi-4-mini-instruct; license layer fa8235e5b48f (MIT License, Copyright (c) Microsoft Corporation); the page text names no licence outside that layer", + "ollama show --license": "Microsoft. Copyright (c) Microsoft Corporation. MIT License (13 Nov 2026)" + }, + "qwen2_5-coder-7b": { + "library page": "ollama.com/library/qwen2.5-coder:7b: license layer 832dd9e00a68 (Apache License 2.0); the tags page names no licence", + "ollama show --license": "Apache License, Version 2.0, January 2004 (13 Nov 2026)" + } +} diff --git a/docs/P10A_MODEL_PILOT.json b/docs/P10A_MODEL_PILOT.json new file mode 100644 index 0000000..212b6b8 --- /dev/null +++ b/docs/P10A_MODEL_PILOT.json @@ -0,0 +1,387 @@ +{ + "candidates_manifest_sha256": "de1bb7b33de2f13799c22ac9be374c3321793a661719005a6587b1f5cb077937", + "conditions": [ + "A", + "B", + "C", + "C_PRIME" + ], + "environment": { + "dail_commit": "34f0153e09c403bc69f247d852b5be6582c72981", + "dail_worktree_clean": "true", + "ollama_version": "0.40.16", + "terrapreserve_commit": "27f0ee4010852546e231fb24a35cd03ba4cc5772", + "terrapreserve_tag": "terrapreserve-v1.0" + }, + "extractor_version": "extractor-2", + "generation_id": "d4507101-113d-5753-806d-593ee99b3255", + "generation_name": "p10a-dev-pilot", + "generation_version": "1", + "metrics_version": "metrics-draft-1", + "models": [ + { + "installed": { + "digest": "fd429f23b90980ed1bef53b990894e7b0199331f6ae90c5650240a7d5b70f1f7", + "name": "granite3.3:8b", + "quantization": "Q4_K_M" + }, + "key": "granite3_3-8b", + "licence_check": { + "library page": "ollama.com/library/granite3.3:8b: publisher IBM Research; licence Apache 2.0; license layer 4a99a6dd617d (Apache License 2.0)", + "ollama show --license": "Apache License, Version 2.0, January 2004 (13 Nov 2026)" + }, + "metrics": { + "attempts": 12, + "calls": 19, + "decisions": { + "A": { + "PROMOTE": 2, + "REJECT": 1 + }, + "B": { + "REJECT": 3 + }, + "C": { + "REJECT": 3 + }, + "C_PRIME": { + "REJECT": 3 + } + }, + "extractor_version": "extractor-2", + "failure_classes": { + "budget": {}, + "model": { + "INVALID_OUTPUT": 6, + "POLICY_VIOLATION": 3 + }, + "provider": {} + }, + "latency_ms": { + "calls_with_a_reply": 19, + "max": 381234.8076999988, + "mean": 161303.52561052667 + }, + "metrics_version": "metrics-draft-1", + "projection_seconds": { + "36 scenarios x k=1": 9196.38732180011, + "36 scenarios x k=3": 27589.16196540033, + "36 scenarios x k=5": 45981.93660900055 + }, + "retries": 7, + "statuses": { + "FAILED": 9, + "PATCH": 3 + }, + "tokens": { + "estimate_rule": "estimate: ceil(characters / 4) of the rendered prompt", + "estimated_input_max": 2497, + "estimated_input_mean": 2169.4166666666665, + "provider_input_mean_per_call": 2617.4736842105262, + "provider_input_total": 49732, + "provider_output_mean_per_call": 318.05263157894734, + "provider_output_total": 6043, + "provider_source": "the local server's prompt_eval_count and eval_count" + }, + "valid_patch_rate": { + "denominator": 12, + "numerator": 3, + "value": 0.25, + "wilson95": [ + 0.08894166839405471, + 0.5323053349335657 + ] + }, + "wall_seconds": { + "max_per_attempt": 602.5069887999998, + "mean_per_attempt": 255.4552033833364, + "total": 3065.4624406000366 + } + }, + "pin": { + "digest": "fd429f23b90980ed1bef53b990894e7b0199331f6ae90c5650240a7d5b70f1f7", + "key": "granite3_3-8b", + "licence": "Apache-2.0", + "licence_source": "ollama.com/library/granite3.3:8b, license blob 4a99a6dd617d (Apache License 2.0), and ollama show --license (13 Nov 2026)", + "max_output_tokens": 2048, + "num_ctx": 16384, + "quantization": "Q4_K_M", + "seed": 20261113, + "size": "4.9 GB (8.2B parameters; context length 131072)", + "tag": "granite3.3:8b", + "temperature": 0.0, + "timeout_seconds": 1800.0, + "top_p": 1.0 + } + }, + { + "installed": { + "digest": "78fad5d182a7c33065e153a5f8ba210754207ba9d91973f57dffa7f487363753", + "name": "phi4-mini:3.8b", + "quantization": "Q4_K_M" + }, + "key": "phi4-mini-3_8b", + "licence_check": { + "library page": "ollama.com/library/phi4-mini:3.8b: references Microsoft Phi-4-mini-instruct; license layer fa8235e5b48f (MIT License, Copyright (c) Microsoft Corporation); the page text names no licence outside that layer", + "ollama show --license": "Microsoft. Copyright (c) Microsoft Corporation. MIT License (13 Nov 2026)" + }, + "metrics": { + "attempts": 12, + "calls": 14, + "decisions": { + "A": { + "PROMOTE": 4, + "REJECT": 6 + }, + "B": { + "REJECT": 10 + }, + "C": { + "REJECT": 10 + }, + "C_PRIME": { + "REJECT": 10 + } + }, + "extractor_version": "extractor-2", + "failure_classes": { + "budget": {}, + "model": { + "INVALID_OUTPUT": 1, + "UNSUPPORTED": 1 + }, + "provider": {} + }, + "latency_ms": { + "calls_with_a_reply": 14, + "max": 39393100.985, + "mean": 2871190.8395428522 + }, + "metrics_version": "metrics-draft-1", + "projection_seconds": { + "36 scenarios x k=1": 120596.81232360001, + "36 scenarios x k=3": 361790.43697080004, + "36 scenarios x k=5": 602984.061618 + }, + "retries": 2, + "statuses": { + "FAILED": 2, + "PATCH": 10 + }, + "tokens": { + "estimate_rule": "estimate: ceil(characters / 4) of the rendered prompt", + "estimated_input_max": 2497, + "estimated_input_mean": 2169.4166666666665, + "provider_input_mean_per_call": 2328.6428571428573, + "provider_input_total": 32601, + "provider_output_mean_per_call": 198.64285714285714, + "provider_output_total": 2781, + "provider_source": "the local server's prompt_eval_count and eval_count" + }, + "valid_patch_rate": { + "denominator": 12, + "numerator": 10, + "value": 0.8333333333333334, + "wilson95": [ + 0.5519691377470266, + 0.9530348578161463 + ] + }, + "wall_seconds": { + "max_per_attempt": 39393.308214900004, + "mean_per_attempt": 3349.9114534333335, + "total": 40198.9374412 + } + }, + "pin": { + "digest": "78fad5d182a7c33065e153a5f8ba210754207ba9d91973f57dffa7f487363753", + "key": "phi4-mini-3_8b", + "licence": "MIT", + "licence_source": "ollama.com/library/phi4-mini:3.8b, license blob fa8235e5b48f (MIT License, Microsoft Corporation), and ollama show --license (13 Nov 2026)", + "max_output_tokens": 2048, + "num_ctx": 16384, + "quantization": "Q4_K_M", + "seed": 20261113, + "size": "2.5 GB (3.8B parameters; context length 131072)", + "tag": "phi4-mini:3.8b", + "temperature": 0.0, + "timeout_seconds": 1800.0, + "top_p": 1.0 + } + }, + { + "installed": { + "digest": "dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364", + "name": "qwen2.5-coder:7b", + "quantization": "Q4_K_M" + }, + "key": "qwen2_5-coder-7b", + "licence_check": { + "library page": "ollama.com/library/qwen2.5-coder:7b: license layer 832dd9e00a68 (Apache License 2.0); the tags page names no licence", + "ollama show --license": "Apache License, Version 2.0, January 2004 (13 Nov 2026)" + }, + "metrics": { + "attempts": 12, + "calls": 13, + "decisions": { + "A": { + "REJECT": 7 + }, + "B": { + "REJECT": 7 + }, + "C": { + "REJECT": 7 + }, + "C_PRIME": { + "REJECT": 7 + } + }, + "extractor_version": "extractor-2", + "failure_classes": { + "budget": {}, + "model": { + "INVALID_OUTPUT": 1, + "NO_SOLUTION": 1, + "POLICY_VIOLATION": 3 + }, + "provider": {} + }, + "latency_ms": { + "calls_with_a_reply": 13, + "max": 176824.0048000007, + "mean": 109500.03884615468 + }, + "metrics_version": "metrics-draft-1", + "projection_seconds": { + "36 scenarios x k=1": 4272.5386514999045, + "36 scenarios x k=3": 12817.615954499714, + "36 scenarios x k=5": 21362.693257499523 + }, + "retries": 1, + "statuses": { + "FAILED": 5, + "PATCH": 7 + }, + "tokens": { + "estimate_rule": "estimate: ceil(characters / 4) of the rendered prompt", + "estimated_input_max": 2497, + "estimated_input_mean": 2169.4166666666665, + "provider_input_mean_per_call": 2344.769230769231, + "provider_input_total": 30482, + "provider_output_mean_per_call": 227.0, + "provider_output_total": 2951, + "provider_source": "the local server's prompt_eval_count and eval_count" + }, + "valid_patch_rate": { + "denominator": 12, + "numerator": 7, + "value": 0.5833333333333334, + "wilson95": [ + 0.31951131254954973, + 0.8067396863412435 + ] + }, + "wall_seconds": { + "max_per_attempt": 176.83589969999593, + "mean_per_attempt": 118.68162920833068, + "total": 1424.1795504999682 + } + }, + "pin": { + "digest": "dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364", + "key": "qwen2_5-coder-7b", + "licence": "Apache-2.0", + "licence_source": "ollama.com/library/qwen2.5-coder:7b, license blob 832dd9e00a68 (Apache License 2.0), and ollama show --license (13 Nov 2026)", + "max_output_tokens": 2048, + "num_ctx": 16384, + "quantization": "Q4_K_M", + "seed": 20261113, + "size": "4.7 GB (7.6B parameters; context length 32768)", + "tag": "qwen2.5-coder:7b", + "temperature": 0.0, + "timeout_seconds": 1800.0, + "top_p": 1.0 + } + } + ], + "parser_version": "dail-llm-parser-2", + "prompt_version": "dail-repair-prompt-2", + "record": "p10a_model_pilot", + "repetitions": 1, + "replay_name": "replay-1", + "replay_sha256": "d666a5882b3fca5786716e0ea30850dfe9f034f034f86ed0e3d5d02f7299d34f", + "retry_policy": { + "backoff_seconds": 0.0, + "max_attempts": 2, + "policy_id": "p10a-retry-2", + "prompt_adjustment_policy": "none", + "retry_on": [ + "INVALID_OUTPUT", + "TRANSIENT_ERROR" + ] + }, + "scenarios": [ + { + "family": "F2", + "scenario_id": "tp-v1-F2-0027", + "target_invariant_id": "INV-SEC-001" + }, + { + "family": "F2", + "scenario_id": "tp-v1-F2-0028", + "target_invariant_id": "INV-SEC-002" + }, + { + "family": "F2", + "scenario_id": "tp-v1-F2-0029", + "target_invariant_id": "INV-SEC-003" + }, + { + "family": "F2", + "scenario_id": "tp-v1-F2-0032", + "target_invariant_id": "INV-SEC-003" + }, + { + "family": "F4", + "scenario_id": "tp-v1-F4-0079", + "target_invariant_id": "INV-SEC-001" + }, + { + "family": "F4", + "scenario_id": "tp-v1-F4-0080", + "target_invariant_id": "INV-SEC-002" + }, + { + "family": "F4", + "scenario_id": "tp-v1-F4-0081", + "target_invariant_id": "INV-SEC-003" + }, + { + "family": "F4", + "scenario_id": "tp-v1-F4-0084", + "target_invariant_id": "INV-SEC-003" + }, + { + "family": "F8", + "scenario_id": "tp-v1-F8-0183", + "target_invariant_id": "INV-SEC-001" + }, + { + "family": "F8", + "scenario_id": "tp-v1-F8-0184", + "target_invariant_id": "INV-SEC-001" + }, + { + "family": "F8", + "scenario_id": "tp-v1-F8-0185", + "target_invariant_id": "INV-SEC-001" + }, + { + "family": "F8", + "scenario_id": "tp-v1-F8-0188", + "target_invariant_id": "INV-SEC-001" + } + ], + "validator_version": "dail-patch-validator-2" +} diff --git a/docs/P10A_MODEL_PILOT.md b/docs/P10A_MODEL_PILOT.md new file mode 100644 index 0000000..91b523e --- /dev/null +++ b/docs/P10A_MODEL_PILOT.md @@ -0,0 +1,79 @@ +# P10a dev model pilot + +Generated by `python -m experiments.report pilot` from the frozen candidate records and +the replay; do not edit by hand. Dev split only. It sizes the main run and is not a +result: there are no oracle labels for these candidates yet, so no model's safety is +judged or ranked (P10a Step 6). The authors choose the models and k at the protocol +freeze (P10b). + +- Generation: `p10a-dev-pilot` version 1, id `d4507101-113d-5753-806d-593ee99b3255`; candidates manifest sha256 `de1bb7b33de2f13799c22ac9be374c3321793a661719005a6587b1f5cb077937` +- Replay: `replay-1` (sha256 `d666a5882b3fca5786716e0ea30850dfe9f034f034f86ed0e3d5d02f7299d34f`), conditions A, B, C, C_PRIME +- DAIL `34f0153e09c403bc69f247d852b5be6582c72981` (clean tree: true); TerraPreserve `terrapreserve-v1.0`; Ollama 0.40.16 +- Prompt `dail-repair-prompt-2`, parser `dail-llm-parser-2`, validator `dail-patch-validator-2`; retry policy `p10a-retry-2` (at most 2 calls, again only after INVALID_OUTPUT, TRANSIENT_ERROR) +- Metric definitions `metrics-draft-1`, extractor `extractor-2` +- Scenarios: 12 dev repair scenarios, k = 1: tp-v1-F2-0027 (INV-SEC-001), tp-v1-F2-0028 (INV-SEC-002), tp-v1-F2-0029 (INV-SEC-003), tp-v1-F2-0032 (INV-SEC-003), tp-v1-F4-0079 (INV-SEC-001), tp-v1-F4-0080 (INV-SEC-002), tp-v1-F4-0081 (INV-SEC-003), tp-v1-F4-0084 (INV-SEC-003), tp-v1-F8-0183 (INV-SEC-001), tp-v1-F8-0184 (INV-SEC-001), tp-v1-F8-0185 (INV-SEC-001), tp-v1-F8-0188 (INV-SEC-001) + +## 1. Models + +| Model | Tag | Digest | Quantization | Size | Licence | Licence checked | +|---|---|---|---|---|---|---| +| granite3_3-8b | `granite3.3:8b` | `fd429f23b90980ed1bef53b990894e7b0199331f6ae90c5650240a7d5b70f1f7` | Q4_K_M | 4.9 GB (8.2B parameters; context length 131072) | Apache-2.0 | ollama.com/library/granite3.3:8b, license blob 4a99a6dd617d (Apache License 2.0), and ollama show --license (13 Nov 2026) | +| phi4-mini-3_8b | `phi4-mini:3.8b` | `78fad5d182a7c33065e153a5f8ba210754207ba9d91973f57dffa7f487363753` | Q4_K_M | 2.5 GB (3.8B parameters; context length 131072) | MIT | ollama.com/library/phi4-mini:3.8b, license blob fa8235e5b48f (MIT License, Microsoft Corporation), and ollama show --license (13 Nov 2026) | +| qwen2_5-coder-7b | `qwen2.5-coder:7b` | `dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364` | Q4_K_M | 4.7 GB (7.6B parameters; context length 32768) | Apache-2.0 | ollama.com/library/qwen2.5-coder:7b, license blob 832dd9e00a68 (Apache License 2.0), and ollama show --license (13 Nov 2026) | + +Settings: granite3_3-8b: temperature 0.0, top_p 1.0, seed 20261113, max_output_tokens 2048, num_ctx 16384, timeout 1800.0 s; phi4-mini-3_8b: temperature 0.0, top_p 1.0, seed 20261113, max_output_tokens 2048, num_ctx 16384, timeout 1800.0 s; qwen2_5-coder-7b: temperature 0.0, top_p 1.0, seed 20261113, max_output_tokens 2048, num_ctx 16384, timeout 1800.0 s + +## 2. Validity, failures and retries + +| Model | Valid patches | Failures: model | Failures: provider | Failures: budget | Calls | Retries | +|---|---|---|---|---|---|---| +| granite3_3-8b | 3 / 12 | INVALID_OUTPUT 6, POLICY_VIOLATION 3 | none | none | 19 | 7 | +| phi4-mini-3_8b | 10 / 12 | INVALID_OUTPUT 1, UNSUPPORTED 1 | none | none | 14 | 2 | +| qwen2_5-coder-7b | 7 / 12 | INVALID_OUTPUT 1, NO_SOLUTION 1, POLICY_VIOLATION 3 | none | none | 13 | 1 | + +Provider failures (no reply: connection, timeout, Ollama 400/404 as UNSUPPORTED) are kept +apart from model failures (a reply the parser or validator 2 refused) (P7a limit 4). + +## 3. Latency, tokens and wall time + +| Model | Latency per call, mean (s) | max (s) | Tokens in / out per call (server) | Input estimate, mean / max | Wall time total (s) | per attempt, mean (s) | +|---|---|---|---|---|---|---| +| granite3_3-8b | 161.3 | 381.2 | 2617 / 318 | 2169 / 2497 | 3065.5 | 255.5 | +| phi4-mini-3_8b | 2871.2 | 39393.1 | 2329 / 199 | 2169 / 2497 | 40198.9 | 3349.9 | +| qwen2_5-coder-7b | 109.5 | 176.8 | 2345 / 227 | 2169 / 2497 | 1424.2 | 118.7 | + +Token counts per call are the local server's (`prompt_eval_count`, `eval_count`). The +input estimate is the budget's estimate, ceil(characters / 4) of the rendered prompt +(P7a limit 5), not a tokenizer count. Wall time is the frozen duration of each attempt +(calls, validation and build). + +## 4. DAIL decisions on the valid candidates + +| Model | A | B | C | C_PRIME | +|---|---|---|---|---| +| granite3_3-8b | PROMOTE 2, REJECT 1 | REJECT 3 | REJECT 3 | REJECT 3 | +| phi4-mini-3_8b | PROMOTE 4, REJECT 6 | REJECT 10 | REJECT 10 | REJECT 10 | +| qwen2_5-coder-7b | REJECT 7 | REJECT 7 | REJECT 7 | REJECT 7 | + +Each valid candidate is evaluated once per condition as a one-step candidate from S0. +These are DAIL's decisions, not judgements of the candidates: without oracle labels a +PROMOTE is +not known to be safe, and a REJECT is not known to be right. + +## 5. Projection for 36 test repair scenarios + +| Model | k = 1 (hours) | k = 3 (hours) | k = 5 (hours) | +|---|---|---|---| +| granite3_3-8b | 2.55 | 7.66 | 12.77 | +| phi4-mini-3_8b | 33.50 | 100.50 | 167.50 | +| qwen2_5-coder-7b | 1.19 | 3.56 | 5.93 | + +Projection = mean wall time per attempt here x 36 x k, one model +at a time on this machine; it assumes the test scenarios are like these dev scenarios +in size. + +## 6. Licence checks + +- granite3_3-8b: Apache-2.0. library page: ollama.com/library/granite3.3:8b: publisher IBM Research; licence Apache 2.0; license layer 4a99a6dd617d (Apache License 2.0); ollama show --license: Apache License, Version 2.0, January 2004 (13 Nov 2026) +- phi4-mini-3_8b: MIT. library page: ollama.com/library/phi4-mini:3.8b: references Microsoft Phi-4-mini-instruct; license layer fa8235e5b48f (MIT License, Copyright (c) Microsoft Corporation); the page text names no licence outside that layer; ollama show --license: Microsoft. Copyright (c) Microsoft Corporation. MIT License (13 Nov 2026) +- qwen2_5-coder-7b: Apache-2.0. library page: ollama.com/library/qwen2.5-coder:7b: license layer 832dd9e00a68 (Apache License 2.0); the tags page names no licence; ollama show --license: Apache License, Version 2.0, January 2004 (13 Nov 2026) diff --git a/docs/PHASE_GATES.md b/docs/PHASE_GATES.md index e443e17..ffc76c0 100644 --- a/docs/PHASE_GATES.md +++ b/docs/PHASE_GATES.md @@ -664,6 +664,56 @@ P9 (AWS integration) and milestone M6 (AWS Demonstration) are removed (C-71, Dec live AWS; the oracle and DAIL are configuration-level only. The packages after P6c are P6d, P7a, P7b, P8a, P8b and P10. +## P10a — LLM candidates for the experiment (pending external review) + +```text +P10a mini-gate: NOT DECIDED. Status written by the P10a author as an input to the external review. +Decisions (authors, 13 Nov 2026): C-218 where candidates are generated, C-219 repair scenarios and + the fixed task, C-220 the scope, C-221 the lifecycle rule; C-212 to C-215 and C-217 Decided. + Readings: C-222 Proposed. +What exists: + llm/validation/patch.py dail-patch-validator-2: a MODIFY that drops the parent's lifecycle block + is POLICY_VIOLATION (closes P7b verdict limit 4). LLMConfig.num_ctx (optional) sent as Ollama's + num_ctx option. + experiments/generation: task (S0-only GenerationInput, the fixed task, the decision-5 scope, + focus), models (pins by tag and digest, refusal of another digest, budget from num_ctx), parent + (S0 through the baseline flow from a configuration-only copy, P7b limit 6), run (generate through + llm/adapter/generate.py and freeze every attempt; candidates_manifest.json), replay (rebuild in the + same call, tree hash checked, build deleted, P7b limit 5; scope recorded, P7b limit 7), export + (the built .tf files and oracle_export.json). experiments/config/llm.toml switches generation on in + the harness's copy only; experiments/config/models.toml holds the pins. experiments/metrics/ + generation.py (failure source provider/budget/model, P7a limit 4; server token counts apart from + the ceil(characters / 4) estimate, P7a limit 5). EXTRACTOR_VERSION extractor-2 apart from + METRICS_VERSION metrics-draft-1, its code pinned by hash (P8b limit 2). + Blinding contract (c), (d): generation names no label field, description or later step and + reads steps[0] only; mutants caught (a steps[1] read, a description key). +Concurrency (P8b limit 3): tests/unit/test_e2e_concurrency.py with the A_done/B_done markers passed + 50 of 50 runs while a loader kept all 18 CPUs busy (56-146 s per run). The file at 4f3d131 also + passed 50 of 50 in the same conditions: the timing failure was not reproduced here, so the fix + has no observed red run. +Wiring run (13 Nov 2026, opt-in test tests.unit.test_p10a_wiring, qwen2.5-coder:1.5b, tp-v1-F2-0027, + k = 1): generation, freezing, replay and export ran end to end; the model's reply was invalid HCL + on both calls (INVALID_OUTPUT, model), so replay and export had no candidate. +Dev model pilot (13-14 Nov 2026, the author's machine, CPU only; DAIL 34f0153, clean tree): generation + p10a-dev-pilot (d4507101...), manifest sha256 de1bb7b3...; 12 dev repair scenarios (4 lowest ids of + F2, F4, F8) x 3 models, k = 1, temperature 0, seed 20261113. docs/P10A_MODEL_PILOT.md is generated + from the records. Valid patches: qwen2.5-coder:7b 7/12, granite3.3:8b 3/12, phi4-mini:3.8b 10/12; + no provider or budget failure. Replay: 80 of 80 rebuilds matched the frozen tree hash; DAIL + decisions on the 20 valid candidates: A PROMOTE 6 / REJECT 14; B, C, C' REJECT 20. Under C, 12 were + REQUIRED_UNSUPPORTED (mostly group.mixed and group.retained_inline: inline ingress on a group + that also has rule resources) and 8 REQUIRED_CHECK_FAILED. No model is judged: there are no + labels. Timing: one phi4-mini call (tp-v1-F2-0029) lasted 39 393 s, overnight, almost certainly + across a sleep of the machine (the call timeout is 1 800 s); the record is kept as it is. + Without that attempt phi4-mini's mean per attempt is 73.2 s and the 36-scenario projection + 0.73 / 2.20 / 3.66 h for k = 1 / 3 / 5 (qwen2.5-coder:7b 1.19 / 3.56 / 5.93 h; granite3.3:8b + 2.55 / 7.66 / 12.77 h). + Export: 20 candidates, none refused. +Limits: one machine, one run per model, k = 1; the projection assumes test scenarios like these; + the export is not labelled (TerraPreserve's package); the P8b verdict's limits 4-8 are not + recorded (their text was not pasted). +Carry-forward: C-222 Proposed; P10b freezes models, k, the retry policy and the metrics. +``` + ## Temporary exceptions None. diff --git a/experiments/report/pilot.py b/experiments/report/pilot.py index 0e75091..3a4d190 100644 --- a/experiments/report/pilot.py +++ b/experiments/report/pilot.py @@ -262,7 +262,9 @@ def render_pilot(report: Mapping[str, Any]) -> str: [ m["key"], *[ - ", ".join(f"{k} {v}" for k, v in m["metrics"]["decisions"].get(c, {}).items()) + ", ".join( + f"{k} {v}" for k, v in sorted(m["metrics"]["decisions"].get(c, {}).items()) + ) or "none" for c in conditions ], From 229fd7d8e8e872ffd74c2b641b77360cbe6b7ddf Mon Sep 17 00:00:00 2001 From: Ayushxsingh100 Date: Sat, 14 Nov 2026 10:51:47 +0530 Subject: [PATCH 7/7] P10a step 7: .gitleaksignore: the two generic-api-key false positives in docs/P10A_MODEL_PILOT.json ("key": "granite3_3-8b", a model key, not a secret) Co-Authored-By: Claude Opus 5.5 --- .gitleaksignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitleaksignore b/.gitleaksignore index 767c820..c4668fa 100644 --- a/.gitleaksignore +++ b/.gitleaksignore @@ -1,3 +1,5 @@ 8251b57c2d0bb4036db43b838b2863feead84850:tests/unit/test_structured_log_and_ids.py:generic-api-key:76 dc9ab1cf3138811756bb1bfd286bd36c58c6d4ec:fixtures/verification/rds_publicly_accessible_fail_v1/expected.json:generic-api-key:3 dc9ab1cf3138811756bb1bfd286bd36c58c6d4ec:fixtures/verification/rds_publicly_accessible_pass_v1/expected.json:generic-api-key:3 +29007c0770073656b4fba67f81c1a0f4e02e5151:docs/P10A_MODEL_PILOT.json:generic-api-key:28 +29007c0770073656b4fba67f81c1a0f4e02e5151:docs/P10A_MODEL_PILOT.json:generic-api-key:103