Strengthening tests

.agents/skills/unit-test-quality/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: unit-test-quality
+description: Write, strengthen, or review unit tests for meaningful behavioral protection. Use when adding tests, reviewing test changes, investigating misleading coverage, or checking generated tests. Focuses on contracts, independent oracles, preconditions, and defect detection; it does not require TDD, property-based testing, mutation frameworks, or coverage targets.
+---
+
+# Unit Test Quality
+
+Treat a passing test as evidence only when it distinguishes conforming behavior from a plausible defect. Optimize for fault detection, not test count, assertion count, or line coverage.
+
+## Establish what the test proves
+
+For each new or materially changed non-trivial test, identify:
+
+- the externally observable contract or invariant;
+- the plausible production defect the test must detect;
+- why the assertion would differ under that defect;
+- the source of the expected result.
+
+Do this in working notes or the final explanation; do not add ceremonial comments to every test. If no plausible defect would change the assertion, remove or redesign the test.
+
+Infer contracts from authoritative specifications, public API documentation, issue reports, compatibility promises, and accepted domain rules. Existing implementation behavior is evidence, not automatically the contract. If the intended behavior is genuinely ambiguous, say so instead of canonizing the current output in a test.
+
+## Keep the oracle independent
+
+Derive expected values independently of the path being tested. Prefer, in order:
+
+1. an explicit specification or hand-derived example;
+2. a simpler, structurally different reference calculation;
+3. a well-established independent implementation;
+4. an invariant or relation whose truth does not depend on the same algorithm.
+
+Do not compute expectations with the function under test, its policy resolver, its parser/serializer pair when both share the same defect, or a helper that merely mirrors the implementation. Do not copy constants, branch tables, or formulas from production code unless the contract itself defines them and the test cites that independent source.
+
+When exact expected values are impractical, use strong invariants, metamorphic relations, differential checks, or bounded properties. Property-based generation is optional; ordinary examples are often enough.
+
+## Verify the setup can reach the behavior
+
+Check that the arranged state actually activates the branch, boundary, error, side effect, or interaction being claimed. Assert a precondition only when it is not already obvious from a small literal fixture and when a false precondition could let the main assertion pass vacuously.
+
+Watch especially for:
+
+- empty parameterized or generated case sets;
+- filters or skips that remove the intended case;
+- fixtures that never create the relevant state;
+- assertions inside callbacks that are never called;
+- exceptions accepted from the wrong operation;
+- numeric inputs that do not straddle the specified boundary.
+
+## Test behavior at the right boundary
+
+Exercise the public or stable unit boundary that owns the contract. Calling a private helper can be useful diagnostically, but it does not protect the public path unless the public path is also exercised.
+
+Mocks may isolate nondeterministic, slow, or external boundaries. Do not assert merely that a mock returned what it was configured to return. Preserve side effects and response structure that the unit relies on, and assert the unit's observable result or required interaction contract. If mock setup dominates the test or replaces the behavior being claimed, use a real collaborator, fake, or a broader test.
+
+## Use discriminating cases
+
+Choose cases that separate common wrong implementations from the correct one. Include boundaries and negative cases when they expose distinct plausible faults, not to satisfy a checklist. For policy code, derive cases from the policy rather than from the implementation's resolved value.
+
+For generated tests, inspect both the generator and representative generated output. Ensure the generated test invokes the production entry point it claims to protect and that its expected values are independent of generator/runtime helpers implementing the same rule.
+
+Avoid tests that only prove:
+
+- code did not throw;
+- a value is non-null, has a type, or contains a key when its meaning matters;
+- an object reproduces the fixture that created it;
+- a mock exists or was called without checking meaningful arguments/effects;
+- the current implementation is internally consistent;
+- a snapshot changed without establishing what must remain stable.
+
+## Challenge important tests
+
+For a new regression test or a test protecting a non-trivial contract, identify one small semantic mutation representing the named defect: change a boundary operator, use a wrong constant, remove validation or a side effect, select the wrong branch, or return a plausible default.
+
+When safe and cheap, apply that mutation temporarily and run the narrowest relevant test. The test must fail for the intended reason. Restore the production file immediately, verify restoration, then run the unmodified test. Never leave a probe mutation in the worktree or mix it with unrelated user changes. If the worktree cannot be safely restored, use an isolated copy or perform a reasoned mutation review instead.
+
+Do not require an executed mutation for trivial accessors, framework wiring, destructive or external behavior, expensive suites, or cases where the test already demonstrably failed on the exact defect. Record why it was skipped when the test's protective value is otherwise uncertain.
+
+## Review outcome
+
+When reviewing, report concrete findings rather than grading style. For each weak test, state:
+
+- claimed contract;
+- weakness in oracle, setup, boundary, or assertion;
+- a plausible defect that survives;
+- the smallest strengthening that would detect it.
+
+Distinguish “code executed” from “behavior protected.” Treat coverage only as a map of code not exercised; never use a percentage as proof of test quality.
+
+Before completion, run the focused tests and then the relevant broader suite when practical. Report commands and outcomes truthfully. Do not weaken assertions, update expected values to buggy output, add skips, or reduce scope merely to make CI green.

.agents/skills/unit-test-quality/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Unit Test Quality"
+  short_description: "Write and review tests that prove behavioral contracts"
+  default_prompt: "Review or write these unit tests for meaningful behavioral protection."

AGENTS.md

@@ -13,3 +13,6 @@ completely sure.
 Follow the [development guide](docs/src/contributing/development.md) for
 dependency management, PTF extraction and generation, target implementation,
 validation, documentation, and commit conventions.
+
+When writing, strengthening, or reviewing codegen tests, use the
+[unit-test-quality skill](.agents/skills/unit-test-quality/SKILL.md).

codegen/src/compile.rs

@@ -5,8 +5,6 @@ use crate::model::{
     Entry, Function, Output, Outputs, ToleranceSource, VerificationInput,
 };
 
-pub(crate) const FLOATING_POINT_GUARD: f64 = 1e-14;
-
 pub(super) fn functions(entries: Vec<Entry>) -> Result<Vec<CompiledFunction>> {
     let mut compiled = Vec::new();
     for entry in entries {

codegen/src/corpus_report.rs

@@ -645,10 +645,7 @@ fn render_values(output: &mut String, values: &[Frequency]) {
 
 #[cfg(test)]
 mod tests {
-    use std::{
-        path::{Path, PathBuf},
-        sync::Arc,
-    };
+    use std::{path::PathBuf, sync::Arc};
 
     use super::{InputKind, Report};
     use crate::model::{Entry, Spec};
@@ -707,6 +704,7 @@ functions:
     verification_cases:
       - {id: one, kind: published, inputs: {x: 1.0}, expected: {alpha: 1.0, zeta: 2.0}, source_location: 'Table 1, row 1'}
       - {id: two, kind: calculated, inputs: {x: 2.0}, expected: {alpha: 2.0, zeta: 3.0}, rationale: Interior input.}
+      - {id: three, kind: calculated, inputs: {x: 3.0}, expected: {alpha: 3.0, zeta: 4.0}, rationale: Another interior input.}
     edge_cases:
       - {id: edge, inputs: {x: 0.0}, expected_behavior: Finite., notes: Metadata only.}
   - name: calc_ptf_test_blocked
@@ -748,18 +746,26 @@ functions:
         let report = Report::from_entries(&[fixture()]);
 
         assert_eq!(report.functions.total, 4);
-        assert!(
+        assert_eq!(
             report
                 .functions
                 .by_status
                 .iter()
-                .all(|status| status.count == 1)
+                .map(|status| (status.value.as_str(), status.count))
+                .collect::<Vec<_>>(),
+            [
+                ("implemented", 1),
+                ("ready-for-implementation", 1),
+                ("blocked", 1),
+                ("draft", 1)
+            ]
         );
-        assert_eq!(report.verification.all_functions.verification_cases, 2);
+        assert_eq!(report.verification.cases_total, 3);
+        assert_eq!(report.verification.all_functions.verification_cases, 3);
         assert_eq!(report.verification.by_kind["published"], 1);
-        assert_eq!(report.verification.by_kind["calculated"], 1);
+        assert_eq!(report.verification.by_kind["calculated"], 2);
         assert_eq!(report.verification.published_cases, 1);
-        assert_eq!(report.verification.calculated_cases, 1);
+        assert_eq!(report.verification.calculated_cases, 2);
         assert_eq!(report.verification.all_functions.edge_cases, 1);
         assert_eq!(
             report
@@ -768,6 +774,20 @@ functions:
                 .functions_with_verification_cases,
             1
         );
+        assert_eq!(
+            report
+                .verification
+                .all_functions
+                .functions_with_verification_cases_percentage,
+            25.0
+        );
+        assert_eq!(
+            report
+                .verification
+                .ready_for_implementation_functions
+                .verification_cases,
+            0
+        );
         assert_eq!(report.verification.implemented_functions.functions, 1);
         assert_eq!(
             report
@@ -818,7 +838,10 @@ functions:
             report.blocked_functions[0].documentation_warnings,
             ["Required coefficients are unavailable."]
         );
-        assert!(report.blocked_functions[0].scientific_notes.is_some());
+        assert_eq!(
+            report.blocked_functions[0].scientific_notes.as_deref(),
+            Some("Blocker evidence from source review.")
+        );
         assert_eq!(
             report
                 .scope
@@ -829,52 +852,41 @@ functions:
     }
 
     #[test]
-    fn serializes_deterministically_to_json() {
-        let report = Report::from_entries(&[fixture()]);
-        let first = serde_json::to_string_pretty(&report).expect("report serializes");
-        let second = serde_json::to_string_pretty(&report).expect("report serializes again");
-        let text = report.to_text();
-
-        assert_eq!(first, second);
-        assert!(first.contains("\"blocked_functions\""));
-        assert!(first.contains("\"implemented_functions\""));
-        assert!(!first.contains("missing_quantity_or_unit_validation_failures"));
-        assert!(text.find("Implemented functions") < text.find("Quantity registry"));
+    fn report_order_is_independent_of_source_order() {
+        let mut earlier = fixture();
+        earlier.slug = "alpha1990".into();
+        let later = fixture();
+        let forward = Report::from_entries(&[earlier.clone(), later.clone()]);
+        let reverse = Report::from_entries(&[later, earlier]);
+        assert_eq!(
+            serde_json::to_value(&forward).unwrap(),
+            serde_json::to_value(&reverse).unwrap()
+        );
+        assert_eq!(forward.to_text(), reverse.to_text());
+        assert_eq!(forward.sources.earliest_publication_year, Some(1990));
+        assert_eq!(forward.sources.latest_publication_year, Some(2020));
     }
 
     #[test]
-    fn repository_corpus_report_is_complete_and_stable() {
-        let root = Path::new(env!("CARGO_MANIFEST_DIR"))
-            .parent()
-            .expect("workspace root exists");
-        let entries = crate::specs::load(root).expect("repository specifications load");
-        assert!(crate::validate::specifications(&entries).is_empty());
-        crate::compile::functions(entries.clone()).expect("repository specifications compile");
-
-        let report = Report::from_entries(&entries);
-        assert_eq!(report.verification.cases_total, 136);
-        assert_eq!(report.verification.by_kind["calculated"], 125);
-        assert_eq!(report.verification.by_kind["published"], 11);
-        assert_eq!(report.sources.specification_files, entries.len());
-        assert_eq!(
-            report.functions.total,
-            entries
-                .iter()
-                .map(|entry| entry.spec.functions.len())
-                .sum::<usize>()
-        );
+    fn serializes_coverage_and_orders_text_sections() {
+        let report = Report::from_entries(&[fixture()]);
+        let json = serde_json::to_value(&report).unwrap();
+        assert_eq!(json["verification"]["cases_total"], 3);
         assert_eq!(
-            report
-                .functions
-                .by_status
-                .iter()
-                .map(|status| status.count)
-                .sum::<usize>(),
-            report.functions.total
+            json["verification"]["implemented_functions"]["functions"],
+            1
         );
         assert_eq!(
-            serde_json::to_string_pretty(&report).unwrap(),
-            serde_json::to_string_pretty(&Report::from_entries(&entries)).unwrap()
+            json["blocked_functions"][0]["function_name"],
+            "calc_ptf_test_blocked"
         );
+        let text = report.to_text();
+        let implemented = text
+            .find("Implemented functions")
+            .expect("implemented coverage section");
+        let registry = text
+            .find("Quantity registry")
+            .expect("quantity registry section");
+        assert!(implemented < registry);
     }
 }

codegen/src/fixtures/quantities.yaml

@@ -0,0 +1,10 @@
+volumetric_water_content:
+  description: Test fraction.
+  units:
+    volume_fraction:
+      absolute: 0.001
+      rationale: Test fraction tolerance.
+    volume_percent:
+      absolute: 0.1
+      relative: 0.01
+      rationale: Test percent tolerance.

codegen/src/fixtures/source.yaml

@@ -0,0 +1,33 @@
+source:
+  summary: Synthetic rendering example.
+  citation_apa: Example (2026).
+  doi: null
+functions:
+  - name: calc_ptf_example_scalar
+    status: implemented
+    public_api: {name: calc_ptf_example_scalar, summary: Scalar example.}
+    scope: {prediction_target: Test fraction., models: {h_theta: null, k_h: null}}
+    inputs:
+      - {name: x, unit: '1', symbol: null, domain: null, description: First input.}
+      - {name: y, unit: '1', symbol: null, domain: null, description: Second input.}
+    outputs: {type: scalar, name: value, quantity: volumetric_water_content, unit: volume_fraction, reported_unit: '1', symbol: null, domain: null, description: Scalar value.}
+    implementation:
+      variables: [{name: value, expr: x + y}]
+    verification_cases:
+      - {id: sum, kind: calculated, inputs: {x: 1.0, y: 2.0}, expected: {value: 3.0}, rationale: Test addition.}
+  - name: calc_ptf_example_record
+    status: implemented
+    public_api: {name: calc_ptf_example_record, summary: Record example.}
+    scope: {prediction_target: Test fractions., models: {h_theta: null, k_h: null}}
+    inputs:
+      - {name: x, unit: '1', symbol: null, domain: null, description: Input value.}
+    outputs:
+      type: record
+      name: ExampleResult
+      fields:
+        - {name: first, quantity: volumetric_water_content, unit: volume_fraction, reported_unit: '1', symbol: null, domain: null, description: First result.}
+        - {name: second, quantity: volumetric_water_content, unit: volume_fraction, reported_unit: '1', symbol: null, domain: null, description: Second result.}
+    implementation:
+      variables: [{name: first, expr: x}, {name: second, expr: x * 2}]
+    verification_cases:
+      - {id: record, kind: calculated, inputs: {x: 1.0}, expected: {first: 1.0, second: 2.0}, rationale: Test record.}

codegen/src/fixtures/units.yaml

@@ -0,0 +1,18 @@
+volume_fraction:
+  preferred_notation: "m^3/m^3"
+  aliases: ["cm^3/cm^3", "cm³/cm³", "1"]
+volume_percent:
+  preferred_notation: "% v/v"
+  aliases: ["vol.%", "% volume/volume", "%"]
+mass_percent:
+  preferred_notation: "% w/w"
+  aliases: ["%"]
+millimeter_per_hour:
+  preferred_notation: "mm/h"
+  aliases: []
+kilopascal:
+  preferred_notation: "kPa"
+  aliases: []
+dimensionless:
+  preferred_notation: "1"
+  aliases: []

codegen/src/main.rs

@@ -18,6 +18,9 @@ mod targets;
 mod validate;
 mod version;
 
+#[cfg(test)]
+mod test_support;
+
 #[derive(Parser)]
 #[command(about = "Validate ptfkit specifications and generate bindings")]
 pub(crate) struct Cli {

codegen/src/model.rs

@@ -847,22 +847,11 @@ mod tests {
 
     #[test]
     fn public_python_generation_defaults_to_generated() {
-        let mut spec = Spec {
-            source: Source {
-                summary: "Test (2026), test territory.".into(),
-                citation_apa: "Test (2026).".into(),
-                doi: None,
-            },
-            scope: Scope::default(),
-            generation: Generation::default(),
-            scientific_notes: String::new(),
-            functions: Vec::new(),
-        };
-
+        let spec: Spec = serde_yaml::from_str(
+            "source: {summary: Test source., citation_apa: Test (2026)., doi: null}\nfunctions: []",
+        )
+        .unwrap();
         assert_eq!(spec.generation.public_python, PythonGeneration::Generated);
-
-        spec.generation.public_python = PythonGeneration::Manual;
-        assert_eq!(spec.generation.public_python, PythonGeneration::Manual);
     }
 
     #[test]