Exclude test code from coverage and run candidates once

README.md

@@ -64,6 +64,8 @@ Run `docker/docker-run` from the same directory to mount the repository at `/hom
 
 ## Workflow
 
+To measure existing tests before preprocessing, run `utgen coverage -p <original-crate-copy>`. It runs tests once, excludes test code from coverage, and writes `coverage.xml` and `coverage.json` without model configuration. Keep this baseline copy separate from the generation copy.
+
 On a fresh working copy of a standalone crate, run `utgen pre-process -p <target-crate-path>` before `utgen analyze -p <target-crate-path>`. Analysis clears Cargo's check cache, runs brinfo and focxt, and verifies their outputs. It rejects existing `brinfo/` or `focxt/` directories so previous analysis results cannot be mixed into the run.
 
 Then use `utgen gen` and `utgen fix` for generation and compilation repair. Pass `--requirement --context` to `utgen gen` to include path constraints and focal context. Preprocessing and test checks modify the target tree; use a working copy of the target project.

docs/minimal-pipeline.md

@@ -18,21 +18,23 @@ The script prints the temporary directory containing source copies, command logs
 
 ## Verified Contracts
 
-- Original tests run on an untouched copy. A separate prepared copy is used for generated tests.
+- Original tests run on a separate original copy; coverage annotations are temporary and restored. A separate prepared copy is used for generated tests.
 - Preprocessing removes test-only source ranges while preserving newlines and byte offsets, including a test and production function sharing one line. Repeating preprocessing is idempotent.
 - `utgen analyze` succeeds even after a prior `cargo check`, since it clears the check cache before invoking the compiler wrapper. It rejects existing analysis output directories.
 - brinfo and focxt agree on five complete function names. Source locations identify the exported focal code, accounting for brinfo's indentation normalization.
 - The `Gauge::label` context contains its struct, the focal method, and the declarations for `classify`, `positive`, and the transitive `threshold` dependency.
 - Missing context and malformed source locations fail before generation changes the manifest or source files. Source-location parse errors no longer panic.
 - Six model responses produce six candidate tests. One candidate deliberately passes a string to an integer parameter and fails compilation; a numbered ChangeLog response repairs it through the actual repair path.
+- Coverage executes each test body once for XML and JSON. Test functions, module helpers, standalone test-only helpers, and generated `llmtests` records are excluded; production coverage remains.
+- Sources are restored after successful coverage, assertion failures, compilation failures, and JSON export failures.
 - Both branches of `classify` are covered. Before repair, the intentionally invalid `double` candidate has zero compilable tests. After repair, all candidates compile and pass.
-- Source files and the target manifest match their prepared state after generation and repair; no source backup remains after successful completion.
+- Direct byte comparisons verify that source files and the target manifest match their prepared state after generation and repair; no source backup remains after successful completion.
 
 The regression script runs in Linux CI after the ordinary workspace tests. It uses the real analyzers, compiler, coverage tool, prompt builder, response parser, candidate checks, and repair machinery. Model replies are deterministic fixtures; these results do not measure LLM quality.
 
 ## Coverage Interpretation
 
-`original/original-coverage.json` measures all original unit and integration tests on the untouched copy. The check requires both original `classify` branches to have executed.
+`original/original-coverage.json` and `.xml` measure production code exercised by all original unit and integration tests through `utgen coverage`. Test bodies and helpers contribute to neither the coverage numerator nor denominator. The check requires both original `classify` branches to have executed. LLVM can emit separate records for the same branch location in different test binaries; the check unions those outcomes instead of requiring both in one record.
 
 `generated/utgen/result/` and `generated/utgen/fixed_result/` contain the tool's per-function statistics for generated candidates. The script checks the known branch count and execution outcomes directly. The separate legacy `utgen/original_result.json` restores only the integration-test backup; it must not be presented as coverage of all original tests. Combined original-plus-generated coverage is not part of this check.
 
@@ -44,4 +46,6 @@ The preprocessor parses `cfg` predicates conservatively, preserves `cfg(not(test
 
 Recursive context lookup now resolves full function names through the compiler's encoded-name index and consumes the accumulated dependencies. External functions without local artifacts remain outside this source context.
 
-The local macOS/Apple Silicon run passed using six generation responses and one repair response. Real-model behavior, bytes-scale execution, general multi-crate support, request concurrency limits, hard process timeouts, recovery after forced termination, and parameter-aware cache invalidation remain later work. Git history is retained.
+The local macOS/Apple Silicon run passed using six generation responses and one repair response. Real-model behavior, bytes-scale execution, general multi-crate support, request concurrency limits, hard process timeouts, recovery after forced termination, and explicit run/resume behavior remain later work. Git history is retained.
+
+The coverage entry temporarily annotates sources with `#[coverage(off)]` and enables the nightly feature at crate roots without adding newlines. It preserves compiler flags, restores original bytes on returned errors, and performs no hashing or source fingerprint checks. `cargo llvm-cov report --json` reuses the first run's data; the first XML-producing command retains the tool's normal profile cleanup between candidates. This does not provide crash recovery or general macro expansion. See the [coverage command reference](../utgen/README.md#coverage-of-existing-tests) for scope.

docs/palm-rust-unit-test-generation.md

@@ -381,7 +381,7 @@ export PALM_CONFIG="$(pwd)/utgen/res/api.json"
 
     ```sh
     cargo llvm-cov --tests --ignore-run-fail --branch --cobertura --output-path coverage.xml
-    cargo llvm-cov --tests --ignore-run-fail --branch --json --output-path coverage.json
+    cargo llvm-cov report --json --output-path coverage.json
     ```
 
   - 错误模式下执行：
@@ -392,6 +392,12 @@ export PALM_CONFIG="$(pwd)/utgen/res/api.json"
 
   - `gen_test_rate` 读取生成结果，运行覆盖率，计算每个被测函数的测试数、编译通过数、运行通过数、行覆盖和分支覆盖。
 
+- `utgen/src/run/coverage.rs`
+  - 覆盖率入口复用预处理的测试识别规则，在测试模块、独立测试函数及 `cfg(test)` 辅助函数前临时添加 `#[coverage(off)]`。整个测试模块（含生成的 `llmtests`）排除后，其中的辅助代码也不计入覆盖率分子和分母；被调用的生产函数仍参与统计。
+  - 在 crate 根文件同一行临时开启 `coverage_attribute`，保持原行号，不覆盖编译 flags。成功或编译/导出报错返回前恢复源文件原始内容；强制终止后的恢复留待后续处理。
+  - 测试只执行一次，先输出 XML，再由 `report --json` 导出同一份运行数据。第一条命令保留默认清理，避免候选之间累积旧覆盖率。
+  - `utgen coverage -p <原始 crate 工作副本>` 可在预处理前测量原有测试，无需模型配置和分析产物；结果保留在该副本的 `coverage.xml`、`coverage.json`。断言失败仍导出覆盖率，编译或报告错误则返回失败。当前支持源码位于 `src/` 的独立 crate，不展开宏，也不自动推断未标记的公共辅助函数是否仅用于测试。
+
 - `utgen/src/run/coverage_json.rs`
   - 解析 `coverage.json` 中的 branch 信息。
 

examples/minimal/README.md

@@ -1,5 +1,5 @@
 # Minimal PALM Target
 
-This standalone library exercises five focal functions: a branch, a struct method, an inline module, and a two-level cross-module call chain. It also includes existing unit/integration tests and a test sharing a line with production code. Its test dependencies are locked for the pinned nightly.
+This standalone library exercises five focal functions: a branch, a struct method, an inline module, and a two-level cross-module call chain. It also includes existing unit/integration tests and a test sharing a line with production code, and test-only helpers. A helper can record a test execution for the coverage regression, which checks that XML and JSON do not cause duplicate runs. Its test dependencies are locked for the pinned nightly.
 
 Run the [minimal pipeline check](../../docs/minimal-pipeline.md) from the PALM repository root. The script copies this directory before preprocessing or analysis; generated data is not written into this fixture.

examples/minimal/src/lib.rs

@@ -2,8 +2,15 @@ mod support;
 
 #[cfg(test)]
 mod tests {
+    fn record_run() {
+        if let Ok(path) = std::env::var("PALM_FIXTURE_RUN_LOG") {
+            use std::io::Write;
+            std::fs::OpenOptions::new().create(true).append(true).open(path).unwrap()
+                .write_all(b"x").unwrap();
+        }
+    }
     #[test]
-    fn existing_unit_test() { assert_eq!(super::classify(2), 1); }
+    fn existing_unit_test() { record_run(); assert_eq!(super::classify(2), 1); }
 }
 
 pub struct Gauge {

examples/minimal/src/support.rs

@@ -1,7 +1,10 @@
-#[test] fn before_helper() { assert!(positive(1)); } pub fn positive(value: i32) -> bool {
+#[test] fn before_helper() { assert!(test_only_helper()); } pub fn positive(value: i32) -> bool {
     threshold(value)
 }
 
 fn threshold(value: i32) -> bool {
     value > 0
 }
+
+#[cfg(test)]
+fn test_only_helper() -> bool { positive(1) }

scripts/check_minimal.py

@@ -1,7 +1,6 @@
 #!/usr/bin/env python3
 """Exercise PALM on an isolated, deterministic crate; never use a real model."""
 import argparse
-import hashlib
 import http.server
 import json
 import os
@@ -11,14 +10,16 @@
 import subprocess
 import tempfile
 import threading
+import xml.etree.ElementTree as ET
 
 REPO = Path(__file__).resolve().parents[1]
 
 
 def snapshot(root):
-    return {str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest()
-            for p in sorted((root / 'src').rglob('*.rs'))} | {
-                'Cargo.toml': hashlib.sha256((root / 'Cargo.toml').read_bytes()).hexdigest()}
+    return {str(p.relative_to(root)): p.read_bytes()
+            for directory in ['src', 'tests', 'tests.bak']
+            for p in sorted((root / directory).rglob('*.rs'))} | {
+                'Cargo.toml': (root / 'Cargo.toml').read_bytes()}
 
 
 def source_fragment(root, loc):
@@ -129,14 +130,68 @@ def run(label, command, cwd=target, expected=0):
 
     version = run('coverage-version', ['cargo', 'llvm-cov', '--version'])
     assert version.stdout.strip() == 'cargo-llvm-cov 0.6.16', version.stdout
-    run('original-coverage', ['cargo', 'llvm-cov', '--tests', '--branch', '--json', '--output-path', 'original-coverage.json'], original)
-    original_coverage = json.loads((original / 'original-coverage.json').read_text())
+    baseline = snapshot(original)
+    # A test side effect must occur once even though both formats are exported.
+    counter = work / 'coverage-runs.txt'
+    env['PALM_FIXTURE_RUN_LOG'] = str(counter)
+    coverage_run = run('original-coverage', ['utgen', 'coverage', '-p', str(original)], original)
+    assert counter.read_text() == 'x', 'coverage ran the test body more than once'
+    env.pop('PALM_FIXTURE_RUN_LOG')
+    assert snapshot(original) == baseline
+    assert '2 passed; 0 failed' in coverage_run.stdout and '1 passed; 0 failed' in coverage_run.stdout
+    original_coverage = json.loads((original / 'coverage.json').read_text())
+    (original / 'coverage.json').rename(original / 'original-coverage.json')
+    (original / 'coverage.xml').rename(original / 'original-coverage.xml')
+    covered_functions = [function['name'] for data in original_coverage['data'] for function in data['functions']]
+    assert not any(marker in name for name in covered_functions
+                   for marker in ['existing_unit_test', 'before_helper', 'test_only_helper', 'record_run', 'existing_integration_test']), covered_functions
+    assert any('classify' in name for name in covered_functions), covered_functions
+    xml = ET.parse(original / 'original-coverage.xml')
+    assert not any(marker in method.get('name', '') for method in xml.iter('method')
+                   for marker in ['existing_unit_test', 'before_helper', 'test_only_helper', 'record_run'])
+    # Test-only lines must disappear from both the numerator and denominator.
+    lib_lines = (original / 'src/lib.rs').read_text().splitlines()
+    test_line = next(i for i, line in enumerate(lib_lines, 1) if 'fn existing_unit_test' in line)
+    for file in (file for data in original_coverage['data'] for file in data['files']):
+        if Path(file['filename']) == original / 'src/lib.rs':
+            assert not any(segment[0] == test_line and segment[3] for segment in file['segments'])
+    for cls in xml.iter('class'):
+        if cls.get('filename', '').endswith('src/lib.rs'):
+            assert not any(int(line.get('number')) == test_line for line in cls.findall('./lines/line'))
     original_file = next(file for data in original_coverage['data'] for file in data['files']
                          if Path(file['filename']) == original / 'src/lib.rs')
     condition_line = (original / 'src/lib.rs').read_text().splitlines().index('    if support::positive(value) {') + 1
     # Library and unit-test binaries may emit separate records for one location.
     records = [branch for branch in original_file['branches'] if branch[0] == condition_line]
     assert any(branch[4] > 0 for branch in records) and any(branch[5] > 0 for branch in records)
+    # Preserve assertion failures as test outcomes, but stop on build errors;
+    # temporary coverage attributes must be removed in either case.
+    lib = original / 'src/lib.rs'
+    saved_lib = lib.read_bytes()
+    try:
+        lib.write_bytes(saved_lib.replace(b'assert_eq!(super::classify(2), 1)', b'assert_eq!(super::classify(2), 99)'))
+        failed_test_source = snapshot(original)
+        failed_test = run('coverage-failed-test', ['utgen', 'coverage', '-p', str(original)], original)
+        assert '1 passed; 1 failed' in failed_test.stdout
+        assert snapshot(original) == failed_test_source
+        lib.write_bytes(saved_lib + b'\ncompile_error!("coverage build failure fixture");\n')
+        failed_build_source = snapshot(original)
+        failure = run('coverage-failed-build', ['utgen', 'coverage', '-p', str(original)], original, expected=1)
+        assert 'coverage build failure fixture' in failure.stderr and 'panicked' not in failure.stderr
+        assert snapshot(original) == failed_build_source
+        lib.write_bytes(saved_lib)
+        # Block only the JSON export, after the tests and XML export succeed.
+        (original / 'coverage.json').unlink()
+        (original / 'coverage.json').mkdir()
+        try:
+            report_failure = run('coverage-failed-report', ['utgen', 'coverage', '-p', str(original)], original, expected=1)
+            assert 'report --json' in report_failure.stderr and 'panicked' not in report_failure.stderr
+            assert snapshot(original) == baseline
+        finally:
+            (original / 'coverage.json').rmdir()
+    finally:
+        lib.write_bytes(saved_lib)
+    assert snapshot(original) == baseline
     state = {'generation': 0, 'repair': 0, 'errors': []}
 
     class Model(http.server.BaseHTTPRequestHandler):
@@ -160,7 +215,7 @@ def do_POST(self):
                         'threshold': 'assert!(threshold(2));',
                         'double': 'assert_eq!(double("bad"), 4);',
                     }[function]
-                    answer = '#[test]\nfn generated() {\n    ' + body + '\n}\n'
+                    answer = 'fn generated_helper() -> i32 { 1 }\n#[test]\nfn generated() {\n    assert_eq!(generated_helper(), 1);\n    ' + body + '\n}\n'
                     state['generation'] += 1
                 else:
                     file = re.search(r'You can only modify lines \d+ to \d+ in file (.+?)\. For your answer', user).group(1)
@@ -194,6 +249,12 @@ def do_POST(self):
         assert snapshot(target) == prepared
         run('repair', ['utgen', 'fix', '-p', str(target)])
         assert snapshot(target) == prepared
+        # The last repaired candidate's raw data remains after per-function parsing.
+        # Exporting it again must not run tests or contain llmtests/helper records.
+        run('generated-coverage-report', ['cargo', 'llvm-cov', 'report', '--json', '--output-path', 'generated-coverage.json'])
+        generated_coverage = json.loads((target / 'generated-coverage.json').read_text())
+        generated_functions = [function['name'] for data in generated_coverage['data'] for function in data['functions']]
+        assert generated_functions and not any('llmtests' in name or 'generated_helper' in name for name in generated_functions), generated_functions
         assert not state['errors'], state
         assert state['generation'] == 6 and state['repair'] >= 1, state
         for directory in ['result', 'fixed_result']:
@@ -203,6 +264,8 @@ def do_POST(self):
                 assert all(r['tests_compiled'] == r['tests'] and r['tests_passed'] == r['tests_run'] and r['tests_run'] > 0 for r in results), results
             classify = next(r for r in results if r['function_name'] == 'palm_fixture::classify')
             assert classify['branches_covered'] == 2 and classify['branches'] == 2, classify
+            assert all(not (branch['positive'] and branch['negative'])
+                       for _, branches in classify['codes_branches_covered'] for branch in branches), classify
         double = json.loads((target / f"utgen/result/{names['palm_fixture::nested::double']}.json").read_text())
         assert double['tests_compiled'] == 0 and double['tests'] == 1, double
         assert not list((target / 'src').rglob('*.bak'))