From 1f1f131d8935d5ab337bea92019c4288dca37972 Mon Sep 17 00:00:00 2001 From: cppbear Date: Fri, 9 Oct 2026 00:09:58 +0800 Subject: [PATCH] docs: turn validation records into current check guides --- README.md | 2 +- docs/README.md | 4 +- docs/build-validation.md | 81 +++++++++++------------ docs/bytes-analysis.md | 57 +++++++--------- docs/minimal-pipeline.md | 90 +++++++++++++------------- docs/palm-rust-unit-test-generation.md | 2 +- 6 files changed, 107 insertions(+), 129 deletions(-) diff --git a/README.md b/README.md index aeded442b6..042f72c7fa 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,7 @@ cargo build --workspace --locked cargo test --workspace --locked ``` -Default tests use local model-response fixtures and do not contact a model service. The real-service test is opt-in; see [utgen testing](utgen/README.md#testing). Dependency downloads may still require network access. [Build validation](docs/build-validation.md) records the baseline and checks performed. +Default tests use local model-response fixtures and do not contact a model service. The real-service test is opt-in; see [utgen testing](utgen/README.md#testing). Dependency downloads may still require network access. See [build and installation checks](docs/build-validation.md) for commands and CI scope. The [GitHub Actions workflow](.github/workflows/ci.yml) runs on pull requests and pushes to main, canceling superseded runs for the same PR or branch. Changes limited to READMEs and the selected documentation paths under `docs/` receive a lightweight patch whitespace check. Other changes and manual workflow runs execute the full offline suite on Linux. diff --git a/docs/README.md b/docs/README.md index 9c1b809551..7e6b489b6b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,10 +10,10 @@ PALM's implementation, technical guide, and conference materials are available i | [License](../LICENSE) | MIT license for PALM. | | [Technical guide in Chinese](palm-rust-unit-test-generation.md) | Condition chains, context construction, generation, repair, and usage. | | [Minimal pipeline](minimal-pipeline.md) | Deterministic program-analysis, generation, repair, and coverage validation. | -| [bytes analysis](bytes-analysis.md) | Release-tool analysis baseline, method identity checks, and timings. | +| [Bytes analysis](bytes-analysis.md) | Analysis artifacts, method identities, constructor context, and timing output. | | [bytes example](../examples/README.md) | Commands for analyzing a working copy of the bundled target. | | [utgen reference](../utgen/README.md) | Configuration, CLI behavior, and output files. | | [ASE 2025 materials](ase2025/README.md) | Original poster and presentation PDFs. | -| [Build validation](build-validation.md) | Fixed-toolchain baseline, runtime-configuration checks, and validation limits. | +| [Build and installation checks](build-validation.md) | Build, tests, CLI entry points, installation, and CI scope. | The conference PDFs describe the work presented in 2025. The technical guide and command reference describe the implementation maintained here; the current code includes changes made after that presentation. diff --git a/docs/build-validation.md b/docs/build-validation.md index 653163dc5e..11083331a7 100644 --- a/docs/build-validation.md +++ b/docs/build-validation.md @@ -1,63 +1,54 @@ -# Build and Runtime Configuration Validation +# Build and Installation Checks -Validation date: 2026-10-07. +Use these checks to verify PALM's build, command entry points, and runtime configuration. Follow the [project prerequisites](../README.md#prerequisites) first. Commands below start in the repository root and use the pinned toolchain and `Cargo.lock`. -This maintenance batch establishes a build and test baseline after the repository integration. It does not validate the full program-analysis, generated-test execution, and coverage pipeline. +## Build and Test -## Baseline +```sh +cargo build --workspace --locked +cargo test --workspace --locked +``` -The baseline is commit `51a644d9e8c0b26c352726b725f9e847d161f5fc`, exported to a separate temporary directory before editing. The pinned toolchain was installed with `rust-src`, `rustc-dev`, and `llvm-tools-preview`. +Neither command needs an `api.json` file or model credentials. Ordinary tests use local HTTP fixtures to check configuration selection, CLI validation, request handling, retries, and budgets. The real-service test is ignored by default; see [utgen testing](../utgen/README.md#testing) for its explicit opt-in command. -| Item | Value | -| --- | --- | -| Local platform | macOS, Apple Silicon (`aarch64-apple-darwin`) | -| Compiler | `rustc 1.87.0-nightly (75530e9f7 2025-03-18)` | -| Toolchain | `nightly-2025-03-19` | -| Dependency resolution | Existing `Cargo.lock`, with `--locked` | +Dependency downloads can require network access. Passing these checks confirms the tested build and request behavior, not compatibility with a particular model service. -| Baseline check | Result | -| --- | --- | -| `cargo build --workspace --locked`, without `utgen/res/api.json` | Failed with exit 101: `include_str!` could not read `api.json`. | -| Same build, with a non-secret placeholder configuration in the temporary copy | Passed. No request was made to the placeholder endpoint. | -| `cargo test --workspace --locked -- --skip gene::llm::tests::test_llm` | Passed: two ordinary tests; the existing real-model test was explicitly filtered out. | +## Command Entry Points -The build emits existing compiler warnings. The baseline confirms that the missing configuration file was a build dependency; it does not establish that the runtime pipeline is correct. +After the workspace build: -## Runtime Configuration +```sh +bash -n install.sh docker/docker-build docker/docker-run +./install.sh --help +./target/debug/cargo-brinfo --help +./target/debug/cargo-call-chain --help +./target/debug/focxt --help +./target/debug/utgen --help +cargo metadata --manifest-path examples/bytes/Cargo.toml --no-deps --format-version 1 +``` -`utgen gen` and `utgen fix` now load configuration once at command startup and pass it into generation and repair tasks. The file is selected by `--config`, or by `PALM_CONFIG` when the option is absent. `PALM_API_BASE`, `PALM_API_KEY`, and `PALM_MODEL` override individual file fields. No implicit configuration-file search is performed. +These commands check shell syntax, CLI startup, and the bundled example's separate workspace. If using a custom Cargo target directory, adjust the binary paths. Shell syntax checks do not build or run the Docker image. -The legacy `base`, `key`, and `model` JSON fields remain supported. Relative configuration paths are resolved against the invoking directory, independently of the target path. Missing, empty, unreadable, or malformed configuration fails before these commands modify the target tree. This validates configuration loading, not service availability. +## Installation -Prompt templates and request parameters are unchanged. Model configuration is no longer embedded in the binary. The real-service smoke test is ignored by default and covers the non-streaming request used by the production workflow. +To check an actual installation, run: -## Checks Completed in This Batch +```sh +./install.sh +cargo brinfo --help +cargo call-chain --help +focxt --help +utgen --help +``` -| Check | Result | -| --- | --- | -| Workspace build with no `api.json` | Passed on the pinned nightly. | -| Default workspace tests with no model credentials | Passed: 13 tests after the proxy-isolation fix, one real-service test ignored. | -| Configuration tests | File loading and reload, explicit-file precedence, environment-selected file, field overrides, environment-only configuration, missing/empty fields, unreadable file, and redacted diagnostics passed. | -| Local HTTP model fixtures | Generation and repair request paths use the runtime endpoint, key, and model; existing request parameters and usage parsing are preserved. | -| Proxy isolation | The new subprocess regression failed before the fix and passed after it. Both local request fixtures explicitly bypass system proxies; the full test suite also passed with upper/lowercase proxy variables set to `http://127.0.0.1:9` and both `NO_PROXY` variants empty. | -| CLI tests | Help and non-model commands do not load model configuration; missing configuration stops both `gen` and `fix` before target modification; `--config` works before or after the subcommand. | -| Installer behavior checks with a substitute Cargo executable | Other working directories, paths with spaces, tool selection, validation before installation, help, preserved diagnostics/exit status, and stopping after failure passed. | -| Actual installation | All four packages installed without model configuration into an isolated installation directory; existing user-installed tools were not replaced. | -| Installed CLI checks | `cargo-brinfo --help`, `cargo-call-chain --help`, `focxt --help`, and `utgen --help` passed from outside the repository. | -| Shell syntax | `bash -n install.sh docker/docker-build docker/docker-run` passed. | +The installer installs all four tool packages using the repository's toolchain and stops on the first Cargo failure. Keep the installation directory in `PATH`. The [installation guide](../README.md#installation) also describes selecting individual tools. -The installer retains Cargo output, uses the repository directory to select its toolchain, and no longer runs an unconditional `cargo clean`. +Model configuration is loaded only when generation or repair starts. See the [runtime configuration reference](../utgen/README.md#prerequisites) for file selection and environment overrides. -The proxy regression configures only child-process environments, so parallel tests do not change one another's environment. The ordinary LLM constructor retains the SDK's default proxy behavior. The fixtures inject a separate HTTP client with proxies disabled. A test-only `reqwest` dependency reuses the existing locked version; dependency versions, checksums, and lockfile format were not changed. +## CI and Further Checks -The [GitHub Actions workflow](../.github/workflows/ci.yml) configures the build, default tests, CLI help checks, installer syntax/help, and standalone bytes metadata checks on Ubuntu 24.04. Its YAML was parsed locally. The [Linux branch run](https://github.com/cppbear/PALM/actions/runs/37619301888) and [PR run](https://github.com/cppbear/PALM/actions/runs/37619401144) passed before [PR #5](https://github.com/cppbear/PALM/pull/5) was merged. Subsequent pipeline validation is recorded in [minimal pipeline](minimal-pipeline.md). +The [CI workflow](../.github/workflows/ci.yml) checks builds, ordinary tests, CLI entry points, installer syntax/help, and example metadata on Linux. It also runs the [minimal pipeline and related regressions](minimal-pipeline.md) and [Bytes analysis checks](bytes-analysis.md). Actual tool installation and Docker image validation are separate checks. -## Validation Limits and Next Batch +Changes limited to READMEs and the documentation paths selected in the workflow receive a patch whitespace check. Other changes and manual workflow runs execute the full offline suite. A documentation-only CI result does not imply that Rust tests ran. -- No real model service was contacted. Local fixtures validate request construction and response handling, not model compatibility or generation quality. -- The tests added here do not exercise preprocessing correctness, focal-context completeness, generated-test insertion, compilation repair, or coverage measurement end to end. -- Docker was not built: the local Docker daemon was unavailable. The basic Linux CI passed; the Docker image itself remains unverified. -- The pinned toolchain and existing dependency versions were retained; `Cargo.lock` adds only the test dependency edge described above. Compatibility of the coverage tool and target-crate dependencies needs separate verification. -- At this baseline, `utgen analyze` was a logging-only command. The following batch connects analysis and adds a minimal end-to-end check; see [minimal pipeline](minimal-pipeline.md). Concurrency, general recovery, and cache invalidation remain follow-up work. - -The next batch should use a small target crate to verify the relationship between prepared source files, analysis locations, function identities, focal context, and generated tests. It should establish failure propagation and coverage semantics before proceeding to a large bytes run. +The offline checks use temporary target copies and local model responses. They cover the scenarios documented in each guide; they do not measure model quality, provide general crash recovery, or establish support for arbitrary Rust projects. diff --git a/docs/bytes-analysis.md b/docs/bytes-analysis.md index be992e533f..da78d06a2d 100644 --- a/docs/bytes-analysis.md +++ b/docs/bytes-analysis.md @@ -1,52 +1,39 @@ -# bytes Analysis Validation +# Bytes Analysis Validation -This check covers the complete analysis command on the bundled bytes 1.10.0 crate with its default features. PALM tools are built in release mode with the pinned `nightly-2025-03-19`; Cargo's target-analysis profile remains unchanged. No model service or coverage tool is required. +This offline check exercises preprocessing and analysis on the bundled bytes 1.10.0 crate with default features. It also checks method identity and constructor context in small fixtures. No model service or coverage tool is required. -## Plan and Acceptance +## Run the Check -The plan is to measure the existing release build, correct observed analysis blockers, and validate two fresh working copies before expanding to model experiments. Acceptance requires: - -- Preprocessing and `utgen analyze` return success on both copies. -- All 663 exported branch functions have corresponding compiler identities, raw/normalized dependency files, and nonempty context files. No function is dropped to satisfy this count. -- Branch code and compiler-exported method code match their source locations; each context contains its focal function's declaration. -- A small separate crate distinguishes generic trait implementations, reference receivers, a trait default method, and a macro-generated forwarding method. Two impls share one source line, and an inactive `cfg` variant must not replace the active method. -- Analysis leaves prepared sources and the manifest unchanged. The bundled example is also unchanged. - -The review identified two traps to avoid: debug-tool timings do not establish release performance, and nonempty context files alone do not establish correct method identity. The check therefore uses release tools and also verifies selected method bodies and their impl headers. Its 300-second per-command limit is an external regression-test deadline, not a new PALM timeout or a performance guarantee. - -## Changes - -The baseline emitted 663 function identities but only 226 contexts. Trait default contexts were constructed without being written. Matching methods by short module/type/trait names also conflated generic trait instantiations and reference receivers. Source parsing alone missed methods supplied by forwarding macros. - -Call-chain now includes each function's source span, enclosing impl header span, and already extracted source code in its index. The source parser binds functions to these compiler identities before building the dependency lookup. Compiler-selected methods with parseable source are added to the matching impl when absent from the source AST. Inactive method bodies are excluded, while required trait signatures are retained. Context files use the compiler's existing encoded names, and trait default contexts are written normally. - -Recursive dependency JSON reads use `BufReader`. No persistent cache or new cache invalidation scheme is introduced. `utgen analyze` reports the duration of clean, brinfo, and focxt; a missing context error identifies the function and path. - -## Reproduce - -From the repository root, with the pinned compiler and components installed: +Follow the [project prerequisites](../README.md#prerequisites), then run from the repository root: ```sh cargo build --workspace --release --locked python3 scripts/check_bytes.py ``` -Use `--bin-dir ` for a custom tool build directory and `--runs 1` for one bytes copy. Rebuild the tools together: focxt consumes the source metadata emitted by the matching call-chain binary. The script prints a temporary directory containing copies, per-command logs, stage timings, and `summary.json`. Its default is two copies; dependency downloads can require network access. +The script uses release tool binaries from `target/release` and analyzes two fresh Bytes copies by default. Use `--bin-dir ` for another binary directory or `--runs 1` for one Bytes copy. The method-identity and constructor fixtures run in either case. Build the tools together so focxt and call-chain use matching analysis data. + +The script prints its temporary working directory and finishes with `All bytes analysis checks passed.` Dependency downloads can require network access. The bundled example remains unchanged. + +## Checked Behavior -## Local Results +| Area | Acceptance | +| --- | --- | +| Bytes artifacts | All 663 exported functions have matching branch/context indices, dependency files, and nonempty context files containing their focal declarations. | +| Source identity | Exported function code matches its source location, accounting for brinfo's indentation normalization. | +| Method identity | Distinct generic trait implementations, reference receivers, default methods, and recoverable macro-generated methods retain the correct bodies and impl headers. Inactive `cfg` alternatives do not replace active methods. | +| Constructor context | Constructors returning `Self` or the concrete impl type retain their bodies, including the checked generic and macro-generated cases. Unrelated methods remain subject to context trimming. | +| Function lists | The checked-in two- and eight-function selections match the analysis index and their expected representative condition chains. | +| Source preservation | Analysis leaves the prepared sources and manifest unchanged. | -Measured on macOS/Apple Silicon with the pinned compiler. Tool compilation is excluded from the analysis times below. +## Logs and Timings -| Run | Analyze | brinfo | focxt | Branch functions / contexts | Outcome | -| --- | ---: | ---: | ---: | ---: | --- | -| Previous main, release | 43.714 s | — | — | 663 / 226 | Missing context, exit 1 | -| Corrected, first copy | 34.632 s | 2.232 s | 32.297 s | 663 / 663 | Passed | -| Corrected, second copy | 34.747 s | 2.235 s | 32.396 s | 663 / 663 | Passed | +The working directory contains the fixture copies, per-command logs, and `summary.json`. The summary records platform/compiler information, function/context counts, preprocessing time, total analysis time, and the clean/brinfo/focxt stage times for each Bytes run. -The before/after rows perform different amounts of successful work, so they are not a speedup benchmark. Both corrected copies passed the source and artifact checks; the method-identity fixture also passed. The check is part of Linux CI alongside the existing minimal pipeline, coverage compatibility, and scheduling regressions. +Tool compilation is excluded from those analysis timings. Compare runs using the same tool build profile and target configuration; the figures are measurements for that environment, not a performance guarantee. The script applies an external 300-second limit to each command. That limit is part of the regression harness, not a PALM CLI timeout. ## Scope -The 663 functions are the functions exported by the current analyzers for this pinned crate/configuration, not a claim to cover every possible Rust function or feature combination. Recovering an already selected method from parseable source does not provide general macro expansion: a macro-generated impl or source containing unexpanded substitutions may still be unsupported. Required trait methods without bodies are context declarations rather than focal functions. +The 663 functions are those exported for this pinned crate and configuration. They do not establish coverage of every Rust function or feature combination. Recovering a selected method from parseable source does not provide general macro expansion; macro-generated impls or unexpanded substitutions may remain unsupported. Required trait methods without bodies are context declarations rather than focal functions. -Full bytes generation, repair, model quality, and coverage experiments remain the next validation stage. This batch does not change their evaluation semantics. +This check runs in the full Linux CI suite. It verifies analysis artifacts and selected context behavior, without generating tests, performing compilation repair, or measuring model quality. See [minimal pipeline validation](minimal-pipeline.md) for deterministic generation/repair/coverage checks and the [small model trial](../examples/README.md#prepare-a-small-model-trial) for scoped experiments. diff --git a/docs/minimal-pipeline.md b/docs/minimal-pipeline.md index 083bad78a6..9ab845936e 100644 --- a/docs/minimal-pipeline.md +++ b/docs/minimal-pipeline.md @@ -1,86 +1,86 @@ # Minimal Pipeline Validation -This batch follows the [build and configuration baseline](build-validation.md), merged through [PR #5](https://github.com/cppbear/PALM/pull/5). It validates one standalone crate with fixed local model responses, the pinned nightly, and `cargo-llvm-cov 0.6.16`. +These offline checks exercise PALM with small standalone crates, real Rust tools, and fixed local model responses. They verify analysis, generation, compilation repair, execution, coverage, and cleanup. No model credentials or external model calls are needed; dependency downloads may require network access. ## Run the Check -From the repository root: +Follow the [project prerequisites](../README.md#prerequisites), then run from the repository root: ```sh cargo build --workspace --locked -cargo +stable install cargo-llvm-cov --version 0.6.16 --locked -python3 scripts/check_coverage.py python3 scripts/check_minimal.py -python3 scripts/check_tasks.py ``` -No model credentials are required. The model endpoint is a temporary loopback HTTP server owned by the script; target commands receive only a dummy key. Dependency downloads may require network access. - -The script prints the temporary directory containing source copies, command logs, model-request counts, coverage, and generated results. Use `--work-dir ` to select it, `--bin-dir ` for a custom tool build directory, or `--analysis-only` to stop before coverage/generation. Work directories must be new. The [fixture](../examples/minimal/README.md) is never modified. - -## Verified Contracts - -- Original tests run on a separate original copy; coverage annotations are temporary and restored. A separate prepared copy is used for generated tests. -- Preprocessing removes test-only source ranges while preserving newlines and byte offsets, including a test and production function sharing one line. Repeating preprocessing is idempotent. -- `utgen analyze` succeeds even after a prior `cargo check`, since it clears the check cache before invoking the compiler wrapper. It rejects existing analysis output directories. -- brinfo and focxt agree on five complete function names. Source locations identify the exported focal code, accounting for brinfo's indentation normalization. -- The `Gauge::label` context contains its struct, the focal method, and the declarations for `classify`, `positive`, and the transitive `threshold` dependency. -- Missing context and malformed source locations fail before generation changes the manifest or source files. Source-location parse errors no longer panic. -- Six model responses produce six candidate tests. One candidate deliberately passes a string to an integer parameter and fails compilation; a numbered ChangeLog response repairs it through the actual repair path. -- Coverage executes each test body once for XML and JSON. Test functions, module helpers, standalone test-only helpers, and generated `llmtests` records are excluded; production coverage remains. -- Sources are restored after successful coverage, assertion failures, compilation failures, and JSON export failures. -- Both branches of `classify` are covered. Before repair, the intentionally invalid `double` candidate has zero compilable tests. After repair, all candidates compile and pass. -- Direct byte comparisons verify that source files and the target manifest match their prepared state after generation and repair; no source backup remains after successful completion. +The script uses the [minimal fixture](../examples/minimal/README.md) in temporary copies. It prints the working directory and finishes with `Minimal analysis/generation/repair/coverage checks passed.` The checked-in fixture remains unchanged. -The regression script runs in Linux CI after the ordinary workspace tests. It uses the real analyzers, compiler, coverage tool, prompt builder, response parser, candidate checks, and repair machinery. Model replies are deterministic fixtures; these results do not measure LLM quality. +Run the related checks when changing coverage/preprocessing or task scheduling: -Generation, repair, and statistics no longer call the partial `target_clean` helper between candidates. Cargo reuses build artifacts as sources are replaced and restored; analysis still clears the check cache, and coverage retains its own profile cleanup. The task regression preserves the target directory across scenarios and checks compilation failures followed by successful repairs. +```sh +python3 scripts/check_coverage.py +python3 scripts/check_tasks.py +``` -Cached-candidate checks exercise the shared five-second test timeout in unit and integration statistics without additional model requests. Ordinary and `should_panic` infinite loops must count as failures, while an expected panic and a later normal test pass. Both production branches must remain covered and temporary source edits must be restored. The regression harness terminates its subprocess group if a check hangs; this external deadline is separate from the candidate timeout. +All three scripts are included in the full Linux CI suite. Their options differ: -## Coverage Interpretation +| Script | Options | +| --- | --- | +| `check_minimal.py` | `--bin-dir `; `--work-dir `; `--analysis-only` to stop after analysis checks. | +| `check_coverage.py` | `--bin-dir `. | +| `check_tasks.py` | `--bin-dir `. | -`original/original-coverage.json` and `.xml` measure production code exercised by all original unit and integration tests through `utgen coverage`. Test bodies and helpers contribute to neither the coverage numerator nor denominator. The check requires both original `classify` branches to have executed. LLVM can emit separate records for the same branch location in different test binaries; the check unions those outcomes instead of requiring both in one record. +The default binary directory is `target/debug`. The coverage tool must be available in `PATH` at the version specified in the prerequisites. `--work-dir` must name a new directory. -`generated/utgen/result/` and `generated/utgen/fixed_result/` contain the tool's per-function statistics for generated candidates. The script checks the known branch count and execution outcomes directly. The separate legacy `utgen/original_result.json` restores only the integration-test backup; it must not be presented as coverage of all original tests. Combined original-plus-generated coverage is not part of this check. +## Verified Contracts -## Changes and Limits +| Area | Checked behavior | +| --- | --- | +| Preprocessing | Removes test-only source ranges, preserves byte offsets and newlines, and is idempotent on the prepared fixture. | +| Analysis | Runs after a prior `cargo check`; rejects old analysis directories; brinfo and focxt agree on all five fixture functions and their source locations. | +| Context | Includes the focal method, its type, and the fixture's transitive local dependencies. | +| Generation and repair | Fixed responses produce candidates; an intentionally invalid candidate fails compilation and is repaired using compiler diagnostics. | +| Coverage | Excludes test bodies and test-only helpers, preserves production coverage, and executes tests once when exporting XML and JSON. | +| Cleanup | Restores sources after the covered success, assertion-failure, compilation-failure, and report-export-failure cases. | +| Candidate timeout | Ordinary and `should_panic` infinite loops count as failures; an expected panic and a later ordinary test pass in unit and integration statistics. | +| Integration directories | Original and generated tests coexist after normal runs and cached reruns; original-test compilation failure restores both directories; an existing staging directory is preserved and reported. | -`utgen analyze` now invokes the tools and validates their artifacts. Its supported scope is one standalone crate with `Cargo.toml` and `src/`. Generation rejects incomplete indices/context and propagates failed asynchronous generation tasks. Repair validates the analysis and pre-repair JSON before starting. +The integration-directory cases retain the original `tests.bak/` while generating `tests/`. They use one cached candidate and make no additional model requests. Separate timeout cases also check integration execution without an original-test backup. -The preprocessor parses `cfg` predicates conservatively, preserves `cfg(not(test))` and unknown production features, handles nested test ranges and UTF-8 safely, and refuses to overwrite `tests.bak`. Whitespace replacement preserves bytes/newlines, not necessarily character columns inside removed UTF-8 ranges; analysis runs after preprocessing. +## Outputs and Coverage Interpretation -Recursive context lookup now resolves full function names through the compiler's encoded-name index and consumes the accumulated dependencies. External functions without local artifacts remain outside this source context. +The minimal check's temporary directory contains `logs/`, separate `original/` and `generated/` copies, `model-requests.json`, and a success summary. The coverage-compatibility check keeps logs beside each temporary case; the task check prints its own working directory and writes logs and `summary.json` there. On failure, inspect the command log named by the script. -The local macOS/Apple Silicon run passed using six generation responses and one repair response. The [bytes analysis check](bytes-analysis.md) separately validates the larger target without a model. Real-model behavior, full bytes generation/repair, general multi-crate support, hard process timeouts, recovery after forced termination, and explicit run/resume behavior remain later work. Git history is retained. +`original/original-coverage.json` and `.xml` measure production code exercised by the original unit and integration tests through `utgen coverage`. Test bodies and identified test-only helpers contribute to neither the coverage numerator nor denominator. The check combines branch outcomes when LLVM reports the same location in multiple test binaries. -The coverage entry temporarily annotates sources with `#[coverage(off)]` and enables the nightly feature at crate roots without adding newlines. It preserves compiler flags, restores original bytes on returned errors, and performs no hashing or source fingerprint checks. `cargo llvm-cov report --json` reuses the first run's data; the first XML-producing command retains the tool's normal profile cleanup between candidates. This does not provide crash recovery or general macro expansion. See the [coverage command reference](../utgen/README.md#coverage-of-existing-tests) for scope. +`generated/utgen/result/` and `generated/utgen/fixed_result/` contain per-function statistics for generated candidates before and after repair. The separate `utgen/original_result.json` measures only the restored original integration tests on prepared source. It is not the complete original-test baseline. Combined original-plus-generated coverage is not measured by these checks. ## Coverage Compatibility Regression -`scripts/check_coverage.py` exercises 11 small standalone crates, without a model service or extra crate dependencies. It covers existing coverage attributes, `cfg_attr(test, ...)`, compound/nested conditions with Cargo features enabled and disabled, test-only methods and impls, file-level test modules, and an expression used by `include!`. Each case has an integration test so both normal-library and test-harness builds are checked. It compares execution outcomes, checks raw production-only coverage, verifies source restoration, and confirms preprocessing leaves no original tests while preserving byte offsets. A custom target entry containing an expression is rejected by both commands. +`check_coverage.py` checks existing coverage attributes, conditional attributes with Cargo features enabled and disabled, test-only functions/methods/impls, file-level test modules, and expression fragments used by `include!`. Integration tests exercise both ordinary-library and test-harness compilation. The cases verify execution outcomes, production-only coverage, source restoration, and preprocessing offsets. -The script runs in Linux CI. Coverage and preprocessing share only source parsing and test predicates; they keep separate edit visitors. Coverage attributes and their feature gate are added under the complement of existing conditions, without a cfg evaluator or compiler-flag overrides. Explicit `coverage(on)` remains an intentional opt-in. Expression fragments are preserved, without macro expansion or automatic test exclusion inside those fragments. +Explicit `coverage(on)` is respected. Expression fragments are preserved; the checks do not provide general macro expansion or infer that arbitrary unmarked helpers are test-only. See the [coverage reference](../utgen/README.md#coverage-of-existing-tests) for supported scope. ## Task Scheduling Regression -`scripts/check_tasks.py` uses nine focal functions sharing one source file, a loopback model, and real Cargo commands. Nine-function runs exercise N=1 and the default N=4; N=2 uses five functions. Each scheduling case has more than 2N functions to exercise queued work. The check requires overlap for N>1 and verifies that compiler invocations never overlap or observe changing source bytes. Import-handling and model-request failure cases use the two-function selection. A paused compiler lets the N=1 generation queue fill: one candidate is being validated, one result is buffered, and one active job waits to send. Releasing compilation must drain every result. - -The same check covers direct and input/prefix/oracle integration generation, removal of invalid imports from saved candidates, and preservation/removal of temporary compiler inputs. Integration coverage reports remain available until every selected function has been read, and each must have passing tests and nonzero covered lines. Valid answers without usage must complete successfully and mark request statistics incomplete. HTTP 401 during generation, oracle generation, and repair must fail without outer-layer retries. A blocked prompt directory exercises worker-panic propagation and slot release; a stale generation backup exercises queue draining after consumer failure, and blocking the diagnostic output file exercises failure while the repair lock is held. Sources must be restored, failed repair backups retained, unrelated backups untouched, existing recovery material rejected, and post-failure statistics skipped. These checks use byte comparisons without hashing and run on macOS/Linux. CLI tests separately verify the default and rejection of zero before configuration loading. +`check_tasks.py` exercises function limits N=1, N=2, and the default N=4, with more than 2N functions in each scheduling case. It checks overlapping model requests, serial compiler activity, bounded-queue backpressure, and completion after queued work resumes. Cargo's target directory is reused between scenarios. +Additional cases cover direct and input/prefix/oracle generation, integration imports, function selection, request budgets, and worker/consumer failures. Checks require source restoration, preservation of unrelated backups, retention of repair backups on failure, and skipped statistics after a candidate-stage error. ## Model Request Regression -Local HTTP tests exercise the shared generation/repair request path, including response usage omissions, empty choices or content, invalid JSON, permanent 4xx errors, transient 429/5xx errors, dropped connections, and delayed response bodies. They verify actual attempt counts and bounded retry exhaustion without calling a real service. Existing task checks also verify request summaries and source/backup handling after a final request failure. `--request-timeout` defaults to 180 seconds per attempt and rejects zero before configuration loading. Request formatting and compilation repair retries remain business-level operations, not additional transport retry layers. +Workspace unit tests use local HTTP fixtures for missing usage, empty or malformed responses, permanent errors, transient errors, dropped connections, and request timeouts. They check attempt counts and retry exhaustion. The task regression checks request summaries and cleanup after request failures. See the [request configuration reference](../utgen/README.md#prerequisites) for the current retry policy. ## Function Selection and Request Budgets -`scripts/check_tasks.py` generates and repairs just two selected functions while other saved candidates remain in the same directories. It checks actual request counts, selected-only statistics, untouched unselected candidate files, and invocation records. Duplicate names and blank lines are accepted. Separate CLI tests reject empty/unknown lists and a zero request limit before configuration or target changes. - -With four function workers and a two-attempt limit, both generation and repair must send exactly two requests, return failure when more work needs a request, restore sources, and skip statistics. Local HTTP unit tests additionally cover retries consuming the limit, concurrent clones sharing it, and success on the final allowed attempt. None of these checks uses real model credentials. The bytes analysis check validates the checked-in two- and eight-function lists against the complete analysis index for a later model trial. +The task checks select functions while leaving other cached candidates untouched, verify that statistics follow the same selection, and check invocation records. Concurrent generation and repair must respect the shared attempt budget, including retries. CLI and request tests cover invalid selections, invalid limits, and successful completion on the final allowed attempt. ## Answers Without Tests -Before this fix, a reply such as `fn helper() {}` parsed as Rust and let generation return success with zero candidate tests. Direct-test and prefix generation now require at least one extracted `#[test]` function and reuse the existing three-attempt format retry loop, with a fixed one-second wait before each retry and no wait before the first attempt. This does not require assertions in a prefix or judge the semantic quality of a test. +Direct-test and prefix cases check recovery after a helper-only response, exhaustion of format retries, and interruption by the request budget. Raw answers remain available before normalization. Prefixes can omit assertions because oracle generation follows separately; final candidates still undergo compilation and execution checks. + +## Scope + +The fixture checks verify deterministic behavior, not model quality or completeness for arbitrary Rust features. General multi-crate operation, macro expansion, and recovery after forced termination are outside their coverage. + +The five-second candidate timeout applies to test bodies. Request timeouts and the regression harness's subprocess deadlines are separate; passing these checks does not establish a general Cargo-process deadline. Expected-panic statistics retain the current any-panic behavior described in the [generation reference](../utgen/README.md#generation). -The task regression selects one function and exercises recovery after a helper-only reply, exhaustion after three such replies, and interruption by the shared request budget, in both generation paths. It compares saved raw answers with the server's replies, including code fences, verifies the accepted code is normalized, checks request/token counts, and requires failures to leave no successful candidate or statistics. The recovered tests must compile, pass and cover production code, including a prefix with no assertion before oracle generation. Unit tests additionally cover invalid Rust, imports/comments without tests, empty test modules, and existing helpers/test attributes. All fixtures use fresh generation directories and local HTTP; no full-project model trial is involved. +Use [Bytes analysis validation](bytes-analysis.md) for a larger analysis target. For model-based experiments, follow the separately scoped [two-function trial](../examples/README.md#prepare-a-small-model-trial) and inspect request reports and test outcomes. diff --git a/docs/palm-rust-unit-test-generation.md b/docs/palm-rust-unit-test-generation.md index a23a3af9ea..2c76ad8ab7 100644 --- a/docs/palm-rust-unit-test-generation.md +++ b/docs/palm-rust-unit-test-generation.md @@ -643,7 +643,7 @@ utgen gen -p "$palm_example_dir" --requirement --context utgen fix -p "$palm_example_dir" ``` -切换到 integration、其他提示词或其他模型时，使用新的目标副本并重新分析，避免复用旧的生成缓存。更详细的示例见 [examples/README.md](../examples/README.md)。当前完整生成与修复回归使用 [minimal 示例](minimal-pipeline.md)。[bytes 分析验收](bytes-analysis.md) 已覆盖默认配置下导出的 663 个函数及其上下文；bytes 的真实模型生成、修复和覆盖率实验仍是后续验证项。 +切换到 integration、其他提示词或其他模型时，使用新的目标副本并重新分析，避免复用旧的生成缓存。更详细的示例见 [examples/README.md](../examples/README.md)。[最小流程验证](minimal-pipeline.md)使用固定本地响应检查生成、修复、执行和覆盖率；[Bytes 分析检查](bytes-analysis.md)核对默认配置下导出的 663 个函数及其上下文。真实模型实验按示例中的函数清单和请求额度单独开展，并检查请求报告及测试结果。 完成后重点查看临时目标目录中的以下路径：