Skip to content

docs: refresh validation guides for current workflows #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cppbear merged 1 commit into from
Oct 8, 2026
Merged

docs: refresh validation guides for current workflows #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ cargo build --workspace --locked
cargo test --workspace --locked
```

Default tests use local model-response fixtures and do not contact a model service. The real-service test is opt-in; see [utgen testing](utgen/README.md#testing). Dependency downloads may still require network access. [Build validation](docs/build-validation.md) records the baseline and checks performed.
Default tests use local model-response fixtures and do not contact a model service. The real-service test is opt-in; see [utgen testing](utgen/README.md#testing). Dependency downloads may still require network access. See [build and installation checks](docs/build-validation.md) for commands and CI scope.

The [GitHub Actions workflow](.github/workflows/ci.yml) runs on pull requests and pushes to main, canceling superseded runs for the same PR or branch. Changes limited to READMEs and the selected documentation paths under `docs/` receive a lightweight patch whitespace check. Other changes and manual workflow runs execute the full offline suite on Linux.

Expand Down
4 changes: 2 additions & 2 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ PALM's implementation, technical guide, and conference materials are available i
| [License](../LICENSE) | MIT license for PALM. |
| [Technical guide in Chinese](palm-rust-unit-test-generation.md) | Condition chains, context construction, generation, repair, and usage. |
| [Minimal pipeline](minimal-pipeline.md) | Deterministic program-analysis, generation, repair, and coverage validation. |
| [bytes analysis](bytes-analysis.md) | Release-tool analysis baseline, method identity checks, and timings. |
| [Bytes analysis](bytes-analysis.md) | Analysis artifacts, method identities, constructor context, and timing output. |
| [bytes example](../examples/README.md) | Commands for analyzing a working copy of the bundled target. |
| [utgen reference](../utgen/README.md) | Configuration, CLI behavior, and output files. |
| [ASE 2025 materials](ase2025/README.md) | Original poster and presentation PDFs. |
| [Build validation](build-validation.md) | Fixed-toolchain baseline, runtime-configuration checks, and validation limits. |
| [Build and installation checks](build-validation.md) | Build, tests, CLI entry points, installation, and CI scope. |

The conference PDFs describe the work presented in 2025. The technical guide and command reference describe the implementation maintained here; the current code includes changes made after that presentation.
81 changes: 36 additions & 45 deletions docs/build-validation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,63 +1,54 @@
# Build and Runtime Configuration Validation
# Build and Installation Checks

Validation date: 2026-10-07.
Use these checks to verify PALM's build, command entry points, and runtime configuration. Follow the [project prerequisites](../README.md#prerequisites) first. Commands below start in the repository root and use the pinned toolchain and `Cargo.lock`.

This maintenance batch establishes a build and test baseline after the repository integration. It does not validate the full program-analysis, generated-test execution, and coverage pipeline.
## Build and Test

## Baseline
```sh
cargo build --workspace --locked
cargo test --workspace --locked
```

The baseline is commit `51a644d9e8c0b26c352726b725f9e847d161f5fc`, exported to a separate temporary directory before editing. The pinned toolchain was installed with `rust-src`, `rustc-dev`, and `llvm-tools-preview`.
Neither command needs an `api.json` file or model credentials. Ordinary tests use local HTTP fixtures to check configuration selection, CLI validation, request handling, retries, and budgets. The real-service test is ignored by default; see [utgen testing](../utgen/README.md#testing) for its explicit opt-in command.

| Item | Value |
| --- | --- |
| Local platform | macOS, Apple Silicon (`aarch64-apple-darwin`) |
| Compiler | `rustc 1.87.0-nightly (75530e9f7 2025-03-18)` |
| Toolchain | `nightly-2025-03-19` |
| Dependency resolution | Existing `Cargo.lock`, with `--locked` |
Dependency downloads can require network access. Passing these checks confirms the tested build and request behavior, not compatibility with a particular model service.

| Baseline check | Result |
| --- | --- |
| `cargo build --workspace --locked`, without `utgen/res/api.json` | Failed with exit 101: `include_str!` could not read `api.json`. |
| Same build, with a non-secret placeholder configuration in the temporary copy | Passed. No request was made to the placeholder endpoint. |
| `cargo test --workspace --locked -- --skip gene::llm::tests::test_llm` | Passed: two ordinary tests; the existing real-model test was explicitly filtered out. |
## Command Entry Points

The build emits existing compiler warnings. The baseline confirms that the missing configuration file was a build dependency; it does not establish that the runtime pipeline is correct.
After the workspace build:

## Runtime Configuration
```sh
bash -n install.sh docker/docker-build docker/docker-run
./install.sh --help
./target/debug/cargo-brinfo --help
./target/debug/cargo-call-chain --help
./target/debug/focxt --help
./target/debug/utgen --help
cargo metadata --manifest-path examples/bytes/Cargo.toml --no-deps --format-version 1
```

`utgen gen` and `utgen fix` now load configuration once at command startup and pass it into generation and repair tasks. The file is selected by `--config`, or by `PALM_CONFIG` when the option is absent. `PALM_API_BASE`, `PALM_API_KEY`, and `PALM_MODEL` override individual file fields. No implicit configuration-file search is performed.
These commands check shell syntax, CLI startup, and the bundled example's separate workspace. If using a custom Cargo target directory, adjust the binary paths. Shell syntax checks do not build or run the Docker image.

The legacy `base`, `key`, and `model` JSON fields remain supported. Relative configuration paths are resolved against the invoking directory, independently of the target path. Missing, empty, unreadable, or malformed configuration fails before these commands modify the target tree. This validates configuration loading, not service availability.
## Installation

Prompt templates and request parameters are unchanged. Model configuration is no longer embedded in the binary. The real-service smoke test is ignored by default and covers the non-streaming request used by the production workflow.
To check an actual installation, run:

## Checks Completed in This Batch
```sh
./install.sh
cargo brinfo --help
cargo call-chain --help
focxt --help
utgen --help
```

| Check | Result |
| --- | --- |
| Workspace build with no `api.json` | Passed on the pinned nightly. |
| Default workspace tests with no model credentials | Passed: 13 tests after the proxy-isolation fix, one real-service test ignored. |
| Configuration tests | File loading and reload, explicit-file precedence, environment-selected file, field overrides, environment-only configuration, missing/empty fields, unreadable file, and redacted diagnostics passed. |
| Local HTTP model fixtures | Generation and repair request paths use the runtime endpoint, key, and model; existing request parameters and usage parsing are preserved. |
| Proxy isolation | The new subprocess regression failed before the fix and passed after it. Both local request fixtures explicitly bypass system proxies; the full test suite also passed with upper/lowercase proxy variables set to `http://127.0.0.1:9` and both `NO_PROXY` variants empty. |
| CLI tests | Help and non-model commands do not load model configuration; missing configuration stops both `gen` and `fix` before target modification; `--config` works before or after the subcommand. |
| Installer behavior checks with a substitute Cargo executable | Other working directories, paths with spaces, tool selection, validation before installation, help, preserved diagnostics/exit status, and stopping after failure passed. |
| Actual installation | All four packages installed without model configuration into an isolated installation directory; existing user-installed tools were not replaced. |
| Installed CLI checks | `cargo-brinfo --help`, `cargo-call-chain --help`, `focxt --help`, and `utgen --help` passed from outside the repository. |
| Shell syntax | `bash -n install.sh docker/docker-build docker/docker-run` passed. |
The installer installs all four tool packages using the repository's toolchain and stops on the first Cargo failure. Keep the installation directory in `PATH`. The [installation guide](../README.md#installation) also describes selecting individual tools.

The installer retains Cargo output, uses the repository directory to select its toolchain, and no longer runs an unconditional `cargo clean`.
Model configuration is loaded only when generation or repair starts. See the [runtime configuration reference](../utgen/README.md#prerequisites) for file selection and environment overrides.

The proxy regression configures only child-process environments, so parallel tests do not change one another's environment. The ordinary LLM constructor retains the SDK's default proxy behavior. The fixtures inject a separate HTTP client with proxies disabled. A test-only `reqwest` dependency reuses the existing locked version; dependency versions, checksums, and lockfile format were not changed.
## CI and Further Checks

The [GitHub Actions workflow](../.github/workflows/ci.yml) configures the build, default tests, CLI help checks, installer syntax/help, and standalone bytes metadata checks on Ubuntu 24.04. Its YAML was parsed locally. The [Linux branch run](https://github.com/cppbear/PALM/actions/runs/37619301888) and [PR run](https://github.com/cppbear/PALM/actions/runs/37619401144) passed before [PR #5](https://github.com/cppbear/PALM/pull/5) was merged. Subsequent pipeline validation is recorded in [minimal pipeline](minimal-pipeline.md).
The [CI workflow](../.github/workflows/ci.yml) checks builds, ordinary tests, CLI entry points, installer syntax/help, and example metadata on Linux. It also runs the [minimal pipeline and related regressions](minimal-pipeline.md) and [Bytes analysis checks](bytes-analysis.md). Actual tool installation and Docker image validation are separate checks.

## Validation Limits and Next Batch
Changes limited to READMEs and the documentation paths selected in the workflow receive a patch whitespace check. Other changes and manual workflow runs execute the full offline suite. A documentation-only CI result does not imply that Rust tests ran.

- No real model service was contacted. Local fixtures validate request construction and response handling, not model compatibility or generation quality.
- The tests added here do not exercise preprocessing correctness, focal-context completeness, generated-test insertion, compilation repair, or coverage measurement end to end.
- Docker was not built: the local Docker daemon was unavailable. The basic Linux CI passed; the Docker image itself remains unverified.
- The pinned toolchain and existing dependency versions were retained; `Cargo.lock` adds only the test dependency edge described above. Compatibility of the coverage tool and target-crate dependencies needs separate verification.
- At this baseline, `utgen analyze` was a logging-only command. The following batch connects analysis and adds a minimal end-to-end check; see [minimal pipeline](minimal-pipeline.md). Concurrency, general recovery, and cache invalidation remain follow-up work.

The next batch should use a small target crate to verify the relationship between prepared source files, analysis locations, function identities, focal context, and generated tests. It should establish failure propagation and coverage semantics before proceeding to a large bytes run.
The offline checks use temporary target copies and local model responses. They cover the scenarios documented in each guide; they do not measure model quality, provide general crash recovery, or establish support for arbitrary Rust projects.
57 changes: 22 additions & 35 deletions docs/bytes-analysis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,52 +1,39 @@
# bytes Analysis Validation
# Bytes Analysis Validation

This check covers the complete analysis command on the bundled bytes 1.10.0 crate with its default features. PALM tools are built in release mode with the pinned `nightly-2025-03-19`; Cargo's target-analysis profile remains unchanged. No model service or coverage tool is required.
This offline check exercises preprocessing and analysis on the bundled bytes 1.10.0 crate with default features. It also checks method identity and constructor context in small fixtures. No model service or coverage tool is required.

## Plan and Acceptance
## Run the Check

The plan is to measure the existing release build, correct observed analysis blockers, and validate two fresh working copies before expanding to model experiments. Acceptance requires:

- Preprocessing and `utgen analyze` return success on both copies.
- All 663 exported branch functions have corresponding compiler identities, raw/normalized dependency files, and nonempty context files. No function is dropped to satisfy this count.
- Branch code and compiler-exported method code match their source locations; each context contains its focal function's declaration.
- A small separate crate distinguishes generic trait implementations, reference receivers, a trait default method, and a macro-generated forwarding method. Two impls share one source line, and an inactive `cfg` variant must not replace the active method.
- Analysis leaves prepared sources and the manifest unchanged. The bundled example is also unchanged.

The review identified two traps to avoid: debug-tool timings do not establish release performance, and nonempty context files alone do not establish correct method identity. The check therefore uses release tools and also verifies selected method bodies and their impl headers. Its 300-second per-command limit is an external regression-test deadline, not a new PALM timeout or a performance guarantee.

## Changes

The baseline emitted 663 function identities but only 226 contexts. Trait default contexts were constructed without being written. Matching methods by short module/type/trait names also conflated generic trait instantiations and reference receivers. Source parsing alone missed methods supplied by forwarding macros.

Call-chain now includes each function's source span, enclosing impl header span, and already extracted source code in its index. The source parser binds functions to these compiler identities before building the dependency lookup. Compiler-selected methods with parseable source are added to the matching impl when absent from the source AST. Inactive method bodies are excluded, while required trait signatures are retained. Context files use the compiler's existing encoded names, and trait default contexts are written normally.

Recursive dependency JSON reads use `BufReader`. No persistent cache or new cache invalidation scheme is introduced. `utgen analyze` reports the duration of clean, brinfo, and focxt; a missing context error identifies the function and path.

## Reproduce

From the repository root, with the pinned compiler and components installed:
Follow the [project prerequisites](../README.md#prerequisites), then run from the repository root:

```sh
cargo build --workspace --release --locked
python3 scripts/check_bytes.py
```

Use `--bin-dir <directory>` for a custom tool build directory and `--runs 1` for one bytes copy. Rebuild the tools together: focxt consumes the source metadata emitted by the matching call-chain binary. The script prints a temporary directory containing copies, per-command logs, stage timings, and `summary.json`. Its default is two copies; dependency downloads can require network access.
The script uses release tool binaries from `target/release` and analyzes two fresh Bytes copies by default. Use `--bin-dir <directory>` for another binary directory or `--runs 1` for one Bytes copy. The method-identity and constructor fixtures run in either case. Build the tools together so focxt and call-chain use matching analysis data.

The script prints its temporary working directory and finishes with `All bytes analysis checks passed.` Dependency downloads can require network access. The bundled example remains unchanged.

## Checked Behavior

## Local Results
| Area | Acceptance |
| --- | --- |
| Bytes artifacts | All 663 exported functions have matching branch/context indices, dependency files, and nonempty context files containing their focal declarations. |
| Source identity | Exported function code matches its source location, accounting for brinfo's indentation normalization. |
| Method identity | Distinct generic trait implementations, reference receivers, default methods, and recoverable macro-generated methods retain the correct bodies and impl headers. Inactive `cfg` alternatives do not replace active methods. |
| Constructor context | Constructors returning `Self` or the concrete impl type retain their bodies, including the checked generic and macro-generated cases. Unrelated methods remain subject to context trimming. |
| Function lists | The checked-in two- and eight-function selections match the analysis index and their expected representative condition chains. |
| Source preservation | Analysis leaves the prepared sources and manifest unchanged. |

Measured on macOS/Apple Silicon with the pinned compiler. Tool compilation is excluded from the analysis times below.
## Logs and Timings

| Run | Analyze | brinfo | focxt | Branch functions / contexts | Outcome |
| --- | ---: | ---: | ---: | ---: | --- |
| Previous main, release | 43.714 s | — | — | 663 / 226 | Missing context, exit 1 |
| Corrected, first copy | 34.632 s | 2.232 s | 32.297 s | 663 / 663 | Passed |
| Corrected, second copy | 34.747 s | 2.235 s | 32.396 s | 663 / 663 | Passed |
The working directory contains the fixture copies, per-command logs, and `summary.json`. The summary records platform/compiler information, function/context counts, preprocessing time, total analysis time, and the clean/brinfo/focxt stage times for each Bytes run.

The before/after rows perform different amounts of successful work, so they are not a speedup benchmark. Both corrected copies passed the source and artifact checks; the method-identity fixture also passed. The check is part of Linux CI alongside the existing minimal pipeline, coverage compatibility, and scheduling regressions.
Tool compilation is excluded from those analysis timings. Compare runs using the same tool build profile and target configuration; the figures are measurements for that environment, not a performance guarantee. The script applies an external 300-second limit to each command. That limit is part of the regression harness, not a PALM CLI timeout.

## Scope

The 663 functions are the functions exported by the current analyzers for this pinned crate/configuration, not a claim to cover every possible Rust function or feature combination. Recovering an already selected method from parseable source does not provide general macro expansion: a macro-generated impl or source containing unexpanded substitutions may still be unsupported. Required trait methods without bodies are context declarations rather than focal functions.
The 663 functions are those exported for this pinned crate and configuration. They do not establish coverage of every Rust function or feature combination. Recovering a selected method from parseable source does not provide general macro expansion; macro-generated impls or unexpanded substitutions may remain unsupported. Required trait methods without bodies are context declarations rather than focal functions.

Full bytes generation, repair, model quality, and coverage experiments remain the next validation stage. This batch does not change their evaluation semantics.
This check runs in the full Linux CI suite. It verifies analysis artifacts and selected context behavior, without generating tests, performing compilation repair, or measuring model quality. See [minimal pipeline validation](minimal-pipeline.md) for deterministic generation/repair/coverage checks and the [small model trial](../examples/README.md#prepare-a-small-model-trial) for scoped experiments.
Loading
Loading