Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
cff-version: 1.2.0
message: "If you use PALM in your research, please cite the ASE 2025 paper below."
title: "PALM"
type: software
authors:
- name: "PALM contributors"
repository-code: "https://github.com/cppbear/PALM"
url: "https://github.com/cppbear/PALM"
license: MIT
preferred-citation:
type: conference-paper
title: "PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage"
authors:
- family-names: Chu
given-names: Bei
- family-names: Feng
given-names: Yang
- family-names: Liu
given-names: Kui
- family-names: Shi
given-names: Hange
- family-names: Nan
given-names: Zifan
- family-names: Guo
given-names: Zhaoqiang
- family-names: Xu
given-names: Baowen
collection-title: "2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)"
year: 2025
start: 2720
end: 2732
doi: "10.1109/ASE63991.2025.00223"
40 changes: 40 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Contributing to PALM

Bug reports, documentation improvements, and focused pull requests are welcome. Contributions to PALM are made under the project's [MIT License](LICENSE). Keep existing third-party copyright and license notices when changing bundled material.

## Set up and check a change

Follow the [prerequisites](README.md#prerequisites), then work from the repository root:

```sh
cargo build --workspace --locked
cargo test --workspace --locked
```

Use the existing check relevant to your change:

| Change | Check |
| --- | --- |
| Generation, repair, result handling, or test execution | `python3 scripts/check_minimal.py` |
| Task scheduling, request budgets, or failure cleanup | `python3 scripts/check_tasks.py` |
| Coverage annotations or preprocessing | `python3 scripts/check_coverage.py` |
| Analysis, focal context, or method identity | `cargo build --workspace --release --locked` then `python3 scripts/check_bytes.py` |

The checks use temporary targets and local model fixtures. They do not need API keys or call an external model service; dependency installation may need network access. The full suite runs in CI for code changes. Documentation changes should preserve working links, command examples, and the distinction between offline validation and real-model results.

## Report a problem

Open a [GitHub issue](https://github.com/cppbear/PALM/issues) with:

- The PALM commit, operating system, `rustc --version`, and `cargo llvm-cov --version`.
- The command you ran, expected behavior, actual behavior, and relevant error output.
- A small reproducible target or function when possible, and the stage that failed: analysis, generation, repair, or coverage.
- For model-related failures, the model identifier and relevant response or request summary with private information removed. Do not include API keys or your `api.json`.

Run PALM on a working copy of a target crate: preprocessing changes its source and test layout, and generation can add a test dependency.

## Submit a pull request

Keep the change focused and explain the problem, resulting behavior, and checks you ran. Add a regression case for a reproduced behavior change when the existing checks do not cover it. Real-model calls are not required for ordinary fixes; use local fixtures or existing saved candidates where practical.

Keep credentials, target build directories, generated answers, coverage reports, and local trial logs out of commits. Preserve the pinned toolchain and dependency lockfile unless changing them is part of the task. Record the origin of imported code or material in [source integration notes](docs/source-integration.md), and retain its applicable notices.
4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,7 @@
resolver = "2"
members = ["brinfo", "focxt", "focxt/call_chain", "utgen", "build-utils"]
exclude = ["examples/bytes", "examples/minimal"]

[workspace.package]
license = "MIT"
repository = "https://github.com/cppbear/PALM"
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2025-2026 PALM contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
95 changes: 70 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,22 @@
# PALM

Source code and documents for ***PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage***.
[![Build and test](https://github.com/cppbear/PALM/actions/workflows/ci.yml/badge.svg)](https://github.com/cppbear/PALM/actions/workflows/ci.yml)

PALM combines program analysis with LLMs to generate Rust tests, repair compilation errors, and collect coverage data. The Rust workspace is located at the repository root.
PALM combines program analysis and large language models to generate Rust unit tests. It extracts path constraints and function context, generates candidates, repairs compilation errors, and reports test results and coverage with test code excluded.

This is the maintained implementation of ***PALM: Synergizing Program Analysis and LLMs to Enhance Rust Unit Test Coverage***, published at ASE 2025. The repository includes changes made after the paper's experiments.

[Paper](https://doi.org/10.1109/ASE63991.2025.00223) · [Preprint](https://arxiv.org/abs/2506.09002) · [中文技术指南](docs/palm-rust-unit-test-generation.md) · [CLI reference](utgen/README.md) · [Citation](#citation)

## Prerequisites

1. Install the pinned Rust toolchain:
Use a native Rust build environment with Git, [rustup](https://rustup.rs/), the stable Rust toolchain, and a C linker. The validation scripts also require Python 3.9 or later. The offline pipeline is checked on Linux CI and has been exercised on macOS with Apple Silicon.

1. Install the pinned analysis toolchain and components:

```sh
rustup install nightly-2025-03-19
rustup component add --toolchain nightly-2025-03-19 rust-src rustc-dev llvm-tools-preview
rustup toolchain install nightly-2025-03-19 --profile minimal \
--component rust-src --component rustc-dev --component llvm-tools-preview --component rust-analyzer
```

2. Install [cargo-llvm-cov](https://github.com/taiki-e/cargo-llvm-cov):
Expand All @@ -19,21 +25,22 @@ PALM combines program analysis with LLMs to generate Rust tests, repair compilat
cargo +stable install cargo-llvm-cov --version 0.6.16 --locked
```

3. Use the same nightly toolchain for the target crate. The analysis tools handle `lib` and `bin` targets.
4. For generation and repair, prepare runtime model configuration as described in [utgen](utgen/README.md#prerequisites). Building, installation, and ordinary tests do not require model credentials.
Use the same nightly toolchain for the target crate. The supported end-to-end workflow operates on one standalone crate at a time; analysis handles `lib` and `bin` targets. Building, installation, and the offline checks do not require model credentials.

## Project Structure
## Quick start

| Directory | Purpose |
| --- | --- |
| [brinfo](brinfo/README.md) | Extract function condition chains from HIR and MIR. |
| [focxt](focxt/README.md) | Construct code context for each focal function. |
| `focxt/call_chain/` | Extract calls and type dependencies through `cargo call-chain`. |
| [utgen](utgen/README.md) | Generate tests, check compilation, repair tests, and collect coverage. |
| `build-utils/` | Configure the compiler library search path during builds. |
| [examples](examples/README.md) | The bundled bytes target and usage instructions. |
| `docker/` | Container build and run scripts. |
| [docs](docs/README.md) | Technical guide, ASE 2025 materials, and source integration notes. |
After installing the prerequisites, run the minimal pipeline without API keys:

```sh
git clone https://github.com/cppbear/PALM.git
cd PALM
cargo build --workspace --locked
python3 scripts/check_minimal.py
```

The script uses fixed responses from a local HTTP server and real analysis, compilation, repair, and coverage tools. It works in temporary copies, prints their location, and finishes with `Minimal analysis/generation/repair/coverage checks passed.` The check verifies pipeline behavior; it does not measure model quality. See [minimal pipeline validation](docs/minimal-pipeline.md) for the checked outcomes and output files.

For model-based generation, install the tools below and follow the [two-function trial](examples/README.md#prepare-a-small-model-trial). That walkthrough uses a working copy, an explicit function list, and separate generation and repair request budgets.

## Installation

Expand All @@ -56,12 +63,6 @@ cargo install --path focxt --locked
cargo install --path utgen --locked
```

## Docker

Run `docker/docker-build` from the repository root to prepare an image with the required toolchain. Mirrors can be configured in `docker/Dockerfile` if needed.

Run `docker/docker-run` from the same directory to mount the repository at `/home/palm/palm` in the container. In the container, change to that directory and follow [Installation](#installation).

## Workflow

To measure existing tests before preprocessing, run `utgen coverage -p <original-crate-copy>`. It runs tests once, excludes test code from coverage, and writes `coverage.xml` and `coverage.json` without model configuration. Keep this baseline copy separate from the generation copy.
Expand All @@ -72,6 +73,27 @@ Then use `utgen gen` and `utgen fix` for generation and compilation repair. Pass

Start with the [minimal pipeline check](docs/minimal-pipeline.md), which uses fixed local model responses and verifies analysis, generation, repair, source restoration, and coverage. The [bytes analysis check](docs/bytes-analysis.md) validates analysis on the larger bundled target without a model service. For model-based generation and repair, follow the [small model trial](examples/README.md#prepare-a-small-model-trial) using an explicit function list and separate request limits, then inspect the request reports and test statistics. See [utgen usage](utgen/README.md) for command details and supported scope.

Generation or repair completing does not mean every candidate passes. Inspect both request reports and test statistics. Repair addresses compilation errors; runtime failures and five-second candidate timeouts remain failed test outcomes.

## Project Structure

| Directory | Purpose |
| --- | --- |
| [brinfo](brinfo/README.md) | Extract function condition chains from HIR and MIR. |
| [focxt](focxt/README.md) | Construct code context for each focal function. |
| `focxt/call_chain/` | Extract calls and type dependencies through `cargo call-chain`. |
| [utgen](utgen/README.md) | Generate tests, check compilation, repair tests, and collect coverage. |
| `build-utils/` | Configure the compiler library search path during builds. |
| [examples](examples/README.md) | The minimal fixture, bundled bytes target, and usage instructions. |
| `docker/` | Container build and run scripts. |
| [docs](docs/README.md) | Technical guide, ASE 2025 materials, and source integration notes. |

## Docker

Run `docker/docker-build` from the repository root to prepare an image with the required toolchain. Mirrors can be configured in `docker/Dockerfile` if needed.

Run `docker/docker-run` from the same directory to mount the repository at `/home/palm/palm` in the container. In the container, change to that directory and follow [Installation](#installation).

## Development checks

From the repository root, using the pinned toolchain:
Expand All @@ -83,10 +105,33 @@ cargo test --workspace --locked

Default tests use local model-response fixtures and do not contact a model service. The real-service test is opt-in; see [utgen testing](utgen/README.md#testing). Dependency downloads may still require network access. [Build validation](docs/build-validation.md) records the baseline and checks performed.

The [GitHub Actions workflow](.github/workflows/ci.yml) runs on pull requests and pushes to main, canceling superseded runs for the same PR or branch. Changes limited to READMEs, Markdown guides, and documentation images/PDFs receive a lightweight patch whitespace check. Code changes and manual workflow runs execute the full offline suite on Linux.
The [GitHub Actions workflow](.github/workflows/ci.yml) runs on pull requests and pushes to main, canceling superseded runs for the same PR or branch. Changes limited to READMEs and the selected documentation paths under `docs/` receive a lightweight patch whitespace check. Other changes and manual workflow runs execute the full offline suite on Linux.

## Contributing

Bug reports and focused pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, relevant checks, and the information to include in a report.

## Documentation

- [Technical guide in Chinese](docs/palm-rust-unit-test-generation.md): architecture, data flow, and usage.
- [ASE 2025 materials](docs/ase2025/README.md): the original poster and presentation.
- [Source integration notes](docs/source-integration.md): source repositories, revisions, and branch decisions.

## Citation

If you use PALM in your research, please cite the ASE 2025 paper. [CITATION.cff](CITATION.cff) provides the machine-readable citation.

```bibtex
@inproceedings{Chu2025PALM,
author = {Bei Chu and Yang Feng and Kui Liu and Hange Shi and Zifan Nan and Zhaoqiang Guo and Baowen Xu},
title = {{PALM}: Synergizing Program Analysis and {LLMs} to Enhance {Rust} Unit Test Coverage},
booktitle = {2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
year = {2025},
pages = {2720--2732},
doi = {10.1109/ASE63991.2025.00223}
}
```

## License

PALM is licensed under the [MIT License](LICENSE). The bundled [bytes example](examples/bytes/LICENSE) retains its original MIT license and copyright notice. See [source integration notes](docs/source-integration.md#licensing-and-attribution) for the origins and attribution of included code and materials.
2 changes: 2 additions & 0 deletions brinfo/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
name = "brinfo"
version = "0.1.0"
edition = "2024"
license.workspace = true
repository.workspace = true

[package.metadata.rust-analyzer]
rustc_private = true
Expand Down
2 changes: 2 additions & 0 deletions build-utils/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@
name = "build-utils"
version = "0.1.0"
edition = "2024"
license.workspace = true
repository.workspace = true
5 changes: 4 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@ PALM's implementation, technical guide, and conference materials are available i

| Resource | Contents |
| --- | --- |
| [Project README](../README.md) | Prerequisites, installation, and workspace layout. |
| [Project README](../README.md) | Quick start, installation, workflow, and workspace layout. |
| [Contribution guide](../CONTRIBUTING.md) | Local checks, issue reports, and pull requests. |
| [Citation](../CITATION.cff) | Preferred citation for the ASE 2025 paper. |
| [License](../LICENSE) | MIT license for PALM. |
| [Technical guide in Chinese](palm-rust-unit-test-generation.md) | Condition chains, context construction, generation, repair, and usage. |
| [Minimal pipeline](minimal-pipeline.md) | Deterministic program-analysis, generation, repair, and coverage validation. |
| [bytes analysis](bytes-analysis.md) | Release-tool analysis baseline, method identity checks, and timings. |
Expand Down
6 changes: 6 additions & 0 deletions docs/source-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ This repository brings together PALM's implementation, the technical guide devel

The source repositories have separate Git histories. Selected files are imported with their origins recorded here; this integration does not merge those histories or remove the source repositories.

## Licensing and Attribution

PALM's source code, scripts, and project documentation use the root [MIT License](../LICENSE), with collective attribution to PALM contributors. The five tool crates declare the same license through workspace metadata. The selected source revisions above did not contain a license for the PALM implementation; this repository now explicitly adopts MIT rather than inheriting the example crate's license.

The bundled `examples/bytes` crate retains its [original MIT license and copyright notice](../examples/bytes/LICENSE). The technical guide and images originate from `rust-utgen`; the conference PDFs originate from `ase2025` and are preserved as described in [ASE 2025 materials](ase2025/README.md). Existing third-party notices and rights in included material remain applicable; the root license does not replace them.

## Layout and Compatibility

The Rust workspace, installation script, and Docker scripts now reside at the repository root. The two original README files are consolidated, and the root `.gitignore` retains all original rules, including `api.json`.
Expand Down
2 changes: 2 additions & 0 deletions focxt/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
name = "focxt"
version = "0.1.0"
edition = "2024"
license.workspace = true
repository.workspace = true

# [workspace]
# members = ["call_chain"]
Expand Down
2 changes: 2 additions & 0 deletions focxt/call_chain/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
name = "call_chain"
version = "0.1.0"
edition = "2024"
license.workspace = true
repository.workspace = true

[package.metadata.rust-analyzer]
rustc_private = true
Expand Down
2 changes: 2 additions & 0 deletions utgen/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
name = "utgen"
version = "0.1.0"
edition = "2024"
license.workspace = true
repository.workspace = true

[dependencies]
async-openai = "0.26"
Expand Down
Loading