From 6897ec0d6ed0f6a9dd4bf52b5598a88f922a7778 Mon Sep 17 00:00:00 2001 From: cppbear Date: Thu, 8 Oct 2026 18:19:57 +0800 Subject: [PATCH] docs: make scoped model runs the example workflow --- README.md | 2 +- docs/palm-rust-unit-test-generation.md | 2 +- examples/README.md | 31 +++++++++++++++++--------- 3 files changed, 23 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 4284900e1e..302f45c50d 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ On a fresh working copy of a standalone crate, run `utgen pre-process -p [--work-dir ...] [--tasks ] 任务异常会在等待所有任务结束后报告，并跳过后续覆盖率统计。修复成功时只删除本次创建的备份；任务失败时恢复源码并保留这些备份，已有备份不会被覆盖。候选本身编译不通过仍按测试结果处理。模型请求现在统一执行：`--request-timeout` 默认每次尝试 180 秒，包含响应体读取；网络错误、429（余额不足除外）和 5xx 最多尝试 3 次，其他 HTTP 错误或空回答直接报错。格式纠正和编译修复轮次单独计算，不再叠加网络重试。缺少 `usage` 的有效回答可以继续，但请求统计标记为不完整；生成和修复分别写入 `utgen/generation/gen-requests.json`、`fix-requests.json`。该期限不涵盖 Cargo 子进程，也不提供跨进程锁或强制终止恢复。 -小规模试跑可在 `gen`、`fix` 中分别传入同一份 `--functions-file`，不会自动沿用上次清单。空清单和未知名称在修改目标前报错；修复时缺少所选函数的候选结果也会报错。`--max-requests` 包含所有并发任务、生成阶段、格式纠正、修复轮次及网络重试的请求尝试，计数与额度检查使用同一个共享计数器。额度用完且仍需请求时，命令失败，等待已有请求收尾并恢复源码；恰好用完额度且工作全部完成则成功。请求报告记录模型、选项、去重后的函数清单、候选处理状态和用量；候选状态不等于测试通过或覆盖率统计完成。未选函数的已有文件保留，跨实验应使用新副本。准备好的 [2 函数和 8 函数清单及操作说明](../examples/README.md#prepare-a-small-model-trial)用于后续真实模型验收，目前不代表已取得真实模型结果。 +小规模试跑可在 `gen`、`fix` 中分别传入同一份 `--functions-file`，不会自动沿用上次清单。空清单和未知名称在修改目标前报错；修复时缺少所选函数的候选结果也会报错。`--max-requests` 包含所有并发任务、生成阶段、格式纠正、修复轮次及网络重试的请求尝试，计数与额度检查使用同一个共享计数器。额度用完且仍需请求时，命令失败，等待已有请求收尾并恢复源码；恰好用完额度且工作全部完成则成功。请求报告记录模型、选项、去重后的函数清单、候选处理状态和用量；候选状态不等于测试通过或覆盖率统计完成。未选函数的已有文件保留，跨实验应使用新副本。[2 函数和 8 函数清单及操作说明](../examples/README.md#prepare-a-small-model-trial)提供限定函数范围和请求额度的用法；先检查请求报告，再查看对应函数的编译、执行和覆盖率统计。 ### 8.2 LLM 配置 diff --git a/examples/README.md b/examples/README.md index 96059a13e5..c34bb7f45a 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,8 +1,8 @@ # Examples -Start with the [minimal pipeline](../docs/minimal-pipeline.md) for an automated, no-credentials validation. The commands below describe the larger bytes target; the complete pipeline is currently regression-tested on the minimal crate. +Start with the [minimal pipeline](../docs/minimal-pipeline.md) for an automated, no-credentials validation. The commands below use a working copy of the larger bytes target and limit model generation and repair to selected functions. -The [bytes analysis check](../docs/bytes-analysis.md) now validates all 663 exported functions and their context artifacts on two fresh copies. Run `cargo build --workspace --release --locked` and `python3 scripts/check_bytes.py` from the repository root. Model-based generation and repair on bytes remain a separate validation stage. +The [bytes analysis check](../docs/bytes-analysis.md) validates all 663 exported functions and their context artifacts on two fresh copies. Run `cargo build --workspace --release --locked` and `python3 scripts/check_bytes.py` from the repository root. This check does not call a model service or measure generated-test quality. ## bytes @@ -37,33 +37,38 @@ Analysis runs brinfo and focxt (including call-chain) on the prepared source, re ### Generate and Repair Unit Tests -```sh -utgen gen -p "$palm_example_dir" --requirement --context -utgen fix -p "$palm_example_dir" -``` - `--requirement` enables generation for representative condition chains; `--context` includes focal context in the prompt. Both flags are off by default. In unit mode, candidates are stored in JSON and temporarily inserted into the target source for checks and execution. +Use the scoped example below to start. The [command reference](../utgen/README.md#limited-function-runs) describes other selections and all-function behavior. + ### Prepare a Small Model Trial The checked-in [two-function list](bytes-smoke-2.txt) selects `Bytes::len` and `BytesMut::len`. The [eight-function list](bytes-smoke-8.txt) adds `is_empty`, `split_off`, and `truncate` for both types. They cover 2 and 16 representative condition chains respectively. The repeated namespace segments are exact compiler-index names. The analysis regression checks both lists without calling a model. -For a future trial, agree on the model configuration and separate generation/repair attempt limits first. Set `palm_gen_limit` and `palm_fix_limit` to those positive integers, then use the prepared working copy: +Choose the model in your runtime configuration and set separate generation and repair attempt limits. The values below are example budgets; adjust them as needed. Omitting `--max-requests` leaves the command without an overall request limit. ```sh +palm_gen_limit=8 +palm_fix_limit=8 + utgen gen -p "$palm_example_dir" --requirement --context \ --functions-file "$palm_repo/examples/bytes-smoke-2.txt" \ --max-requests "$palm_gen_limit" +``` + +After generation completes, run compilation repair with the same function list. Repair handles compilation errors; runtime test failures remain in the statistics. + +```sh utgen fix -p "$palm_example_dir" \ --functions-file "$palm_repo/examples/bytes-smoke-2.txt" \ --max-requests "$palm_fix_limit" ``` -Both commands default to four active function tasks and a 180-second deadline per request attempt. Every retry consumes the command's attempt limit. Review the request reports, candidate compilation/pass counts, coverage, and restored sources before expanding to eight functions on another fresh copy. Keep configuration and generation mode fixed when comparing these runs. No real-model result is claimed by the deterministic checks; the lists and limits prepare that later validation. +Both commands default to four active function tasks and a 180-second deadline per request attempt. Every retry consumes the command's attempt limit. Review the [results](#results) and restored sources before expanding to eight functions on another fresh copy. Replace the function-list path in both commands for that selection. Keep configuration and generation mode fixed when comparing runs; the deterministic checks do not measure model quality. ### Integration Test Mode -Use a fresh working copy and repeat extraction and preprocessing before choosing this mode, since existing generation results are reused independently of these flags: +Use a fresh working copy and repeat preprocessing and analysis before choosing this mode, since existing generation results are reused independently of these flags: ```sh utgen gen -p --integration --requirement --context @@ -73,6 +78,12 @@ Integration mode generates files under the target's `tests/` directory after fil ### Results +Read the outputs in this order: + +1. Check the request reports under `utgen/generation/` for candidate-stage completion, request counts, and budget exhaustion. A completed candidate stage does not mean every test passed or coverage statistics finished. +2. Inspect `utgen/result/` or `utgen/fixed_result/` for the selected functions' compilation, execution, and passing-test counts. +3. Read coverage from those per-function statistics alongside the test outcomes. + Paths below are relative to the working copy: | Path | Contents |