Complete bytes analysis with compiler-based function identities

.github/workflows/ci.yml

@@ -44,3 +44,7 @@ jobs:
         run: python3 scripts/check_minimal.py
       - name: Check generation and repair task limits
         run: python3 scripts/check_tasks.py
+      - name: Build release analysis tools
+        run: cargo build --workspace --release --locked
+      - name: Validate bytes analysis and method identities
+        run: python3 scripts/check_bytes.py

README.md

@@ -70,7 +70,7 @@ On a fresh working copy of a standalone crate, run `utgen pre-process -p <target
 
 Then use `utgen gen` and `utgen fix` for generation and compilation repair. Pass `--requirement --context` to `utgen gen` to include path constraints and focal context. Preprocessing and test checks modify the target tree; use a working copy of the target project.
 
-Start with the [minimal pipeline check](docs/minimal-pipeline.md), which uses fixed local model responses and verifies analysis, generation, repair, source restoration, and coverage. The [bytes example](examples/README.md) remains a larger follow-up target. See [utgen usage](utgen/README.md) for command details.
+Start with the [minimal pipeline check](docs/minimal-pipeline.md), which uses fixed local model responses and verifies analysis, generation, repair, source restoration, and coverage. The [bytes analysis check](docs/bytes-analysis.md) validates the larger bundled target; model-based generation and repair on bytes remain the next stage. See [utgen usage](utgen/README.md) for command details.
 
 ## Development checks
 

docs/README.md

@@ -7,6 +7,7 @@ PALM's implementation, technical guide, and conference materials are available i
 | [Project README](../README.md) | Prerequisites, installation, and workspace layout. |
 | [Technical guide in Chinese](palm-rust-unit-test-generation.md) | Condition chains, context construction, generation, repair, and usage. |
 | [Minimal pipeline](minimal-pipeline.md) | Deterministic program-analysis, generation, repair, and coverage validation. |
+| [bytes analysis](bytes-analysis.md) | Release-tool analysis baseline, method identity checks, and timings. |
 | [bytes example](../examples/README.md) | Commands for analyzing a working copy of the bundled target. |
 | [utgen reference](../utgen/README.md) | Configuration, CLI behavior, and output files. |
 | [ASE 2025 materials](ase2025/README.md) | Original poster and presentation PDFs. |

docs/bytes-analysis.md

@@ -0,0 +1,52 @@
+# bytes Analysis Validation
+
+This check covers the complete analysis command on the bundled bytes 1.10.0 crate with its default features. PALM tools are built in release mode with the pinned `nightly-2025-03-19`; Cargo's target-analysis profile remains unchanged. No model service or coverage tool is required.
+
+## Plan and Acceptance
+
+The plan is to measure the existing release build, correct observed analysis blockers, and validate two fresh working copies before expanding to model experiments. Acceptance requires:
+
+- Preprocessing and `utgen analyze` return success on both copies.
+- All 663 exported branch functions have corresponding compiler identities, raw/normalized dependency files, and nonempty context files. No function is dropped to satisfy this count.
+- Branch code and compiler-exported method code match their source locations; each context contains its focal function's declaration.
+- A small separate crate distinguishes generic trait implementations, reference receivers, a trait default method, and a macro-generated forwarding method. Two impls share one source line, and an inactive `cfg` variant must not replace the active method.
+- Analysis leaves prepared sources and the manifest unchanged. The bundled example is also unchanged.
+
+The review identified two traps to avoid: debug-tool timings do not establish release performance, and nonempty context files alone do not establish correct method identity. The check therefore uses release tools and also verifies selected method bodies and their impl headers. Its 300-second per-command limit is an external regression-test deadline, not a new PALM timeout or a performance guarantee.
+
+## Changes
+
+The baseline emitted 663 function identities but only 226 contexts. Trait default contexts were constructed without being written. Matching methods by short module/type/trait names also conflated generic trait instantiations and reference receivers. Source parsing alone missed methods supplied by forwarding macros.
+
+Call-chain now includes each function's source span, enclosing impl header span, and already extracted source code in its index. The source parser binds functions to these compiler identities before building the dependency lookup. Compiler-selected methods with parseable source are added to the matching impl when absent from the source AST. Inactive method bodies are excluded, while required trait signatures are retained. Context files use the compiler's existing encoded names, and trait default contexts are written normally.
+
+Recursive dependency JSON reads use `BufReader`. No persistent cache or new cache invalidation scheme is introduced. `utgen analyze` reports the duration of clean, brinfo, and focxt; a missing context error identifies the function and path.
+
+## Reproduce
+
+From the repository root, with the pinned compiler and components installed:
+
+```sh
+cargo build --workspace --release --locked
+python3 scripts/check_bytes.py
+```
+
+Use `--bin-dir <directory>` for a custom tool build directory and `--runs 1` for one bytes copy. Rebuild the tools together: focxt consumes the source metadata emitted by the matching call-chain binary. The script prints a temporary directory containing copies, per-command logs, stage timings, and `summary.json`. Its default is two copies; dependency downloads can require network access.
+
+## Local Results
+
+Measured on macOS/Apple Silicon with the pinned compiler. Tool compilation is excluded from the analysis times below.
+
+| Run | Analyze | brinfo | focxt | Branch functions / contexts | Outcome |
+| --- | ---: | ---: | ---: | ---: | --- |
+| Previous main, release | 43.714 s | — | — | 663 / 226 | Missing context, exit 1 |
+| Corrected, first copy | 34.632 s | 2.232 s | 32.297 s | 663 / 663 | Passed |
+| Corrected, second copy | 34.747 s | 2.235 s | 32.396 s | 663 / 663 | Passed |
+
+The before/after rows perform different amounts of successful work, so they are not a speedup benchmark. Both corrected copies passed the source and artifact checks; the method-identity fixture also passed. The check is part of Linux CI alongside the existing minimal pipeline, coverage compatibility, and scheduling regressions.
+
+## Scope
+
+The 663 functions are the functions exported by the current analyzers for this pinned crate/configuration, not a claim to cover every possible Rust function or feature combination. Recovering an already selected method from parseable source does not provide general macro expansion: a macro-generated impl or source containing unexpanded substitutions may still be unsupported. Required trait methods without bodies are context declarations rather than focal functions.
+
+Full bytes generation, repair, model quality, and coverage experiments remain the next validation stage. This batch does not change their evaluation semantics.

docs/minimal-pipeline.md

@@ -48,7 +48,7 @@ The preprocessor parses `cfg` predicates conservatively, preserves `cfg(not(test
 
 Recursive context lookup now resolves full function names through the compiler's encoded-name index and consumes the accumulated dependencies. External functions without local artifacts remain outside this source context.
 
-The local macOS/Apple Silicon run passed using six generation responses and one repair response. Real-model behavior, bytes-scale execution, general multi-crate support, hard process timeouts, recovery after forced termination, and explicit run/resume behavior remain later work. Git history is retained.
+The local macOS/Apple Silicon run passed using six generation responses and one repair response. The [bytes analysis check](bytes-analysis.md) separately validates the larger target without a model. Real-model behavior, full bytes generation/repair, general multi-crate support, hard process timeouts, recovery after forced termination, and explicit run/resume behavior remain later work. Git history is retained.
 
 The coverage entry temporarily annotates sources with `#[coverage(off)]` and enables the nightly feature at crate roots without adding newlines. It preserves compiler flags, restores original bytes on returned errors, and performs no hashing or source fingerprint checks. `cargo llvm-cov report --json` reuses the first run's data; the first XML-producing command retains the tool's normal profile cleanup between candidates. This does not provide crash recovery or general macro expansion. See the [coverage command reference](../utgen/README.md#coverage-of-existing-tests) for scope.
 

docs/palm-rust-unit-test-generation.md

@@ -635,7 +635,7 @@ utgen gen -p "$palm_example_dir" --requirement --context
 utgen fix -p "$palm_example_dir"
 ```
 
-切换到 integration、其他提示词或其他模型时，使用新的目标副本并重新分析，避免复用旧的生成缓存。更详细的示例见 [examples/README.md](../examples/README.md)。当前完整回归使用 [minimal 示例](minimal-pipeline.md)；bytes 的完整运行仍是后续验证项。
+切换到 integration、其他提示词或其他模型时，使用新的目标副本并重新分析，避免复用旧的生成缓存。更详细的示例见 [examples/README.md](../examples/README.md)。当前完整生成与修复回归使用 [minimal 示例](minimal-pipeline.md)。[bytes 分析验收](bytes-analysis.md) 已覆盖默认配置下导出的 663 个函数及其上下文；bytes 的真实模型生成、修复和覆盖率实验仍是后续验证项。
 
 完成后重点查看临时目标目录中的以下路径：
 

docs/source-integration.md

@@ -31,4 +31,4 @@ The `comment_out_test_module` branch only handles test modules. PALM already han
 
 The `focxt_rustc_api` branch changes the entry point to `cargo focxt` and produces `focxt/name_map.json`, while its generation code still requires `focxt/impl_informations.json`. It also uses the older compiler API. Integration of that extractor requires a separate interface migration and validation.
 
-The integration commits preserved the existing generation and repair behavior. Subsequent maintenance adds runtime API configuration, isolates real-model tests, and validates installation and basic CI; see [build validation](build-validation.md). Request concurrency controls and the unfinished batch test injection path remain follow-up work.
+The integration commits preserved the existing generation and repair behavior. Subsequent maintenance adds runtime API configuration, isolates real-model tests, and validates installation and basic CI; see [build validation](build-validation.md). Subsequent checks cover [function-level task limits](minimal-pipeline.md#task-scheduling-regression) and [bytes analysis](bytes-analysis.md). The unfinished batch test injection path remains outside the active CLI.

examples/README.md

@@ -2,6 +2,8 @@
 
 Start with the [minimal pipeline](../docs/minimal-pipeline.md) for an automated, no-credentials validation. The commands below describe the larger bytes target; the complete pipeline is currently regression-tested on the minimal crate.
 
+The [bytes analysis check](../docs/bytes-analysis.md) now validates all 663 exported functions and their context artifacts on two fresh copies. Run `cargo build --workspace --release --locked` and `python3 scripts/check_bytes.py` from the repository root. Model-based generation and repair on bytes remain a separate validation stage.
+
 ## bytes
 
 The bundled target is [tokio-rs/bytes](https://github.com/tokio-rs/bytes), version 1.10.0. It is excluded from PALM's tool workspace and retains its own source files, tests, and license.
@@ -31,7 +33,7 @@ utgen pre-process -p "$palm_example_dir"
 utgen analyze -p "$palm_example_dir"
 ```
 
-Analysis runs brinfo and focxt (including call-chain) on the prepared source. If a function lacks corresponding context, generation reports the missing artifact; support for larger targets is still being validated.
+Analysis runs brinfo and focxt (including call-chain) on the prepared source, reports stage timings, and validates the resulting artifacts. If a function lacks corresponding context, the command reports the function and missing path.
 
 ### Generate and Repair Unit Tests
 

focxt/README.md

@@ -22,3 +22,5 @@ Options:
 ```
 
 `focxt -c crate_path` or `focxt --crate crate_path`
+
+Function contexts use call-chain's compiler identities and source positions, including distinct generic trait impls and reference receivers. Recoverable macro-generated method bodies are associated with their enclosing source impl; this is not general macro expansion. Rebuild focxt and call-chain together. See the [bytes analysis validation](../docs/bytes-analysis.md) for release builds, reproducible checks, and current limits.

focxt/call_chain/src/analysis/hirvisitor.rs

@@ -32,6 +32,9 @@ pub struct ImplInformation {
     pub trait_name: String,
     pub full_name: String,
     pub encoded_name: String,
+    pub loc: SourceInfo,
+    pub impl_loc: Option<SourceInfo>,
+    pub code: String,
 }
 
 pub struct VisitorData<'tcx> {
@@ -243,6 +246,11 @@ impl<'tcx> Visitor<'tcx> for HirVisitor<'tcx> {
             trait_name,
             full_name: full_name,
             encoded_name: encoded_name,
+            loc: fn_source.clone(),
+            impl_loc: matches!(parent_kind, DefKind::Impl { .. }).then(|| {
+                SourceInfo::from_span(self.tcx.def_span(parent_def_id), self.tcx.sess.source_map())
+            }),
+            code,
         };
 
         let data = VisitorData {

focxt/src/collect_context/crate_context.rs

@@ -118,6 +118,12 @@ impl CrateContext {
         }
     }
 
+    pub fn bind_compiler_functions(&self, infos: &[ImplInformation]) {
+        for context in &self.main_mod_contexts {
+            context.borrow_mut().bind_compiler_functions(infos);
+        }
+    }
+
     pub fn get_all_new_calls_and_types(
         &self,
         impl_informations: &Vec<ImplInformation>,

focxt/src/collect_context/items_context.rs

@@ -1,5 +1,6 @@
 use std::{cell::RefCell, rc::Rc};
 
+use call_chain::analysis::hirvisitor::ImplInformation;
 use quote::ToTokens;
 use syn::{
     ImplItemConst, ImplItemFn, ImplItemType, Item, ItemConst, ItemEnum, ItemFn, ItemImpl,
@@ -751,6 +752,7 @@ impl TypeItem {
 #[derive(Debug, Clone)]
 pub struct FnItem {
     fn_name: Name,
+    encoded_name: Option<String>,
     complete_name_in_file: String,
     item: Option<ItemFn>,
     // has_items: bool,
@@ -763,6 +765,7 @@ impl FnItem {
     pub fn new() -> Self {
         FnItem {
             fn_name: Name::none(),
+            encoded_name: None,
             complete_name_in_file: String::new(),
             // complete_function_name_in_file: String::new(),
             item: None,
@@ -821,6 +824,16 @@ impl FnItem {
         self.fn_name.get_name()
     }
 
+    pub fn bind_compiler_info(&mut self, info: &ImplInformation) {
+        self.fn_name.insert_complete_name(&info.full_name);
+        self.fn_name.insert_import_name(&info.full_name);
+        self.encoded_name = Some(info.encoded_name.clone());
+    }
+
+    pub fn get_encoded_name(&self) -> Option<String> {
+        self.encoded_name.clone()
+    }
+
     pub fn get_complete_name(&self) -> String {
         self.fn_name.get_import_name().to_string()
     }
@@ -854,6 +867,10 @@ impl FnItem {
         Item::Fn(self.item.clone().unwrap())
     }
 
+    pub fn get_item(&self) -> &ItemFn {
+        self.item.as_ref().unwrap()
+    }
+
     pub fn insert_visibility(&mut self, visibility: MyVisibility) {
         self.visibility = visibility;
     }
@@ -922,6 +939,7 @@ impl ImplConstItem {
 #[derive(Debug, Clone)]
 pub struct ImplFnItem {
     fn_name: Name,
+    encoded_name: Option<String>,
     complete_name_in_file: String,
     item: Option<ImplItemFn>,
     // has_items: bool,
@@ -934,6 +952,7 @@ impl ImplFnItem {
     pub fn new() -> Self {
         ImplFnItem {
             fn_name: Name::none(),
+            encoded_name: None,
             complete_name_in_file: String::new(),
             item: None,
             inside_items: Vec::new(),
@@ -1042,6 +1061,16 @@ impl ImplFnItem {
         self.fn_name.get_name()
     }
 
+    pub fn bind_compiler_info(&mut self, info: &ImplInformation) {
+        self.fn_name.insert_complete_name(&info.full_name);
+        self.fn_name.insert_import_name(&info.full_name);
+        self.encoded_name = Some(info.encoded_name.clone());
+    }
+
+    pub fn get_encoded_name(&self) -> Option<String> {
+        self.encoded_name.clone()
+    }
+
     pub fn get_complete_name(&self) -> String {
         self.fn_name.get_import_name().to_string()
     }
@@ -1172,6 +1201,10 @@ impl ImplItem {
         Item::Impl(item_impl)
     }
 
+    pub fn get_fns_mut(&mut self) -> &mut Vec<ImplFnItem> {
+        &mut self.functions
+    }
+
     pub fn get_fns(&self) -> &Vec<ImplFnItem> {
         &self.functions
     }
@@ -1510,6 +1543,7 @@ impl TraitConstItem {
 #[derive(Debug, Clone)]
 pub struct TraitFnItem {
     fn_name: Name,
+    encoded_name: Option<String>,
     complete_name_in_file: String,
     item: Option<TraitItemFn>,
     // has_items: bool,
@@ -1521,6 +1555,7 @@ impl TraitFnItem {
     pub fn new() -> Self {
         TraitFnItem {
             fn_name: Name::none(),
+            encoded_name: None,
             complete_name_in_file: String::new(),
             item: None,
             inside_items: Vec::new(),
@@ -1584,6 +1619,16 @@ impl TraitFnItem {
         self.fn_name.get_name()
     }
 
+    pub fn bind_compiler_info(&mut self, info: &ImplInformation) {
+        self.fn_name.insert_complete_name(&info.full_name);
+        self.fn_name.insert_import_name(&info.full_name);
+        self.encoded_name = Some(info.encoded_name.clone());
+    }
+
+    pub fn get_encoded_name(&self) -> Option<String> {
+        self.encoded_name.clone()
+    }
+
     pub fn get_complete_name(&self) -> String {
         self.fn_name.get_import_name().to_string()
     }
@@ -1697,6 +1742,10 @@ impl TraitItem {
         Item::Trait(item_trait)
     }
 
+    pub fn get_fns_mut(&mut self) -> &mut Vec<TraitFnItem> {
+        &mut self.functions
+    }
+
     pub fn get_fns(&self) -> &Vec<TraitFnItem> {
         &self.functions
     }

focxt/src/collect_context/mod_context.rs

@@ -209,6 +209,14 @@ impl ModContext {
         self.lib_mod = Some(Rc::clone(lib_mod));
     }
 
+    pub fn bind_compiler_functions(&mut self, infos: &[ImplInformation]) {
+        self.syntax_context
+            .bind_compiler_functions(&self.mod_info.get_mod_tree().to_string(), infos);
+        for context in &self.sub_mods {
+            context.borrow_mut().bind_compiler_functions(infos);
+        }
+    }
+
     pub fn parse_from_items(
         parent: &Rc<RefCell<ModContext>>,
         items: &Vec<Item>,