fix: bound model requests and propagate generation and repair failures

docs/minimal-pipeline.md

@@ -62,4 +62,9 @@ The script runs in Linux CI. Coverage and preprocessing share only source parsin
 
 `scripts/check_tasks.py` uses nine focal functions sharing one source file, a loopback model, and real Cargo commands. It measures generation and repair request peaks for N=1, N=2, and the default N=4, requires overlap for N>1, and checks that compiler invocations never overlap or observe changing source bytes. A paused compiler lets the N=1 generation queue fill: one candidate is being validated, one result is buffered, and one active job waits to send. Releasing compilation must drain every result.
 
-The same check covers direct and input/prefix/oracle integration generation, removal of invalid imports from saved candidates, and preservation/removal of temporary compiler inputs. Integration coverage reports remain available until every function has been read, and all nine functions must have passing tests and nonzero covered lines. Missing usage in one fixture response exercises worker-panic propagation and slot release; a stale generation backup exercises queue draining after consumer failure, and blocking the diagnostic output file exercises failure while the repair lock is held. Sources must be restored, failed repair backups retained, unrelated backups untouched, existing recovery material rejected, and post-failure statistics skipped. These checks use byte comparisons without hashing and run on macOS/Linux. CLI tests separately verify the default and rejection of zero before configuration loading.
+The same check covers direct and input/prefix/oracle integration generation, removal of invalid imports from saved candidates, and preservation/removal of temporary compiler inputs. Integration coverage reports remain available until every function has been read, and all nine functions must have passing tests and nonzero covered lines. Valid answers without usage must complete successfully and mark request statistics incomplete. HTTP 401 during generation, oracle generation, and repair must fail without outer-layer retries. A blocked prompt directory exercises worker-panic propagation and slot release; a stale generation backup exercises queue draining after consumer failure, and blocking the diagnostic output file exercises failure while the repair lock is held. Sources must be restored, failed repair backups retained, unrelated backups untouched, existing recovery material rejected, and post-failure statistics skipped. These checks use byte comparisons without hashing and run on macOS/Linux. CLI tests separately verify the default and rejection of zero before configuration loading.
+
+
+## Model Request Regression
+
+Local HTTP tests exercise the shared generation/repair request path, including response usage omissions, empty choices or content, invalid JSON, permanent 4xx errors, transient 429/5xx errors, dropped connections, and delayed response bodies. They verify actual attempt counts and bounded retry exhaustion without calling a real service. Existing task checks also verify request summaries and source/backup handling after a final request failure. `--request-timeout` defaults to 180 seconds per attempt and rejects zero before configuration loading. Request formatting and compilation repair retries remain business-level operations, not additional transport retry layers.

docs/palm-rust-unit-test-generation.md

@@ -283,7 +283,7 @@ utgen fix --project-dir <PROJECT_DIR> [--work-dir <WORK_DIR>...] [--tasks <N>]
 
 生成任务取得名额后才开始工作，将结果送入容量为 N 的队列后释放名额；串行消费者负责导入检查和候选编译。队列中的结果及正在验证的候选不占生成名额。修复任务则在该函数的修复结果保存后释放名额。源码插入、target 清理、编译或测试、诊断读取及源码恢复串行执行，模型请求可以重叠；该参数不改变 Cargo 内部的依赖编译并行度。同一工作副本不能供多个 PALM 进程同时使用，不同实验应使用独立副本和 target 目录。
 
-任务异常会在等待所有任务结束后报告，并跳过后续覆盖率统计。修复成功时只删除本次创建的备份；任务失败时恢复源码并保留这些备份，已有备份不会被覆盖。候选本身编译不通过仍按测试结果处理。当前没有增加请求超时、重试策略、跨进程锁或强制终止后的恢复机制。
+任务异常会在等待所有任务结束后报告，并跳过后续覆盖率统计。修复成功时只删除本次创建的备份；任务失败时恢复源码并保留这些备份，已有备份不会被覆盖。候选本身编译不通过仍按测试结果处理。模型请求现在统一执行：`--request-timeout` 默认每次尝试 180 秒，包含响应体读取；网络错误、429（余额不足除外）和 5xx 最多尝试 3 次，其他 HTTP 错误或空回答直接报错。格式纠正和编译修复轮次单独计算，不再叠加网络重试。缺少 `usage` 的有效回答可以继续，但请求统计标记为不完整；生成和修复分别写入 `utgen/generation/gen-requests.json`、`fix-requests.json`。该期限不涵盖 Cargo 子进程，也不提供跨进程锁或强制终止恢复。
 
 ### 8.2 LLM 配置
 

scripts/check_tasks.py

@@ -136,14 +136,21 @@ def do_POST(self):
                     answer = f'ChangeLog:1@{file}\nFixDescription: Use an integer.\nOriginalCode@{number}-{number}:\n[{number}]{old}\nFixedCode@{number}-{number}:\n[{number}]{new}\n'
                 response = {'id': 'fixture', 'object': 'chat.completion', 'created': 0, 'model': 'fixture',
                             'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': answer}, 'finish_reason': 'stop'}]}
-                # Exercise a real worker panic in the existing response parser.
-                if not (state['panic'] and serial == 1):
+                # Valid answers must remain usable when a service omits usage.
+                if not (state['missing_usage'] and serial == 1):
                     response['usage'] = {'prompt_tokens': 1, 'completion_tokens': 1, 'total_tokens': 2}
+                status = 200
+                if ((state['fault'] == 'request' and serial == 1)
+                        or (state['fault'] == 'oracle' and 'generate accurate test oracles' in system
+                            and not state['fault_sent'])):
+                    response = {'error': {'message': 'fixture unauthorized'}}
+                    status = 401
+                    state['fault_sent'] = True
                 body = json.dumps(response).encode()
                 # Count requests awaiting a response, excluding HTTP-handler teardown.
                 with lock:
                     state['active'] -= 1
-                self.send_response(200)
+                self.send_response(status)
                 self.send_header('Content-Type', 'application/json')
                 self.send_header('Content-Length', str(len(body)))
                 self.end_headers()
@@ -157,15 +164,15 @@ def do_POST(self):
     threading.Thread(target=server.serve_forever, daemon=True).start()
     env.update(PALM_API_BASE=f'http://127.0.0.1:{server.server_port}/v1', PALM_API_KEY='fixture', PALM_MODEL='fixture')
 
-    def run(label, command, limit=1, integration=False, panic=False, fail=False, backpressure=False):
+    def run(label, command, limit=1, integration=False, missing_usage=False, fault=None, fail=False, backpressure=False):
         monitor = work / label
         monitor.mkdir()
         (monitor / 'compiler.json').write_text(json.dumps(dict(active=0, peak=0, total=0, source_changes=0)))
         env['PALM_TASKS_MONITOR'] = str(monitor)
         env.pop('PALM_HOLD_FIRST_BUILD', None)
         with lock:
             state.clear()
-            state.update(active=0, peak=0, total=0, errors=[], limit=limit, integration=integration, panic=panic)
+            state.update(active=0, peak=0, total=0, errors=[], limit=limit, integration=integration, missing_usage=missing_usage, fault=fault, fault_sent=False)
         ready.clear()
         watcher = None
         if backpressure:
@@ -209,6 +216,11 @@ def check_queue():
         assert (root / 'src/lib.rs').read_text() == original, label
         assert (root / 'src/unrelated.rs').read_text() == '// unrelated source file\n', label
         assert unrelated.read_bytes() == b'previous recovery material\n', label
+        if command[1] in ['gen', 'fix'] and state['total']:
+            report = json.loads((root / f'utgen/generation/{command[1]}-requests.json').read_text())
+            assert report['attempts'] == state['total'], report
+            assert report['responses_without_usage'] == int(missing_usage), report
+            assert report['usage_complete'] == (not missing_usage and not state['fault_sent']), report
         return state.copy()
 
     def reset(seed=None):
@@ -233,15 +245,19 @@ def results(directory, compiled):
         seed = work / 'broken-candidates'
         for n in [1, 2, 4]:
             reset()
-            measured = run(f'gen-{n}', command('gen', n), limit=n, backpressure=n == 1)
+            measured = run(f'gen-{n}', command('gen', n), limit=n, missing_usage=n == 1, backpressure=n == 1)
             assert measured['total'] == count and measured['peak'] == n, measured
             results('result', 0)
             assert not (root / 'src/lib.rs.bak').exists()
             if n == 1:
                 shutil.copytree(root / 'utgen/generation/pre_fix', seed)
+                measured = run('gen-cached', command('gen', 1))
+                assert measured['total'] == 0
+                report = json.loads((root / 'utgen/generation/gen-requests.json').read_text())
+                assert report['attempts'] == 0 and report['usage_complete'], report
         for n in [1, 2, 4]:
             reset(seed)
-            measured = run(f'fix-{n}', command('fix', n), limit=n)
+            measured = run(f'fix-{n}', command('fix', n), limit=n, missing_usage=n == 1)
             assert measured['total'] == count and measured['peak'] == n, measured
             results('fixed_result', 1)
             assert not (root / 'src/lib.rs.bak').exists()
@@ -269,7 +285,7 @@ def results(directory, compiled):
                     assert not path.exists(), path
         for mode in ['gen', 'fix']:
             reset(seed if mode == 'fix' else None)
-            measured = run(f'{mode}-panic', command(mode, 1), panic=True, fail=True)
+            measured = run(f'{mode}-request-error', command(mode, 1), fault='request', fail=True)
             assert measured['total'] == count, measured  # Remaining workers still finish.
             assert not (root / 'utgen' / ('result' if mode == 'gen' else 'fixed_result')).exists()
             if mode == 'fix':
@@ -281,6 +297,19 @@ def results(directory, compiled):
                 assert measured['total'] == 0
                 assert backup.read_bytes() == b'old recovery material\n'
                 backup.unlink()
+        reset()
+        measured = run('oracle-request-error', command('gen', 1, '--integration', '--oracle'),
+                       integration=True, fault='oracle', fail=True)
+        assert measured['total'] == count * 3 and measured['fault_sent']
+        assert not (root / 'utgen/result').exists()
+        reset()
+        encoded = next(iter(json.loads((root / 'brinfo/name_map.json').read_text()).values()))
+        blocker = root / 'utgen/generation/prompt' / encoded
+        blocker.parent.mkdir(parents=True)
+        blocker.write_text('not a directory')
+        measured = run('gen-worker-panic', command('gen', 1), fail=True)
+        assert measured['total'] == count - 1
+        assert not (root / 'utgen/result').exists()
         # A consumer failure must still drain the queue and let every sender finish.
         reset()
         backup = root / 'src/lib.rs.bak'

utgen/Cargo.toml

@@ -15,6 +15,7 @@ quick-xml = "0.37"
 quote = "1.0"
 rand = "0.9"
 regex = "1.11"
+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 simplelog = "0.12"
@@ -24,8 +25,5 @@ time = "0.3"
 tokio = { version = "1", features = ["full"] }
 walkdir = "2.5"
 
-[dev-dependencies]
-reqwest = { version = "0.12", default-features = false }
-
 [build-dependencies]
 build-utils = { path = "../build-utils" }

utgen/README.md

@@ -32,6 +32,12 @@ An absolute path continues to work after changing into the target crate. Relativ
 
 Changing configuration takes effect on the next invocation without rebuilding. There are no built-in model credentials. Configuration values are not printed in configuration diagnostics.
 
+Generation and repair share one non-streaming request path. `--request-timeout <SECONDS>` defaults to 180 and covers each attempt, including its response body. Transport failures, HTTP 429 (except `insufficient_quota`), and 5xx responses get at most three attempts, with 1- and 2-second delays. Other HTTP errors and malformed/empty answers are returned without network retries. SDK retries are not layered underneath this policy; the existing request/response types are retained while the existing reqwest dependency executes HTTP requests.
+
+A valid answer without `usage` remains usable. Numeric token fields contain reported usage only; missing usage contributes no known tokens, which must not be interpreted as zero actual usage. The command writes `utgen/generation/gen-requests.json` or `fix-requests.json` with attempt/failure counts, missing-usage counts, reported token totals, and `usage_complete`. That flag is false after any failed attempt or response without usage. These files also cover requests whose answers failed later parsing. They are written after workers finish, including request failures; successful cached runs record zero attempts. Validation failures before any request leave the target untouched.
+
+Formatting retries and compiler-guided repair rounds remain separate from transport retries. A final request failure, including during oracle generation or repair, fails the command and skips subsequent statistics. The request deadline does not terminate Cargo processes or impose a whole-function deadline.
+
 Install the toolchain and coverage tool described in the [project README](../README.md#prerequisites). On a fresh working copy, run `utgen pre-process -p <target-crate-path>` and then `utgen analyze -p <target-crate-path>` to produce:
 
 ```text
@@ -139,6 +145,7 @@ utgen gen -p <target-crate-path> --requirement --context
 | `-o, --oracle` | Use separate input-range, test-prefix, and oracle generation. Default: off; otherwise generate complete tests directly. |
 | `-i, --integration` | Generate integration tests under `tests/`, using the analysis visibility flag to select functions and compilation checks to filter candidates. Default: off. |
 | `-t, --tasks` | Default: 4. Maximum active focal-function generation jobs; must be positive. |
+| `--request-timeout` | Default: 180 seconds per model request attempt; also available for `fix`. Must be positive. |
 
 Generation validates the branch index, context index, context files, and source paths before modifying the target. Missing or inconsistent artifacts are errors. A failed generation task is reported instead of being silently lost before statistics. `--tasks 0` is rejected.
 
@@ -156,7 +163,7 @@ Repair reads the generated candidates and uses compiler diagnostics to revise th
 
 Source insertion, target cleanup, compilation/test execution, diagnostic reading, and source restoration are serialized within each command. Model requests may overlap. Cargo retains its own dependency-build parallelism. Concurrent PALM commands or experiments require separate working copies and separate target directories.
 
-All workers are joined before cleanup. A task panic or infrastructure error returns a nonzero status and skips subsequent coverage statistics; candidate compilation errors remain ordinary repair outcomes. Repair restores its source backups before returning, deletes only backups created by that invocation on success, and retains them after a worker failure. Existing source backups are rejected without overwriting them. Temporary import/candidate files are restored or removed after validation, including unwinding. This is not recovery from forced process termination, and request/subprocess deadlines remain unchanged.
+All workers are joined before cleanup. A task panic or infrastructure error returns a nonzero status and skips subsequent coverage statistics; candidate compilation errors remain ordinary repair outcomes. Repair restores its source backups before returning, deletes only backups created by that invocation on success, and retains them after a worker failure. Existing source backups are rejected without overwriting them. Temporary import/candidate files are restored or removed after validation, including unwinding. This is not recovery from forced process termination. Model request deadlines are described above; Cargo subprocess deadlines remain separate work.
 
 There is no integration-mode option for repair. Candidates are inserted into source files as unit tests, and post-repair statistics use that same mode. Do not interpret `gen --integration` followed by `fix` as preserving an integration-only evaluation.
 
@@ -169,6 +176,7 @@ Paths under `utgen/` below are relative to `--project-dir`:
 | `utgen/generation/prompt/` and `answer/` | Prompts and model responses. |
 | `utgen/generation/pre_fix/` | Candidate tests and their compilation status before repair. |
 | `utgen/generation/llm_fix/` | Candidates and their compilation status after repair. |
+| `utgen/generation/gen-requests.json` and `fix-requests.json` | Request attempts, failures, reported token totals, and usage completeness for the invocation. |
 | `utgen/result/` | Pre-repair coverage and execution statistics per focal function. |
 | `utgen/fixed_result/` | Post-repair coverage and execution statistics per focal function. |
 | `utgen/original_result.json` | Comparison statistics when the original integration-test backup is available. |

utgen/src/gene/cot/input_infer.rs

@@ -1,49 +1,30 @@
-use super::Prompt;
 use super::LLM;
-use log::{error, info};
-use rand::Rng;
-use tokio::time::{sleep, Duration};
+use super::Prompt;
+use log::error;
 
 fn postprocess(inputs: &mut Vec<String>) {
     for input in inputs.iter_mut() {
         *input = input.replace("```rust", "").replace("```", "");
     }
 }
 
-pub async fn gen_input_range(llm: &LLM, pt_info: &Prompt, conds: &Vec<String>) -> Option<(String, u32, u32)> {
+pub async fn gen_input_range(
+    llm: &LLM,
+    pt_info: &Prompt,
+    conds: &Vec<String>,
+) -> Option<(String, u32, u32)> {
     let system_pt = &pt_info.system_pt;
     let static_pt = &pt_info.static_pt;
-    let mut completion_tokens = 0;
-    let mut prompt_tokens = 0;
 
     let user_pt = static_pt.clone() + &conds.join("");
-    let mut retry = 0;
-    let mut answers = None;
-    while retry < 3 {
-        if retry != 0 {
-            let random_secs = {
-                let mut rng = rand::rng();
-                rng.random_range(10..=30)
-            };
-            sleep(Duration::from_secs(random_secs)).await;
-        }
-        let result = llm.fetch_answer(Some(&system_pt), &user_pt, 1, false).await;
-        if result.is_ok() {
-            let (result_answers, usage_completion, usage_prompt) = result.unwrap();
-            completion_tokens += usage_completion;
-            prompt_tokens += usage_prompt;
-            answers = Some(result_answers);
-            break;
-        }
-        error!("{}. Retrying...", result.unwrap_err());
-        retry += 1;
-    }
-    if answers.is_none() {
-        error!("Failed to fetch answer.");
-        return None;
-    }
-    let mut answers = answers.unwrap();
-    assert!(answers.len() == 1);
+    let (mut answers, completion_tokens, prompt_tokens) =
+        match llm.fetch_answer(Some(system_pt), &user_pt, 1, false).await {
+            Ok(answer) => answer,
+            Err(error) => {
+                error!("Model request failed: {error}");
+                return None;
+            }
+        };
 
     postprocess(&mut answers);
     // info!("Answers: {:?}", answers);