fix: lint/CI、agent 稳定性与 swe-bench 评估逻辑修复 - #9
Merged
Conversation
added 30 commits
June 16, 2026 23:52
- Add 6 complex end-to-end workflow tests covering refactor/index consistency, bug-fix pipeline, dangerous-op decline/approve, todo-driven multi-file task, patch rollback recovery, and REPL history persistence - Add scripts/run_e2e.py to run e2e tests and report accuracy - Update GitHub Actions CI with dedicated e2e job - All e2e tests pass: 8/8 (100.0%)
- 实现流式输出与工具调用过程可视化 - 修复流式响应中 tool_call_id 为空/不匹配导致的 400 错误 - REPL 统一保存 assistant message,避免重复与历史不一致 - 启动时自动加载工作目录 .env - Kimi Code 兼容:自动注入 User-Agent,强制 temperature=1.0 - 修复 set_todo 循环导入 - 补全流式与工具调用相关单元测试
- REPL 捕获 LLMError/KeyboardInterrupt/通用异常,避免 API 失败直接崩溃 - config.toml 从 workspace 目录加载,而非当前工作目录 - 历史加载时校验完整性,自动丢弃崩溃残留的 assistant(tool_calls) - apply_patch 确认前展示 diff 变更摘要(文件、增删行数) - execute_shell 禁止永久放行,每次危险 shell 都需确认 新增/更新测试覆盖以上修复
P0 - 核心交互: - /sessions /switch /rename /delete 会话管理 - /tokens /history 信息显示 - 写操作自动备份 + /undo 撤销 - --run batch 模式 + 非零退出码 P1 - 增强可用性: - /compact 手动上下文压缩 + 自动压缩 - /reload 配置热重载 - /git git 状态感知 - 每次 turn 后 token 累计显示 - 工具失败后自动重试(排除 forbidden/用户拒绝) P2 - 扩展能力: - 自定义 system prompt - 结构化日志系统 - 真实 LLM 冒烟测试(默认跳过) - MCP client 支持(实验性) 新增文件: - agent/context.py - agent/logging_config.py - agent/mcp_client.py - agent/tools/mcp_adapter.py - tests/test_context.py - tests/test_logging_config.py - tests/test_mcp_client.py - tests/smoke/test_llm_smoke.py 测试:235 passed, 3 skipped, e2e 8 passed
- _process_user_input 保存 user message 后自动设置会话标题 - 标题取消息前 30 字,超过部分截断为 ... - /sessions 不再满屏显示'未命名' 测试:236 passed, 3 skipped
- agent/llm/parser.py 的 _parse_tool_call 在 raw.id 为空时生成 fallback id - agent/repl.py 的 _load_history 清洗 tool_call_id 为空或不匹配的脏 tool 消息 - 防止旧历史或异常响应导致 'tool_call_id is not found' 400 错误 测试:238 passed, 3 skipped
- agent/llm/client.py: _prepare_messages 记录发送的 tool_calls/tool_call_id - agent/llm/client.py: LLM 失败时记录错误 - agent/repl.py: 历史加载、工具调用、tool result 消息增加 debug 日志 - agent/repl.py: LLMError 时记录当前消息链的 tool_call_id 分布 测试:238 passed, 3 skipped
…esign - Update main design doc with current architecture, tool set, and roadmap - Update config spec with timeout and YOLO defaults - Update LLM protocol spec with streaming, timeout, empty-message fallback - Update persistence spec with file permissions and compaction notes - Update safety spec with YOLO mode and /yolo command - Update tool schema spec with P2 tools and multi-agent permission model - Mark P1 plan as completed, add status to P2 plan - Add P5 multi-agent and /goals spec
Prevent local CODING_AGENT_* env vars from leaking into this test.
- Add Goal, AgentRole, IPCMessage data models - Add GoalPersistence with create/get/update/list/cancel/resume - Add unit tests for models and persistence
- IPCServer accepts sequential client connections and routes messages - IPCClient connects, sends, and receives newline-delimited JSON messages - Supports reconnect and invalid-message recovery - Includes unit tests
- RoleLoader reads agents/*.yaml with built-in fallback roles - Add 6 default roles: default, architect, coder, reviewer, tester, git - Add pyyaml dependency - Add unit tests
- Worker connects to supervisor, receives ASSIGN_GOAL, runs LLM loop - All tool calls are sent as TOOL_REQUEST to supervisor and executed there - Supports role-based tool allowlists - Add worker_main.py subprocess entry point - Add integration test with mock LLM and IPC server - Fix persistence.list naming conflict with built-in type - Fix IPC mypy union-attr warnings
- Supervisor manages goals, spawns workers, and handles IPC - Scheduler tracks goal dependencies and returns ready goals - Worker sends READY message; supervisor assigns pending goal on connect - Supervisor executes tool requests from workers using local tool registry - Add integration test with mock worker thread - Add MessageType.READY for worker-supervisor handshake
- Add Supervisor instance to REPL with workspace-local goals.db - Add /goals commands: list, all, add, show, cancel, resume, clear-done - Add /agent commands: list roles and switch current role - Detect complex inputs and route them through supervisor - Supervisor executes tool requests from workers - Add REPL supervisor integration tests
- Harden _connect_mcp to require enabled=True and command to be a non-empty str - Ignore malformed args instead of passing them to MCPClient - Prevents MagicMock config from triggering Pydantic validation errors
- supervisor: remove duplicated log_dir block (copy-paste leftover) and close the worker log file handle when the forward thread exits, fixing an fd leak per spawned worker. - worker: _wait_for now uses a total deadline instead of resetting the timeout on every receive(), so interleaved heartbeats no longer extend the wait indefinitely.
- environment: make the pip index URL configurable via SWE_BENCH_PIP_INDEX_URL (defaults to the Tsinghua mirror) so runs outside China can use a closer/default mirror; fix the install script that accidentally embedded an assignment inside a list literal. - runner: _run_goal now returns a timed-out flag instead of raising, so a timed-out goal still produces/evaluates a partial patch; empty patches become a resolved=False TaskResult instead of crashing the run. - docker: document the local-build fallback semantics (host workspace mounted at /testbed) and its divergence from the official harness. - scripts: add build_swe_bench_base_image / setup_colima_docker and fix ruff lint (unused import, long line).
…er data Six end-to-end REPL tests were missing the isolated_home fixture, so they read/wrote the real ~/.coding-agent/history.db (154MB, thousands of messages) instead of a per-test temp DB. This both made the tests flaky (loaded real history into the message list, causing count assertions like '6 == 3' to fail) and silently mutated the developer's actual history.
- .gitignore: exclude output/, data/, logs/, swe-bench caches and tmp_* so multi-GB benchmark outputs and repo clones are never accidentally committed. - README: complete the REPL slash-command table (was missing /compact, /tokens, /sessions, /switch, /rename, /delete, /undo, /git, /mcp, /reload, /history).
…vider - load_dotenv now uses override=True in both REPL and swe_bench CLI, so the .env file (the user's latest intent) wins over stale CODING_AGENT_LLM_* exports lingering in the shell. Previously editing .env had no effect when the same vars were already exported, making the documented .env workflow silently broken. - LLMConfig.provider was whitelisted to only 'kimi'/'openai' even though it is purely a display label (no code branches on it). Allow any non-empty value so users can point at Volces/DeepSeek/local servers without lying about the provider name.
HistoryManager.prune_old_sessions(keep=200) deletes the oldest sessions beyond a retention limit and VACUUMs the freed space. REPL calls it best-effort on startup (env: CODING_AGENT_HISTORY_KEEP, 0 disables). Motivated by a 154MB history.db after sustained local use.
- Default per-task timeout 600s -> 1200s; overridable via SWE_BENCH_TASK_TIMEOUT env so real LLM agents aren't cut off mid-fix. - DockerEvaluator container pip mirror now reads SWE_BENCH_PIP_INDEX_URL (matching the host environment.py), instead of a hardcoded Tsinghua URL. - CI sets SWE_BENCH_PIP_INDEX_URL to official PyPI since runners are outside China.
…n-UTF-8 files - read_file: check file size before reading (refuse >10MB to avoid OOM); decode with errors='replace' so binary/mixed-encoding files don't raise UnicodeDecodeError (which is a ValueError, not OSError, and would escape the tool's except-OSError handler). - read_multiple_files: a single bad/missing file no longer aborts the whole batch — the error is recorded inline and readable files are still returned; same large-file and UTF-8 hardening; stop once output budget is hit. - str_replace_file: catch UnicodeDecodeError separately with a clear message (it only supports text files).
subprocess.run(text=True) defaults to the locale encoding and can raise UnicodeDecodeError on commands that emit non-UTF-8 bytes. Pin encoding=utf-8 with errors=replace so garbled output is degraded gracefully instead of crashing the tool.
- code_search: skip .git/__pycache__/node_modules/venv/build dirs and binary file extensions; cap matches at 200 and skip files >2MB. The previous version walked every file including .git/objects (tens of thousands of files), making searches extremely slow. - context: estimate_tokens now weights CJK characters at ~1 token each instead of len//4 (which counted a Chinese char as 0.25 token). The old estimate severely under-counted Chinese text, causing is_near_limit to think there was headroom when the context was actually over the limit, triggering LLM 400 errors.
…hardening - fetch_url / web_search: these call Moonshot-specific /fetch and /search endpoints. When a non-Kimi provider is configured (e.g. Volces), fail fast with a clear message instead of sending a request that 404s. - symbol_search: wrap indexer call in try/except (corrupt/missing index no longer crashes the tool) and cap results at 100.
Previously read_file read the whole file then truncated to 5000 chars from the start — the agent had no way to read the middle or end of a large file. Now read_file streams line by line (constant memory) and supports: - offset: 0-based starting line, for paging into later parts of a file - limit: max lines to return (default 2000) - line numbers in output so the agent knows exact positions - metadata with total_lines, has_more, next_offset for easy pagination - character budget still enforced to protect LLM context - MAX_READ_BYTES raised to 50MB since we only hold cputime unlimited filesize unlimited datasize unlimited stacksize 7MB coredumpsize 0kB addressspace unlimited memorylocked unlimited maxproc 2666 descriptors 65535 lines in memory
Add a new --mode docker-bash execution mode for SWE-bench that runs the agent directly inside the official Docker container with a single execute_shell tool, inspired by mini-swe-agent. This sidesteps the IPC + multi-tool + conda-env fragility of the default supervisor pipeline. Key components: - DockerBashAgent: minimal LLM → bash-in-docker → observation loop. Single tool, no IPC, no worker subprocess. Observations use head+tail truncation (10KB) so error messages at the end of output are visible. - DockerShell: wraps container.exec_run with UTF-8-safe decoding and blocks destructive git commands (git checkout <file>, git stash, git reset --hard) that the agent kept using to destroy its own edits, producing empty patches. - runner: --mode docker-bash starts the container, resets to base_commit, runs the agent, collects git diff (including untracked files), and evaluates in the same container. Verified on pytest-dev__pytest-5103: agent produced a 1474-byte patch (modifying rewrite.py) and completed full docker evaluation, vs 0/24 empty patches in the previous supervisor mode batch.
- runner.py: use gitee.com/mirrors for git clone (depth 500), fetch_commit checks local cache first before hitting network, 15s timeout for all git network ops - docker.py: _ensure_image pull timeout 5s to fail fast on blocked Docker Hub - supervisor.py: _safe_execute wrapper prevents tool exceptions from killing worker IPC connection (was causing empty patches) - set_todo.py: graceful ValueError handling for missing todo IDs - .env: switch to deepseek-v4-pro Co-Authored-By: Claude <noreply@anthropic.com>
- coder.yaml: rewrite system prompt with workflow guide, tool usage rules, explicit prohibitions (no pyproject.toml, no pip install, no env debug) - runner.py: simplify goal description, remove mandatory test-running steps that wasted ~30% of turns, add explicit constraints - config.toml: reduce max_steps_per_turn 100->50 for v4-pro reasoning model - coder.yaml: remove write_file/apply_patch/ask_user/symbol_search/ find_definition/find_references from allowed tools (too many confused model) Co-Authored-By: Claude <noreply@anthropic.com>
Agent frequently pins setuptools version in pyproject.toml during debugging, which breaks evaluation. Post-process patches to remove hunks touching pyproject.toml, setup.cfg, setup.py, Makefile, tox.ini, .github/, etc. Co-Authored-By: Claude <noreply@anthropic.com>
- Remove trailing blank/whitespace lines after stripping config hunks - Ensure patch always ends with exactly one newline - Config patterns: pyproject.toml, setup.cfg, setup.py, tox.ini, Makefile, .github/, .circleci/, .travis.yml, conftest.py Co-Authored-By: Claude <noreply@anthropic.com>
Stripping config hunks from unified diffs breaks the context line count, leading to 'patch does not apply cleanly' errors. The pyproject.toml pollution does not affect SWE-bench evaluation (test env uses Docker images with its own build deps). Kept _strip_config_changes function for potential future use with a more robust implementation. Co-Authored-By: Claude <noreply@anthropic.com>
- coder.yaml: detailed 5-phase workflow, tool cheat sheet, explicit DON'Ts - str_replace_file.py: descriptive tool hint (recommended over execute_shell+sed) - execute_shell.py: explicit warning not to use for file reading/editing Co-Authored-By: Claude <noreply@anthropic.com>
Extracted from Claude Code's prompts.ts (914-line prompt engineering): - CRITICAL tool usage rules: dedicated tools over execute_shell, with table - Code modification principles: no extra features, no speculative abstractions, no error handling for impossible scenarios, don't edit unread code - Comment policy: default to no comments, explain WHY not WHAT - Error handling: diagnose before switching, don't blindly retry, no --force - Reporting: honest outcomes, verify before claiming complete - Task management: break down with set_todo, mark done immediately Co-Authored-By: Claude <noreply@anthropic.com>
Add DirectAgent that skips the supervisor/worker/IPC pipeline entirely. LLM calls tools directly in-process — same architecture as Claude Code. - agent/direct_agent.py: in-process LLM loop with direct tool calls - runner.py: _run_task_direct mode (no subprocess, no IPC serialization) - cli.py: --mode direct option Co-Authored-By: Claude <noreply@anthropic.com>
- build_tools_payload is in agent.llm.parser, not agent.tools - ToolContext takes 'workspace' kwarg, not 'workspace_path' Co-Authored-By: Claude <noreply@anthropic.com>
In direct mode, there is no user to confirm dangerous shell commands.
execute_shell calls (pytest, git diff, cd) were all being blocked by
the safety classifier ('requires user confirmation'). Use execute_forced
to auto-approve, matching supervisor's confirm_callback=lambda: True.
Also added detailed tool call logging for debugging.
Co-Authored-By: Claude <noreply@anthropic.com>
v4-pro at temperature=0.7 tends to get distracted by test failures and wanders into environment debugging loops. Temperature=0 makes it more focused on the source code fix. Co-Authored-By: Claude <noreply@anthropic.com>
…use analysis Documents every experiment: git mirror, tool reduction, prompt alignment, supervisor removal, config stripping attempt, temperature tuning. Explains why v4-pro converges at 20% regardless of architecture: model attention drift in multi-turn conversations, triggered by test failure output in tool results. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
排查项目时发现一批影响 CI、运行稳定性、配置正确性和测试卫生的问题,本 PR 一并修复。所有改动已通过 `ruff check` / `ruff format --check` / `mypy` / `pytest (308 passed, 3 skipped)`。
改动
🔴 配置正确性(实际影响用户)
🔴 CI / 功能性
🟠 swe-bench 评估
🟡 测试卫生(顺带修的真实 bug)
🟢 工程卫生
验证
```
ruff check → All checks passed
ruff format --check → 104 files already formatted
mypy agent tests → Success: no issues found in 89 source files
pytest tests/ → 308 passed, 3 skipped
```
Volces GLM-5.2 连通性已验证(`base_url=https://ark.cn-beijing.volces.com/api/coding/v3\`)。