Skip to content

fix: lint/CI、agent 稳定性与 swe-bench 评估逻辑修复 - #9

Merged
wq19901103wq merged 93 commits into
mainfrom
feat/swe-bench-m1
Jun 28, 2026
Merged

fix: lint/CI、agent 稳定性与 swe-bench 评估逻辑修复#9
wq19901103wq merged 93 commits into
mainfrom
feat/swe-bench-m1

Conversation

@wq19901103wq

@wq19901103wq wq19901103wq commented Jun 22, 2026

Copy link
Copy Markdown
Owner

背景

排查项目时发现一批影响 CI、运行稳定性、配置正确性和测试卫生的问题,本 PR 一并修复。所有改动已通过 `ruff check` / `ruff format --check` / `mypy` / `pytest (308 passed, 3 skipped)`。

改动

🔴 配置正确性(实际影响用户)

  • load_dotenv override=True:REPL 和 swe_bench CLI 之前用 `override=False`,导致用户在 `.env` 里写的最新配置会被 shell 里陈旧的 `CODING_AGENT_LLM_*` export 静默盖掉——编辑 `.env` 完全不生效。改成 `override=True`,让 `.env`(用户最新意图)优先。
  • provider 白名单放宽:`LLMConfig.provider` 之前硬编码只允许 `kimi`/`openai`,但它纯粹是个显示标签(代码不分支)。改成允许任意非空字符串,这样用户能配 Volces / DeepSeek / 本地服务器而不用撒谎。

🔴 CI / 功能性

  • agent/supervisor:删除 `_default_spawn_worker` 里复制粘贴遗留的重复 `log_dir` 代码块;worker 日志文件句柄在转发线程结束时正确 `close()`,修复每个 worker 一次的 fd 泄漏。
  • agent/worker:`_wait_for` 改用总 deadline 而非每次 `receive` 重置超时,避免心跳/状态消息穿插导致实际等待远超设定值。

🟠 swe-bench 评估

  • timeout 调大并可配置:默认 600s → 1200s,可通过 `SWE_BENCH_TASK_TIMEOUT` 环境变量覆盖;真实 LLM agent 不再被中途切断。
  • environment / docker:pip 镜像改为可配置(`SWE_BENCH_PIP_INDEX_URL`),host 和容器内一致;修复安装脚本里误把赋值语句写进列表字面量的语法错误。
  • runner:`_run_goal` 超时后返回 `timed_out` 标志而非抛异常,让超时分支也能收集并评估 agent 已完成的部分 patch;空 patch 改为返回 `resolved=False` 的 `TaskResult` 而非抛异常导致整个 run 崩溃。
  • docker:为本地构建 fallback 补充文档说明,讲清"挂载宿主工作区到 /testbed"与官方 harness 语义的差异和权衡。
  • CI:GitHub Actions runner 在海外,显式设 `SWE_BENCH_PIP_INDEX_URL=https://pypi.org/simple\`。

🟡 测试卫生(顺带修的真实 bug)

  • tests/test_repl:6 个 e2e 测试缺少 `isolated_home` fixture,一直在读写真实用户的 `~/.coding-agent/history.db`(154MB、4781 条 tool 消息)。既让测试 flaky(把真实历史加载进 message 列表,导致断言如 `6 == 3` 失败),也默默污染开发者的真实历史数据。补上 `isolated_home` 后用隔离的临时 DB。

🟢 工程卫生

  • history 轮转:`HistoryManager.prune_old_sessions(keep=200)` 删除超出保留数量的旧 session 并 `VACUUM` 回收空间;REPL 启动时 best-effort 触发(env `CODING_AGENT_HISTORY_KEEP=0` 禁用)。起因是 `history.db` 持续使用后膨胀到 154MB。
  • .gitignore:补齐 `output/ data/ logs/ swe-bench-output/ tmp_*/`,避免 GB 级 benchmark 产物和仓库 clone 被误提交。
  • README:补全 REPL 命令表(9 → 22 条)。
  • 清理根目录临时调试脚本 `tmp_manual_supervisor.py`、`test_llm_call.py`。

验证

```
ruff check → All checks passed
ruff format --check → 104 files already formatted
mypy agent tests → Success: no issues found in 89 source files
pytest tests/ → 308 passed, 3 skipped
```

Volces GLM-5.2 连通性已验证(`base_url=https://ark.cn-beijing.volces.com/api/coding/v3\`)。

Coding Agent added 30 commits June 16, 2026 23:52
- Add 6 complex end-to-end workflow tests covering refactor/index consistency,
  bug-fix pipeline, dangerous-op decline/approve, todo-driven multi-file task,
  patch rollback recovery, and REPL history persistence
- Add scripts/run_e2e.py to run e2e tests and report accuracy
- Update GitHub Actions CI with dedicated e2e job
- All e2e tests pass: 8/8 (100.0%)
- 实现流式输出与工具调用过程可视化
- 修复流式响应中 tool_call_id 为空/不匹配导致的 400 错误
- REPL 统一保存 assistant message,避免重复与历史不一致
- 启动时自动加载工作目录 .env
- Kimi Code 兼容:自动注入 User-Agent,强制 temperature=1.0
- 修复 set_todo 循环导入
- 补全流式与工具调用相关单元测试
- REPL 捕获 LLMError/KeyboardInterrupt/通用异常,避免 API 失败直接崩溃
- config.toml 从 workspace 目录加载,而非当前工作目录
- 历史加载时校验完整性,自动丢弃崩溃残留的 assistant(tool_calls)
- apply_patch 确认前展示 diff 变更摘要(文件、增删行数)
- execute_shell 禁止永久放行,每次危险 shell 都需确认

新增/更新测试覆盖以上修复
P0 - 核心交互:
- /sessions /switch /rename /delete 会话管理
- /tokens /history 信息显示
- 写操作自动备份 + /undo 撤销
- --run batch 模式 + 非零退出码

P1 - 增强可用性:
- /compact 手动上下文压缩 + 自动压缩
- /reload 配置热重载
- /git git 状态感知
- 每次 turn 后 token 累计显示
- 工具失败后自动重试(排除 forbidden/用户拒绝)

P2 - 扩展能力:
- 自定义 system prompt
- 结构化日志系统
- 真实 LLM 冒烟测试(默认跳过)
- MCP client 支持(实验性)

新增文件:
- agent/context.py
- agent/logging_config.py
- agent/mcp_client.py
- agent/tools/mcp_adapter.py
- tests/test_context.py
- tests/test_logging_config.py
- tests/test_mcp_client.py
- tests/smoke/test_llm_smoke.py

测试:235 passed, 3 skipped, e2e 8 passed
- _process_user_input 保存 user message 后自动设置会话标题
- 标题取消息前 30 字,超过部分截断为 ...
- /sessions 不再满屏显示'未命名'

测试:236 passed, 3 skipped
- agent/llm/parser.py 的 _parse_tool_call 在 raw.id 为空时生成 fallback id
- agent/repl.py 的 _load_history 清洗 tool_call_id 为空或不匹配的脏 tool 消息
- 防止旧历史或异常响应导致 'tool_call_id is not found' 400 错误

测试:238 passed, 3 skipped
- agent/llm/client.py: _prepare_messages 记录发送的 tool_calls/tool_call_id
- agent/llm/client.py: LLM 失败时记录错误
- agent/repl.py: 历史加载、工具调用、tool result 消息增加 debug 日志
- agent/repl.py: LLMError 时记录当前消息链的 tool_call_id 分布

测试:238 passed, 3 skipped
…esign

- Update main design doc with current architecture, tool set, and roadmap
- Update config spec with timeout and YOLO defaults
- Update LLM protocol spec with streaming, timeout, empty-message fallback
- Update persistence spec with file permissions and compaction notes
- Update safety spec with YOLO mode and /yolo command
- Update tool schema spec with P2 tools and multi-agent permission model
- Mark P1 plan as completed, add status to P2 plan
- Add P5 multi-agent and /goals spec
Prevent local CODING_AGENT_* env vars from leaking into this test.
- Add Goal, AgentRole, IPCMessage data models
- Add GoalPersistence with create/get/update/list/cancel/resume
- Add unit tests for models and persistence
- IPCServer accepts sequential client connections and routes messages
- IPCClient connects, sends, and receives newline-delimited JSON messages
- Supports reconnect and invalid-message recovery
- Includes unit tests
- RoleLoader reads agents/*.yaml with built-in fallback roles
- Add 6 default roles: default, architect, coder, reviewer, tester, git
- Add pyyaml dependency
- Add unit tests
- Worker connects to supervisor, receives ASSIGN_GOAL, runs LLM loop
- All tool calls are sent as TOOL_REQUEST to supervisor and executed there
- Supports role-based tool allowlists
- Add worker_main.py subprocess entry point
- Add integration test with mock LLM and IPC server
- Fix persistence.list naming conflict with built-in type
- Fix IPC mypy union-attr warnings
- Supervisor manages goals, spawns workers, and handles IPC
- Scheduler tracks goal dependencies and returns ready goals
- Worker sends READY message; supervisor assigns pending goal on connect
- Supervisor executes tool requests from workers using local tool registry
- Add integration test with mock worker thread
- Add MessageType.READY for worker-supervisor handshake
- Add Supervisor instance to REPL with workspace-local goals.db
- Add /goals commands: list, all, add, show, cancel, resume, clear-done
- Add /agent commands: list roles and switch current role
- Detect complex inputs and route them through supervisor
- Supervisor executes tool requests from workers
- Add REPL supervisor integration tests
- Harden _connect_mcp to require enabled=True and command to be a non-empty str
- Ignore malformed args instead of passing them to MCPClient
- Prevents MagicMock config from triggering Pydantic validation errors
Coding Agent and others added 29 commits June 20, 2026 19:24
- supervisor: remove duplicated log_dir block (copy-paste leftover) and
  close the worker log file handle when the forward thread exits, fixing an
  fd leak per spawned worker.
- worker: _wait_for now uses a total deadline instead of resetting the
  timeout on every receive(), so interleaved heartbeats no longer extend
  the wait indefinitely.
- environment: make the pip index URL configurable via
  SWE_BENCH_PIP_INDEX_URL (defaults to the Tsinghua mirror) so runs outside
  China can use a closer/default mirror; fix the install script that
  accidentally embedded an assignment inside a list literal.
- runner: _run_goal now returns a timed-out flag instead of raising, so a
  timed-out goal still produces/evaluates a partial patch; empty patches
  become a resolved=False TaskResult instead of crashing the run.
- docker: document the local-build fallback semantics (host workspace
  mounted at /testbed) and its divergence from the official harness.
- scripts: add build_swe_bench_base_image / setup_colima_docker and fix
  ruff lint (unused import, long line).
…er data

Six end-to-end REPL tests were missing the isolated_home fixture, so they
read/wrote the real ~/.coding-agent/history.db (154MB, thousands of
messages) instead of a per-test temp DB. This both made the tests flaky
(loaded real history into the message list, causing count assertions like
'6 == 3' to fail) and silently mutated the developer's actual history.
- .gitignore: exclude output/, data/, logs/, swe-bench caches and tmp_*
  so multi-GB benchmark outputs and repo clones are never accidentally
  committed.
- README: complete the REPL slash-command table (was missing /compact,
  /tokens, /sessions, /switch, /rename, /delete, /undo, /git, /mcp,
  /reload, /history).
…vider

- load_dotenv now uses override=True in both REPL and swe_bench CLI, so the
  .env file (the user's latest intent) wins over stale CODING_AGENT_LLM_*
  exports lingering in the shell. Previously editing .env had no effect when
  the same vars were already exported, making the documented .env workflow
  silently broken.
- LLMConfig.provider was whitelisted to only 'kimi'/'openai' even though it
  is purely a display label (no code branches on it). Allow any non-empty
  value so users can point at Volces/DeepSeek/local servers without lying
  about the provider name.
HistoryManager.prune_old_sessions(keep=200) deletes the oldest sessions
beyond a retention limit and VACUUMs the freed space. REPL calls it
best-effort on startup (env: CODING_AGENT_HISTORY_KEEP, 0 disables).
Motivated by a 154MB history.db after sustained local use.
- Default per-task timeout 600s -> 1200s; overridable via
  SWE_BENCH_TASK_TIMEOUT env so real LLM agents aren't cut off mid-fix.
- DockerEvaluator container pip mirror now reads SWE_BENCH_PIP_INDEX_URL
  (matching the host environment.py), instead of a hardcoded Tsinghua URL.
- CI sets SWE_BENCH_PIP_INDEX_URL to official PyPI since runners are
  outside China.
…n-UTF-8 files

- read_file: check file size before reading (refuse >10MB to avoid OOM);
  decode with errors='replace' so binary/mixed-encoding files don't raise
  UnicodeDecodeError (which is a ValueError, not OSError, and would escape
  the tool's except-OSError handler).
- read_multiple_files: a single bad/missing file no longer aborts the whole
  batch — the error is recorded inline and readable files are still returned;
  same large-file and UTF-8 hardening; stop once output budget is hit.
- str_replace_file: catch UnicodeDecodeError separately with a clear message
  (it only supports text files).
subprocess.run(text=True) defaults to the locale encoding and can raise
UnicodeDecodeError on commands that emit non-UTF-8 bytes. Pin encoding=utf-8
with errors=replace so garbled output is degraded gracefully instead of
crashing the tool.
- code_search: skip .git/__pycache__/node_modules/venv/build dirs and
  binary file extensions; cap matches at 200 and skip files >2MB. The
  previous version walked every file including .git/objects (tens of
  thousands of files), making searches extremely slow.
- context: estimate_tokens now weights CJK characters at ~1 token each
  instead of len//4 (which counted a Chinese char as 0.25 token). The old
  estimate severely under-counted Chinese text, causing is_near_limit to
  think there was headroom when the context was actually over the limit,
  triggering LLM 400 errors.
…hardening

- fetch_url / web_search: these call Moonshot-specific /fetch and /search
  endpoints. When a non-Kimi provider is configured (e.g. Volces), fail
  fast with a clear message instead of sending a request that 404s.
- symbol_search: wrap indexer call in try/except (corrupt/missing index
  no longer crashes the tool) and cap results at 100.
Previously read_file read the whole file then truncated to 5000 chars from
the start — the agent had no way to read the middle or end of a large file.

Now read_file streams line by line (constant memory) and supports:
- offset: 0-based starting line, for paging into later parts of a file
- limit: max lines to return (default 2000)
- line numbers in output so the agent knows exact positions
- metadata with total_lines, has_more, next_offset for easy pagination
- character budget still enforced to protect LLM context
- MAX_READ_BYTES raised to 50MB since we only hold cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       7MB
coredumpsize    0kB
addressspace    unlimited
memorylocked    unlimited
maxproc         2666
descriptors     65535 lines in memory
Add a new --mode docker-bash execution mode for SWE-bench that runs the
agent directly inside the official Docker container with a single
execute_shell tool, inspired by mini-swe-agent. This sidesteps the
IPC + multi-tool + conda-env fragility of the default supervisor pipeline.

Key components:
- DockerBashAgent: minimal LLM → bash-in-docker → observation loop. Single
  tool, no IPC, no worker subprocess. Observations use head+tail truncation
  (10KB) so error messages at the end of output are visible.
- DockerShell: wraps container.exec_run with UTF-8-safe decoding and blocks
  destructive git commands (git checkout <file>, git stash, git reset --hard)
  that the agent kept using to destroy its own edits, producing empty patches.
- runner: --mode docker-bash starts the container, resets to base_commit,
  runs the agent, collects git diff (including untracked files), and
  evaluates in the same container.

Verified on pytest-dev__pytest-5103: agent produced a 1474-byte patch
(modifying rewrite.py) and completed full docker evaluation, vs 0/24 empty
patches in the previous supervisor mode batch.
- runner.py: use gitee.com/mirrors for git clone (depth 500), fetch_commit checks
  local cache first before hitting network, 15s timeout for all git network ops
- docker.py: _ensure_image pull timeout 5s to fail fast on blocked Docker Hub
- supervisor.py: _safe_execute wrapper prevents tool exceptions from killing
  worker IPC connection (was causing empty patches)
- set_todo.py: graceful ValueError handling for missing todo IDs
- .env: switch to deepseek-v4-pro

Co-Authored-By: Claude <noreply@anthropic.com>
- coder.yaml: rewrite system prompt with workflow guide, tool usage rules,
  explicit prohibitions (no pyproject.toml, no pip install, no env debug)
- runner.py: simplify goal description, remove mandatory test-running steps
  that wasted ~30% of turns, add explicit constraints
- config.toml: reduce max_steps_per_turn 100->50 for v4-pro reasoning model
- coder.yaml: remove write_file/apply_patch/ask_user/symbol_search/
  find_definition/find_references from allowed tools (too many confused model)

Co-Authored-By: Claude <noreply@anthropic.com>
Agent frequently pins setuptools version in pyproject.toml during debugging,
which breaks evaluation. Post-process patches to remove hunks touching
pyproject.toml, setup.cfg, setup.py, Makefile, tox.ini, .github/, etc.

Co-Authored-By: Claude <noreply@anthropic.com>
- Remove trailing blank/whitespace lines after stripping config hunks
- Ensure patch always ends with exactly one newline
- Config patterns: pyproject.toml, setup.cfg, setup.py, tox.ini,
  Makefile, .github/, .circleci/, .travis.yml, conftest.py

Co-Authored-By: Claude <noreply@anthropic.com>
Stripping config hunks from unified diffs breaks the context line count,
leading to 'patch does not apply cleanly' errors. The pyproject.toml
pollution does not affect SWE-bench evaluation (test env uses Docker
images with its own build deps).

Kept _strip_config_changes function for potential future use with a
more robust implementation.

Co-Authored-By: Claude <noreply@anthropic.com>
- coder.yaml: detailed 5-phase workflow, tool cheat sheet, explicit DON'Ts
- str_replace_file.py: descriptive tool hint (recommended over execute_shell+sed)
- execute_shell.py: explicit warning not to use for file reading/editing

Co-Authored-By: Claude <noreply@anthropic.com>
Extracted from Claude Code's prompts.ts (914-line prompt engineering):
- CRITICAL tool usage rules: dedicated tools over execute_shell, with table
- Code modification principles: no extra features, no speculative abstractions,
  no error handling for impossible scenarios, don't edit unread code
- Comment policy: default to no comments, explain WHY not WHAT
- Error handling: diagnose before switching, don't blindly retry, no --force
- Reporting: honest outcomes, verify before claiming complete
- Task management: break down with set_todo, mark done immediately

Co-Authored-By: Claude <noreply@anthropic.com>
Add DirectAgent that skips the supervisor/worker/IPC pipeline entirely.
LLM calls tools directly in-process — same architecture as Claude Code.

- agent/direct_agent.py: in-process LLM loop with direct tool calls
- runner.py: _run_task_direct mode (no subprocess, no IPC serialization)
- cli.py: --mode direct option

Co-Authored-By: Claude <noreply@anthropic.com>
- build_tools_payload is in agent.llm.parser, not agent.tools
- ToolContext takes 'workspace' kwarg, not 'workspace_path'

Co-Authored-By: Claude <noreply@anthropic.com>
In direct mode, there is no user to confirm dangerous shell commands.
execute_shell calls (pytest, git diff, cd) were all being blocked by
the safety classifier ('requires user confirmation'). Use execute_forced
to auto-approve, matching supervisor's confirm_callback=lambda: True.

Also added detailed tool call logging for debugging.

Co-Authored-By: Claude <noreply@anthropic.com>
v4-pro at temperature=0.7 tends to get distracted by test failures
and wanders into environment debugging loops. Temperature=0 makes it
more focused on the source code fix.

Co-Authored-By: Claude <noreply@anthropic.com>
…use analysis

Documents every experiment: git mirror, tool reduction, prompt alignment,
supervisor removal, config stripping attempt, temperature tuning.
Explains why v4-pro converges at 20% regardless of architecture:
model attention drift in multi-turn conversations, triggered by
test failure output in tool results.

Co-Authored-By: Claude <noreply@anthropic.com>
@wq19901103wq
wq19901103wq merged commit 7810498 into main Jun 28, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant