Repository navigation
fix(skills): rollback reads its history file off the event loop - #5729
Conversation
rollback_custom_skill constructed the user-scoped storage, probed whether the skill exists and parsed custom/.history/<name>.jsonl inline, while the sibling get_custom_skill_history two lines above already offloads exactly those three steps and documents them as "blocking filesystem IO that must stay off the event loop". The history file grows one entry per edit carrying full previous and new content, so a rollback request parsed the whole edit history on the Gateway loop; the strict Blockbuster gate catches the construction as a blocking os.getcwd() through SkillsConfig.get_skills_path() -> project_root(). Offload the three steps through the same worker-thread closure and anchor both pre-scan branches under tests/blocking_io/.
willem-bd
left a comment
There was a problem hiding this comment.
Reviewed against head edb34c2. The production change is correct and minimal: I verified that LocalSkillStorage.read_history returns [] (never None) when the history file is missing, so the history is None 404 sentinel cannot collide with the existing 400 has no history path, and status-code precedence (404 missing → 400 empty → 400 out-of-range) is unchanged. SkillStorage is imported at module top so the nested-def annotation resolves, and asyncio.to_thread copies the current context so get_effective_user_id() inside the worker behaves as on the loop — matching the established get_custom_skill_history pattern. Two suggestions and a nit on the new anchor file below.
|
|
||
| ``rollback_custom_skill`` builds the user-scoped storage, probes whether the | ||
| skill exists and parses ``custom/.history/<name>.jsonl`` inline, while every | ||
| other filesystem phase in the same handler already runs through |
There was a problem hiding this comment.
This docstring overclaims: "every other filesystem phase in the same handler already runs through asyncio.to_thread" is not true at this head — rollback_custom_skill still executes current_content = skill_file.read_text(encoding="utf-8") if skill_file.exists() else None inline on the event loop (backend/app/gateway/routers/skills.py:644), which this PR's own collision audit deliberately leaves to #5645. Worth rewording (e.g. "every other pre-scan filesystem phase ...") so a future reader of this anchor doesn't conclude the handler is already fully off-loop and mis-attribute line 644's blocking read to a regression.
| async def _allow(_request, **_kwargs) -> None: | ||
| return None | ||
|
|
||
| monkeypatch.setattr(skills_router, "require_admin_user", _allow) |
There was a problem hiding this comment.
Consider the sibling pattern instead of stubbing a security predicate: test_skills_router.py in this same directory builds a real starlette Request with an admin in request.state.user (_admin_request()) and lets the real require_admin_user → is_admin_user path run — no DB or extra fixtures needed. Reusing that here would drop the require_admin_user monkeypatch (so the gate also keeps covering anything authz itself might do) and keep the anchor closer to production. Optional, since the PR documents the stub choice.
| (history_dir / f"{_SKILL_NAME}.jsonl").write_text("".join(json.dumps(r) + "\n" for r in records), encoding="utf-8") | ||
|
|
||
|
|
||
| async def test_rollback_missing_skill_does_not_block_event_loop(monkeypatch: pytest.MonkeyPatch) -> None: |
There was a problem hiding this comment.
Nit: the monkeypatch parameter is unused — path isolation comes from the autouse _isolate_paths fixture. The parameter can be dropped.
…loop (#5751) * fix(gateway): keep custom-skill rollback filesystem IO off the event loop Closes #5747. Three blocking sites remained on the Gateway event loop in the rollback route and its shared response reader, all offloaded through asyncio.to_thread following the #5729 pattern: - storage.validate_skill_markdown_content() creates a temp dir and writes SKILL.md into it (site 1) - the post-scan read of the file being replaced (site 2, named by #5729's test docstring) - _read_custom_skill_response(): load_skills() walks every skill directory and read_custom_skill() opens SKILL.md — O(installed skills) loop time paid per accepted rollback and per GET /skills/custom/{name} (site 3) A blocking_io regression drives the full accepted-rollback path (past the security scan) under the strict gate; the two pre-existing cases only covered the pre-scan branches. * fix(gateway): keep the skill element type in the response-parts closure Review nit on #5751: tuple[object, ...] erased the element type for the _skill_to_response call; Skill is already imported in this module.
Why
Self-found; no issue.
rollback_custom_skillconstructs the user-scoped storage, probes whether theskill exists and parses
custom/.history/<name>.jsonlinline, whileget_custom_skill_historytwo handlers above already offloads those exact threesteps and documents why (
backend/app/gateway/routers/skills.py:597-606):So this is not a new policy proposal — it is one handler breaking a rule the
adjacent handler states verbatim, and the rest of the same handler obeys it
(
append_historyandwrite_custom_skillboth go throughasyncio.to_thread).The cost is not one
stat():read_historyreads the whole file andjson.loadsevery line (skills/storage/local_skill_storage.py:260-270), andeach history entry carries the full
prev_contentandnew_contentof theskill, so the file grows with every edit. A rollback request therefore parses
that skill's entire edit history on the Gateway event loop, stalling every other
run and stream in the worker for its duration.
The storage construction is blocking too, which the strict gate confirms
directly rather than by reading — see Bug fix verification.
Why the gate never caught it:
scripts/detect_blocking_io_static.pyreports thisfile's existence probe and the post-scan
read_text, but cannot seeread_historyat all, because its async reachability is same-file only(
ASYNC_REACHABLE_SAME_FILE) andread_historylives in the storage module. TheSOP in
.agent/skills/blocking-io-guardcalls out exactly this blind spot, andthat cross-file rule is how the call was found.
What changed
POST /api/skills/custom/{skill_name}/rollbacknow performs storageconstruction, the existence probes and the history-file read on a worker
thread, in one closure mirroring
get_custom_skill_history.tests/blocking_io/, somake test-blocking-ioguards the route from now on.Status codes and their precedence are unchanged (missing skill → 404, empty
history → 400, out-of-range index → 400), the response shape is unchanged, and no
user-visible default changes, so no README/AGENTS.md text is affected.
Deliberately not included: the post-scan
skill_file.read_text(...)in thesame handler, which #5645 already offloads on its own line — see Collision audit.
Surface area
frontend/backend/applanggraph.json, or prompt changedocker/or sandboxed executionskills/(the custom-skill rollback route)backend/pyproject.tomlorfrontend/package.json(say what it buys us)Backend API is left unchecked because the request/response shape and every status
code are identical; only the thread the blocking work runs on changes.
Screenshots / Recording
Not applicable — no UI surface.
Bug fix verification
Test path that reproduces the bug:
backend/tests/blocking_io/test_skills_custom_router.py(
test_rollback_missing_skill_does_not_block_event_loop,test_rollback_history_read_does_not_block_event_loop)Red on
main: yes, on9ae1e58with only the test file added androuters/skills.pyuntouched:Green on this branch: yes —
2 passed.The anchor drives the real handler and real local filesystem; the only stubbed
collaborator is
require_admin_user, because this is a direct handler call withno HTTP layer to run auth through. Both branches return before the awaited
security scan, so no scanner or model stub is needed.
Test-side setup IO (
_install_skill,_write_history) is itself dispatched viaasyncio.to_thread, matchingtest_agents_router, so only production work runson the loop.
Collision audit
Method: all 392 open PRs enumerated with
pulls?state=open&per_page=100&page=1..4, then each PR's file list and the@@hunk headers of the candidate files compared against the lines this change
touches.
backend/app/gateway/routers/skills.py: feat(skills): add guarded evolution host APIs #5645, feat(skills): install local archives from settings #4739,Add App Key control plane #4646, fix(skills): preserve extension config on toggle #3777, feat(skills): per-user custom skill isolation #3388, chroe(PMC): improvements of MCP and Skills management #1091. Five of them have no hunk below line 353, and Add App Key control plane #4646
only edits the body of
_get_user_skill_storage(not its signature or callsites), so none of them can conflict here.
rollback_custom_skilland already offloads the post-scancurrent_content = skill_file.read_text(...)viaawait asyncio.to_thread(read_optional_text, storage, skill_file). That is whythis PR stops at the three pre-scan steps: the remaining blocking read in this
handler is theirs, and this change does not touch that line.
backend/tests/blocking_io/test_skills_custom_router.py.[bug] describe_skill prints "Allowed tools: (all)" for a skill that declared an empty allowed-tools list #5590/[bug] Skill export renders empty allowed-tools and required-secrets as blank #5658/[bug] Assembly fingerprint collapses undeclared and empty Skill allowed-tools #5665 are about
describe_skillrendering, export requirements andassembly fingerprints, and each already has its own PR).
Validation
Branch
fix/rollback-history-offloop@edb34c2, rebased onto upstreammain9ae1e58. Every command below was run on that combination.9ae1e58):2 failed,BlockingError: Blocking call to os.getcwdwith the frame chain quoted above.2 passed.make test-blocking-ioequivalent(
pytest tests/blocking_io -q): 151 passed.(
pytest tests/test_skills_custom_router.py tests/test_skill_storage_lifecycle.py -q):36 passed, including the existing 404 / 400-no-history /
scanner-blocked-400 and out-of-range cases.
python3 scripts/detect_blocking_io_static.py: total findings63 → 62;
routers/skills.pygoes from 3 findings (lines 622, 635, 635) to 2(the
skill_file.read_text/.existspair at 644, which is feat(skills): add guarded evolution host APIs #5645's line,shifted by this diff).
read_historynever appeared in either listing, per thesame-file limitation above.
ruff check .andruff format --check .over the whole backend: clean(1574 files already formatted).
python3 scripts/check_agent_guidance.py: 31 AGENTS.md, 0 errors, 0 warnings.git diff --check: clean.make testequivalent,pytest -m "not live" --ignore=tests/blocking_io tests/):18161 passed, 205 skipped, 3 deselected, 10 failed in 8:22. Those 10 are
pre-existing in this environment, not caused by this change: the same 10 node
ids fail on the untouched base
9ae1e58(4 intests/test_extension_manager.py, which install real git/requirement sourcesthrough
uv; 3 intests/test_lark_broker.py, 1 intests/test_openviking_mcp_integration.pyand 2 intests/test_parallel_mcp_user_agent.py, which need a live local broker /lark-clishim —test_health_endpointfails withurllib.error). This branchadds no failure beyond those.
AI assistance
Tool(s) used: Qoder CLI (AI coding agent)
How you used it: the agent ran the repository's blocking-IO scanner and the
blocking-io-guardSOP, traced the same-file reachability gap that hidesread_history, drafted the anchor against the strict Blockbuster gate, andapplied the offload. It also produced the collision audit by paging all 392 open
PRs and diffing hunk headers. A human read every line of the resulting diff,
verified the 404/400 precedence claim against the existing route tests, and
reviewed and re-ran the red/green, scan, lint and gate numbers above (all of
which are actual command output on
9ae1e58, not summaries).