Skip to content

fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome - #7505

Merged
WillemJiang merged 2 commits into
bytedance:mainfrom
llb555:fix/artifact-lark-temp-cleanup
Oct 25, 2026
Merged

WillemJiang merged 2 commits into
bytedance:mainfrom
llb555:fix/artifact-lark-temp-cleanup

Conversation

@llb555

@llb555 llb555 commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Two more writers in the same family: each publishes through a same-directory temporary and discards it on exit, and a failing discard replaced the outcome the caller needed.

app/gateway/routers/artifacts.py::_replace_artifact_atomically — the finally caught only FileNotFoundError around temp_path.unlink(), so any other OSError (a scanner or peer holding the handle on Windows) turned a failed or successful artifact edit into a cleanup error. It now logs the leftover temp and keeps the edit's actual result; the descriptor close is guarded the same way. The success path is unchanged.

integrations/lark_cli.py::_write_json_file_atomic — finally: temp_path.unlink(missing_ok=True) was unguarded, the same shape this file already fixed for publish_bytes_atomically in #7200. It now goes through the imported cleanup_failed_atomic_write.

tests/test_atomic_cleanup_writers.py pins both, plus the Lark success path:

  • a failing os.replace plus a failing unlink keeps replace failed and leaves the previous artifact bytes intact;
  • the same on the Lark flow-state writer;
  • the success path still writes the payload and leaves no temporary behind.

The two failure tests fail on the base revision with PermissionError: temporary unlink failed in place of replace failed, and pass here.

Verification: pytest tests/test_atomic_cleanup_writers.py tests/test_lark_cli_atomic_publish.py tests/test_artifacts_router.py → 116 passed, 1 skipped; ruff check and ruff format --check clean on all three files.

@github-actions github-actions Bot added area:backend Gateway / runtime / core backend under backend/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/S PR changes 20-100 lines labels Oct 25, 2026
@llb555

llb555 commented Oct 25, 2026

Copy link
Copy Markdown
Contributor Author

CI shard 4 failed on test_bench_sandbox_prewarm.py::test_prewarm_reused_survives_a_coarse_clock_tie[asyncio], which this PR does not touch — this is a pre-existing flake, not a regression here.

Evidence: on a tree with only this PR's three files changed, that test fails 2 of 3 local runs with

AssertionError: precondition: the acquire and the tool request must share one quantized tick
tests/test_bench_sandbox_prewarm.py:180

The other 18 tests in the file pass every time, and the same file ran green in the shard that contains it on 2d834f3's earlier attempt. Everything this PR actually changes is covered by tests/test_atomic_cleanup_writers.py, tests/test_lark_cli_atomic_publish.py and tests/test_artifacts_router.py — 116 passed, 1 skipped locally.

Happy to rebase or re-push to get a fresh CI run if that helps; I did not want to push an empty commit just to retrigger it.

@willem-bd willem-bd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 2d834f3. The failure-direction goal is right and the tests genuinely pin it (I re-ran both failure tests against the base revision: they fail there with PermissionError: temporary unlink failed replacing replace failed, and pass here; the Lark side's helper adoption matches the #7279 convention; 117 passed for the three suites on this tree, ruff check/format clean). One real regression on the artifact side: the broadened except OSError also catches the success path's expected FileNotFoundError, so every successful artifact edit now emits a warning traceback — the success path is not unchanged as the description states. Details inline, plus a symmetric success-path test gap.

temp_path.unlink()
except FileNotFoundError:
pass
except OSError:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This broadened catch also swallows the success path's expected FileNotFoundError. After a successful os.replace(temp_path, actual_path) the temp no longer exists, so temp_path.unlink() above raises FileNotFoundError on every successful edit — a subclass of OSError — and now produces logger.warning(..., exc_info=True) with a full traceback each time. Verified on this head: a clean, successful _replace_artifact_atomically emits exactly one WARNING ... Failed to remove the artifact edit temp file record. So the success path is not unchanged (contrary to the PR description), the message is misleading when the file was actually published, and genuine leftover-temp failures drown in per-edit noise.

The family helper already encodes the right semantics and can replace both guards here (the close block above and this unlink) in one call:

finally:
    cleanup_failed_atomic_write(temp_path, fd=temp_fd if temp_fd >= 0 else None)

Its unlink(missing_ok=True) keeps the published-temp case silent (verified on this tree: success → no log records; unremovable temp → warning, no raise), while still suppressing a genuine removal failure so the edit's outcome survives in both directions. It also changes the close-failure log from DEBUG to WARNING, which I'd consider an improvement, but flagging since it's a semantic shift. Minimum fix if you keep the hand-roll: temp_path.unlink(missing_ok=True) under the broad handler.

assert not target.exists()


def test_lark_flow_state_write_still_succeeds(monkeypatch, tmp_path):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Lark success path is pinned here, but the artifact writer has no symmetric success test: nothing asserts _replace_artifact_atomically writes the payload, leaves no .artifact-edit-* temp behind, and stays log-clean. A caplog-asserted artifact success test would have caught the regression at artifacts.py (the warning now emitted on every successful edit) and would keep both writers' success paths pinned equally.

@llb555

llb555 commented Oct 25, 2026

Copy link
Copy Markdown
Contributor Author

You are right on both counts, and the regression was worse than "semantically different": with the broadened handler every successful edit logged WARNING ... Failed to remove the artifact edit temp file plus a FileNotFoundError traceback, because os.replace has already moved the temp name away. Pushed f5b685b.

Artifact writer. Adopted the family helper rather than the missing_ok=True minimum, for exactly the reason you gave:

    finally:
        # The helper keeps the published case silent (the temp name is already
        # gone), warns when a leftover temp cannot be removed, and never lets a
        # cleanup error replace the edit's outcome.
        cleanup_failed_atomic_write(temp_path, temp_fd if temp_fd >= 0 else None)

That also collapses the hand-rolled close into the single call, so the descriptor and the name cannot drift apart. I am fine with the DEBUG→WARNING shift for a failed close: a descriptor we could not close is a real leak, and it is the same severity the helper already uses elsewhere.

The test gap. Added the symmetric success case and made it assert silence, not just bytes:

    with caplog.at_level("WARNING", logger="deerflow.utils.atomic_write"):
        artifacts._replace_artifact_atomically(target, b"replacement", file_stat)

    assert target.read_bytes() == b"replacement"
    assert [p.name for p in tmp_path.iterdir() if p.name.startswith(artifacts._ARTIFACT_EDIT_TEMP_PREFIX)] == []
    assert caplog.text == ""

Verified it would have caught this: restoring the previous hand-rolled finally makes that test fail with the exact FileNotFoundError: ... '\.artifact-edit-...' traceback on the success path.

Re-verified after both changes: pytest tests/test_atomic_cleanup_writers.py tests/test_lark_cli_atomic_publish.py tests/test_artifacts_router.py → 117 passed, 1 skipped; ruff check and ruff format --check clean on all three files.

@github-actions github-actions Bot added size/M PR changes 100-300 lines and removed size/S PR changes 20-100 lines labels Oct 25, 2026
@WillemJiang WillemJiang added this to the 2.2.0 milestone Oct 25, 2026
@WillemJiang
WillemJiang merged commit 885062e into bytedance:main Oct 25, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:backend Gateway / runtime / core backend under backend/ needs-validation Touches front/back contract surface; needs real-path validation risk:high High risk: backend API, agents, sandbox, auth, deps, CI size/M PR changes 100-300 lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants