Repository navigation
fix(gateway,integrations): keep temp cleanup from replacing the caller's outcome - #7505
Conversation
|
CI shard 4 failed on Evidence: on a tree with only this PR's three files changed, that test fails 2 of 3 local runs with The other 18 tests in the file pass every time, and the same file ran green in the shard that contains it on Happy to rebase or re-push to get a fresh CI run if that helps; I did not want to push an empty commit just to retrigger it. |
willem-bd
left a comment
There was a problem hiding this comment.
Reviewed at 2d834f3. The failure-direction goal is right and the tests genuinely pin it (I re-ran both failure tests against the base revision: they fail there with PermissionError: temporary unlink failed replacing replace failed, and pass here; the Lark side's helper adoption matches the #7279 convention; 117 passed for the three suites on this tree, ruff check/format clean). One real regression on the artifact side: the broadened except OSError also catches the success path's expected FileNotFoundError, so every successful artifact edit now emits a warning traceback — the success path is not unchanged as the description states. Details inline, plus a symmetric success-path test gap.
| temp_path.unlink() | ||
| except FileNotFoundError: | ||
| pass | ||
| except OSError: |
There was a problem hiding this comment.
This broadened catch also swallows the success path's expected FileNotFoundError. After a successful os.replace(temp_path, actual_path) the temp no longer exists, so temp_path.unlink() above raises FileNotFoundError on every successful edit — a subclass of OSError — and now produces logger.warning(..., exc_info=True) with a full traceback each time. Verified on this head: a clean, successful _replace_artifact_atomically emits exactly one WARNING ... Failed to remove the artifact edit temp file record. So the success path is not unchanged (contrary to the PR description), the message is misleading when the file was actually published, and genuine leftover-temp failures drown in per-edit noise.
The family helper already encodes the right semantics and can replace both guards here (the close block above and this unlink) in one call:
finally:
cleanup_failed_atomic_write(temp_path, fd=temp_fd if temp_fd >= 0 else None)Its unlink(missing_ok=True) keeps the published-temp case silent (verified on this tree: success → no log records; unremovable temp → warning, no raise), while still suppressing a genuine removal failure so the edit's outcome survives in both directions. It also changes the close-failure log from DEBUG to WARNING, which I'd consider an improvement, but flagging since it's a semantic shift. Minimum fix if you keep the hand-roll: temp_path.unlink(missing_ok=True) under the broad handler.
| assert not target.exists() | ||
|
|
||
|
|
||
| def test_lark_flow_state_write_still_succeeds(monkeypatch, tmp_path): |
There was a problem hiding this comment.
The Lark success path is pinned here, but the artifact writer has no symmetric success test: nothing asserts _replace_artifact_atomically writes the payload, leaves no .artifact-edit-* temp behind, and stays log-clean. A caplog-asserted artifact success test would have caught the regression at artifacts.py (the warning now emitted on every successful edit) and would keep both writers' success paths pinned equally.
|
You are right on both counts, and the regression was worse than "semantically different": with the broadened handler every successful edit logged Artifact writer. Adopted the family helper rather than the finally:
# The helper keeps the published case silent (the temp name is already
# gone), warns when a leftover temp cannot be removed, and never lets a
# cleanup error replace the edit's outcome.
cleanup_failed_atomic_write(temp_path, temp_fd if temp_fd >= 0 else None)That also collapses the hand-rolled close into the single call, so the descriptor and the name cannot drift apart. I am fine with the DEBUG→WARNING shift for a failed close: a descriptor we could not close is a real leak, and it is the same severity the helper already uses elsewhere. The test gap. Added the symmetric success case and made it assert silence, not just bytes: with caplog.at_level("WARNING", logger="deerflow.utils.atomic_write"):
artifacts._replace_artifact_atomically(target, b"replacement", file_stat)
assert target.read_bytes() == b"replacement"
assert [p.name for p in tmp_path.iterdir() if p.name.startswith(artifacts._ARTIFACT_EDIT_TEMP_PREFIX)] == []
assert caplog.text == ""Verified it would have caught this: restoring the previous hand-rolled Re-verified after both changes: |
Two more writers in the same family: each publishes through a same-directory temporary and discards it on exit, and a failing discard replaced the outcome the caller needed.
app/gateway/routers/artifacts.py::_replace_artifact_atomically— thefinallycaught onlyFileNotFoundErroraroundtemp_path.unlink(), so any otherOSError(a scanner or peer holding the handle on Windows) turned a failed or successful artifact edit into a cleanup error. It now logs the leftover temp and keeps the edit's actual result; the descriptor close is guarded the same way. The success path is unchanged.integrations/lark_cli.py::_write_json_file_atomic—finally: temp_path.unlink(missing_ok=True)was unguarded, the same shape this file already fixed forpublish_bytes_atomicallyin #7200. It now goes through the importedcleanup_failed_atomic_write.tests/test_atomic_cleanup_writers.pypins both, plus the Lark success path:os.replaceplus a failing unlink keepsreplace failedand leaves the previous artifact bytes intact;The two failure tests fail on the base revision with
PermissionError: temporary unlink failedin place ofreplace failed, and pass here.Verification:
pytest tests/test_atomic_cleanup_writers.py tests/test_lark_cli_atomic_publish.py tests/test_artifacts_router.py→ 116 passed, 1 skipped;ruff checkandruff format --checkclean on all three files.