Skip to content

add mtime logic to file difference checking - #14

Open
arturmakoev-positronic wants to merge 4 commits into
Positronic-Robotics:mainfrom
arturmakoev-positronic:timestamp-difference
Open

add mtime logic to file difference checking#14
arturmakoev-positronic wants to merge 4 commits into
Positronic-Robotics:mainfrom
arturmakoev-positronic:timestamp-difference

Conversation

@arturmakoev-positronic

@arturmakoev-positronic arturmakoev-positronic commented Sep 9, 2026

Copy link
Copy Markdown

Summary

Change detection used to compare only presence and size. A file rewritten in place with the same byte count (an overwritten checkpoint, a fixed-shape array, a same-length text edit) was silently skipped by upload, download, sync, and the background interval loop.

This PR adds modification time to the diff and ships it as v0.3.2.

What changed

  • FileInfo.mtime: new optional field (POSIX timestamp). _scan_local fills it from st_mtime, and _scan_s3 fills it from the object's LastModified. It is None for directories or when the backend doesn't report a time.

  • _compute_sync_diff: a file is now copied when it is

    1. missing on the target, or
    2. a different size, or
    3. newer on the source than on the target by more than _MTIME_TOLERANCE_SECONDS (2 s).

    If either side has no mtime, that entry falls back to size-only comparison, as before.

  • Downloads keep the S3 timestamp (like rsync -t): after download_file, the local copy's mtime is set to the object's LastModified via os.utime. Without this, a fresh download would carry "now" as its mtime, and the next sync() would re-upload the whole tree it had just pulled. If stamping fails, a warning is logged and the download still succeeds.

  • README: the change-detection bullet now says "size/presence/timestamp".

Because every path goes through _compute_sync_diff, the behavior applies to download, upload, sync, interval uploads, plan_download / plan_upload, and the CLI --dry-run.

Why a 2 s tolerance

S3 LastModified has whole-second resolution, while local filesystems keep sub-second mtimes. Right after an upload, the local file can look a fraction of a second "newer" than its S3 copy. The margin stops a freshly synced tree from re-transferring on every tick. The comparison is strict: a source ahead by exactly the tolerance does not count as newer.

Release (v0.3.2)

  • pyproject.toml and uv.lock bumped to 0.3.2. The lock was also stale at 0.3.0.
  • CHANGELOG.md: the [Unreleased] section became [0.3.2] - 2026-09-10. The documented tolerance was corrected from 1 s to 2 s to match the constant.

⚠️ Upgrade note

Files downloaded by pos3 ≤ 0.3.1 have their download time as mtime, which is newer than their S3 LastModified. The first upload() / sync() of such a tree after upgrading will re-upload it once. After that, syncs are incremental again. This is also in the changelog.

Tests

TestMtimeComparison in tests/test_s3.py covers:

  • the diff rule: newer copies, older skips, within/at/just-over tolerance, a missing mtime falls back to size, and a size difference wins regardless of mtime;
  • mtime reporting from both _scan_local and _scan_s3;
  • end-to-end same-size upload and download, both when newer and when not newer, including checking that downloads get the S3 timestamp stamped;
  • sync() not re-uploading what it just downloaded;
  • plan_upload reporting a same-size newer file.

Tolerance-sensitive tests are written in terms of _MTIME_TOLERANCE_SECONDS, so re-tuning the constant doesn't break them.

After merge

Publish a GitHub release tagged v0.3.2 from main. release.yml builds and pushes to PyPI, and it checks that the tag matches project.version.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 70ba11a506

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread pos3/__init__.py
@arturmakoev-positronic

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: b3d07f4400

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@vertix vertix left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good, but there's one question about reliability of the current approach.
Some other things

  • Add a normal PR description
  • Look into the recent PRs and do the proper minor release in this PR. I.e. update releasenotes, pyproject.toml etc. This should be a new release

Comment thread pos3/__init__.py Outdated
def _compute_sync_diff(source: Iterator[FileInfo], target: Iterator[FileInfo]) -> tuple[list[FileInfo], list[FileInfo]]:
"""Return ``(to_copy, to_delete)`` to make ``target`` mirror ``source``.

A file is copied when it is missing on the target, differs in size, or

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make the docstring oneliner, too verbose now
the last sentence is not needed, just drop it, i mean

The
mtime check is what catches an in-place edit that
leaves the byte count unchanged (a rewritten checkpoint, a fixed-shape
array, a same-length text edit); size alone cannot see it.

Comment thread pos3/__init__.py
# file can therefore look "newer" than its S3 twin by a fraction of a second.
# Treat a source as newer only when it is ahead by more than this margin, so
# a freshly synced tree does not re-transfer on every tick.
_MTIME_TOLERANCE_SECONDS = 2.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we know that 2 seconds is enough? what if the connection lags, or smth like this. Is there a more "hard" method? For example, can we use an extra meta information to store the local data, and fallback to the "default" LastModified only when there's no our meta?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the download from s3 direction is OK, since it assigns the Last-Modified info from s3 to a local file.
However, the upload direction is broken and prone to many bugs since we don't control Last-Modified metadata on the s3 side.
Claude's possible solution is:

The other way to match them is to change the local file instead. After an upload, read back LastModified (one extra request per uploaded file) and set the local file's mtime to it. It works, but I wouldn't do it

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this might be concerning:

Upgrade note
Files downloaded by pos3 ≤ 0.3.1 have their download time as mtime, which is newer than their S3 LastModified. The first upload() / sync() of such a tree after upgrading will re-upload it once. After that, syncs are incremental again. This is also in the changelog.

Bump version to 0.3.2 in pyproject.toml and uv.lock, and move the mtime
change-detection entries from [Unreleased] to [0.3.2].

Also correct the documented tolerance (2 s, matching
_MTIME_TOLERANCE_SECONDS, not 1 s) and add an upgrade note: trees
downloaded by <= 0.3.1 carry download-time mtimes, so their first
upload/sync after upgrading re-uploads once.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants