Skip to content
nijaruPublic

About

File synchronization in Rust with delta transfer and SSH; rewrite in progress.

Resources

Contributing

Stars

42 stars

Watchers

1 watching

Forks

Repository files navigation

sy

Warning

Under active development.

CI License: MIT

Fast file synchronization. Same mental model as rsync, built in Rust.

Development on main tracks the unfinished 0.5 rewrite. The published cargo install sy crate is still 0.4.x; build from source to track 0.5.

Install

cargo install sy

For the 0.5 development version, build from source. ACL preservation is behind the acl feature and needs platform ACL libraries (for example libacl on Linux):

cargo build --release --features acl

Static Linux/musl artifacts intentionally omit ACL support so they remain standalone remote-agent binaries.

Quick Start

# Basic sync
sy /source /destination

# Preview changes
sy /source /destination --dry-run

# Mirror (delete extra files)
sy /source /destination --delete

# Remote sync via SSH
sy /local user@host:/remote
sy user@host:/remote /local

Features

  • Parallel — bounded concurrent transfers, -j N to tune
  • Delta sync — only transfers changed blocks for large files
  • COW updates — descriptor-bound reflink patching on Linux Btrfs/XFS; macOS currently uses native whole-file copying
  • rsync-compatible flags — --delete, --exclude, --compress, -i, etc.
  • SSH sync — custom v3 protocol over SSH stdin/stdout (sy __serve)
  • Integrity — BLAKE3-verified transfers; race-checked, staged atomic commits

Usage

sy [OPTIONS] <SOURCE> <DESTINATION>

Common Flags

Flag Description
-n, --dry-run Preview changes without applying
-d, --delete Delete files not in source
-v, --verbose Increase verbosity (repeatable)
-q, --quiet Suppress output
--stats Show transfer statistics
--exclude <PATTERN> Exclude files matching pattern
--exclude-from <FILE> Read exclude patterns from file
--include <PATTERN> Include files matching pattern (use after --exclude)
--compress Compress transfers (auto-detected)
-j, --parallel <N> Parallel transfers (default: 10)

Use -t or -a to preserve source modification times in local, push, and pull transfers. Without them, sy does not apply source timestamps. Because the default comparison uses size and mtime, a later bare sync may copy those files again; use -t for timestamp-based incremental synchronization.

Sync Modes

# Mirror mode
sy /source /dest --delete

# Directories only (no recursion)
sy /source /dest --dirs

Remote Sync

# Push to remote
sy /local user@host:/remote

# Pull from remote
sy user@host:/remote /local

# With SSH timeout
sy /local user@host:/remote --timeout 30

Remote-to-local pulls support --delete with remote source-ignore protection and the same --max-delete safety threshold as local sync and push. --remove-source-files and --copy-links remain unsupported for pulls.

Filters

# Exclude patterns
sy /source /dest --exclude "*.log" --exclude ".git"

# Exclude from file
sy /source /dest --exclude-from .syignore

# Include specific patterns (after exclude)
sy /source /dest --exclude "*" --include "*.rs"

Backup & Safety

# Backup before overwrite
sy /source /dest --backup

# Custom backup directory
sy /source /dest --backup --backup-dir /backups

# Custom suffix
sy /source /dest --backup --suffix .bak

# Force delete when threshold exceeded
sy /source /dest --delete --force-delete

Verification

# Verify staged content against the source before commit
sy /source /dest --verify=after

# Audit file integrity without modifying anything
sy /source /dest --verify=only

# Show itemized changes
sy /source /dest --itemize-changes

Remote transfers are always BLAKE3-verified against the source before the destination commit; --verify adds staged verification on local copies.

--verify=only reports exact totals, but retains at most 32 examples per category under a shared 64 KiB detail budget. Error messages are capped at 2 KiB and marked when shortened. Human output notes omitted entries. With --json, the files_mismatched_count, files_only_in_source_count, files_only_in_dest_count, and errors_count fields are the totals; their corresponding arrays contain examples, not necessarily every finding.

JSON paths are strings when valid Unicode. Otherwise they use {"encoding":"unix_bytes","bytes":[...]} on Unix, or {"encoding":"windows_utf16","units":[...]} on Windows, preserving the native name without replacement characters. This encoding applies to event paths and verification examples; it does not imply Windows sync support.

Feature Status

Linux and macOS are the sync platforms exercised by CI. Windows is compile-checked only; synchronization is not supported there yet because native race-safe file identity and root-confined filesystem operations are not implemented.

Feature Status Notes
Local sync Stable Staged atomic commits, race-checked
SSH push Stable v3 protocol over ssh host sy __serve
SSH pull Stable Whole-file fetch with staged verification
Delta sync (push) Stable Rolling weak checksums + BLAKE3 strong block signatures
Filters (--exclude/--include) Stable rsync-style patterns, --filter, templates
Delete mode (--delete) Supported Local/push/pull, with source-ignore protection and --max-delete safety threshold
Compression (-z) Stable zstd, auto/always/never
Hard links (-H) Supported Whole-group topology and metadata for local, SSH push and SSH pull
Symlinks Stable --links=preserve/follow/skip
Backup mode (--backup) Stable Replacements and deletions
Atomic writes Stable Private staging, verified, atomic replace
Preservation (-X/-A/-F) Stable xattrs; ACLs with the acl feature; BSD flags on macOS. Unsupported combinations fail loudly rather than being silently skipped
--bwlimit Stable Paced at the byte stream
--checksum Stable BLAKE3 content comparison instead of mtime+size
--update / --existing Stable Comparison modes for selective sync
--ignore-times / --ignore-existing Stable Force transfer / skip existing
--verify Stable Staged verification (after/only)
Directory type transitions Limited Local/push file or symlink over an empty directory uses atomic exchange; nonempty replacements, directory over file/symlink, and pull directory transitions are refused in preflight
Bidirectional sync (bisync) Removed Not part of 0.5
S3/GCS endpoints Planned Not part of 0.5 (local and SSH sync engine focus)

Nonempty directory replacements are refused even with --delete: descendant identities are not yet carried into transaction cleanup. Existing deletion scope, protected-descendant and threshold checks still apply before refusal. Empty-directory replacement checks for children before exchange and uses only rmdir afterward. If a child arrives after the check, the replacement may be published, but cleanup fails and the old directory stays in private staging; its children are never recursively deleted.

Symlink replacements use private same-filesystem staging, apply requested mtime before publication, and validate the scanned destination and held parent. Creates do not overwrite a name that appeared concurrently. Updates still use separate identity checks and rename/exchange syscalls, not atomic compare-and-swap; concurrent namespace writers can race those checks. Rename also does not promise power-loss durability or whole-run rollback.

With --remove-source-files, transferred local or push sources are removed only after commit and required preservation succeed. Remote-to-local pulls refuse --remove-source-files. An unchanged regular file is eligible only with --checksum: sy re-reads both files and checks requested permissions, mtime, xattrs, ACLs and flags against the observed identities before removing its source. Local and SSH push -H groups also verify actual byte parity before any member's source removal, including quick-equal members. A changed destination or required preservation failure stops the operation and retains the source. Other quick-check skips and unchanged symlinks are retained. Source/destination namespace aliases are refused; distinct hardlink names are allowed. Remote files with indistinguishable inode and namespace identifiers are conservatively treated as aliases. These checks are not atomic with source unlink and do not provide isolation from concurrent namespace writers.

For staged file replacements, cancellation or a remote session ending closes publication admission. A commit not yet admitted aborts staging and retains the old destination. An admitted native commit may finish afterward; cancellation cannot interrupt it or undo publication. A lost acknowledgement leaves completion uncertain. This cutoff does not yet cover every directory, link, backup or in-place metadata operation.

Local and SSH -H complete one selected group at a time. The first selected path in catalogue order supplies baseline metadata; explicit preservation overrides it. If any member transfers or destination links need repair, source bytes become canonical for the group—even for quick-equal members. Already-linked groups needing only metadata retain destination bytes; a private staged replacement isolates changes from outside aliases. Fully matching linked groups stay unchanged. Excluded and policy-skipped names are not members to coalesce.

SSH peers must support the same exact protocol version, currently v3.12. Group declarations and final acknowledgements bind all selected names; a lost final acknowledgement does not authorize source removal.

Outside -H group replacement, needed metadata writes to shared regular files are refused: an alias may belong to the source, even an excluded file. Matching requested state needs no write. Observation checks are not snapshots or atomic compare-and-swap; earlier published members remain if a later member fails. Source removal waits for verified completion of every selected member.

Deletion backups apply to regular files only. Symlink targets are never copied. A backup is staged and identity-checked before publication; failure leaves the deletion candidate intact. Local deletion backups now copy then unlink rather than rename, which adds I/O. As with source removal, final checks and namespace mutation are not an atomic compare-and-swap.

Benchmarks

No validated performance comparison is available yet for the 0.5 engine. The CLI harness compares archive-style copies of caller-owned files. It checks exact destination trees outside timing and reports unchanged samples only when public transfer evidence proves zero file payload. Rates count logical file bytes, not wire traffic.

cargo build --release
python3 scripts/benchmark.py --quick --sy-binary target/release/sy
cargo bench  # Engine and codec microbenchmarks

The quick run checks the harness; it is not a representative performance study.

Configuration

sy reads ~/.config/sy/config.toml for defaults:

max_concurrent = 8
compress = "auto"
exclude = [".git", "node_modules", "*.pyc"]

Comparison to rsync

Feature sy rsync
Delta sync Yes (rolling + BLAKE3) Yes
COW reflink updates Linux Btrfs/XFS No
SSH sync Yes Yes
Wire protocol Custom rsync protocol
Incremental Yes Yes
Compression zstd Multiple codecs

sy is not a drop-in rsync replacement. Same mental model, different protocol. For rsync-to-rsync compatibility, use rsync.

Contributing

See CONTRIBUTING.md for development setup and workflow.

cargo build
cargo test
cargo clippy -- -D warnings
cargo fmt --check

Changelog

See CHANGELOG.md for release history.

License

MIT

About

File synchronization in Rust with delta transfer and SSH; rewrite in progress.

Resources

Contributing

Stars

42 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages