Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,60 @@
# Changelog

## [0.3.1] - 2026-05-21

### Added
- `pos3` console-script entry point with `ls`, `download`, `upload` subcommands
([#10](https://github.com/Positronic-Robotics/pos3/issues/10)).
- `pos3 ls <prefix> [-r] [--profile NAME]` lists objects, one full `s3://` URL
per line on stdout.
- `pos3 download <url> [--local PATH] [--delete] [--exclude PATTERN]... [--profile NAME]`
prints only the resulting local path to stdout; progress and logs go to
stderr, so `data_dir=$(pos3 download s3://bucket/dataset/)` is safe.
- `pos3 upload <url> [--local PATH] [--delete] [--exclude PATTERN]... [--profile NAME]`
is one-shot (no background loop or interval). Source defaults to the cache
path `pos3 download` would have produced; errors if the source doesn't
exist.
- `--delete` defaults OFF for both `download` and `upload` (the Python API
defaults to `True`; the CLI is more conservative for interactive use).
- `--profile` is supported alongside the URL form `s3://<profile>@bucket/...`;
the URL form wins on conflict, matching the Python precedence.
- `-n` / `--dry-run` on `download` and `upload` prints the planned per-file
actions to stdout (in `aws s3 sync --dryrun` style) and performs no
transfers, no deletes, and no directory creation.
- `download` and `upload` require an `s3://` URL. The Python API's
local-path passthrough still works in code; the CLI rejects non-S3
inputs with a clear error so a typo can't silently succeed. `ls` is
unchanged and still accepts both forms.
- `pos3.TransferPlan` dataclass plus module-level `pos3.plan_download(remote, ...)`
and `pos3.plan_upload(remote, ...)` wrappers: compute the set of
`(source, destination)` copies and target deletes a real call would
perform, without performing any of them. Same calling pattern as
`pos3.download` / `pos3.upload` — use them inside a `with pos3.mirror():`
block. The CLI's `-n` / `--dry-run` is implemented on top of these.
- `TransferError` and `TransferPlan` are now in `pos3.__all__`.

### Changed
- Per-object transfer failures now raise `pos3.TransferError` instead of
being logged and swallowed. Previously, if any worker in a download or
upload batch failed, the error was sent to the logger and the call
returned normally — `data_dir = pos3.download(...)` could return a path
to a partial cache, and `pos3 download` exited 0 after a failed S3 GET.
Both now propagate. The new exception exposes `.operation` and
`.failures` (list of the underlying per-worker exceptions). The CLI
catches it and exits 1 with the failure on stderr. **Background
interval syncs** (`upload(..., interval=N)`) are best-effort: a
`TransferError` from one tick is logged and the daemon continues so
the next interval can retry. Only the final sync on context exit and
any one-shot call propagate. **Cleanup on error** — when the
`mirror()` body is unwinding with an exception, a `TransferError` from
the `sync_on_error=True` cleanup sync is logged but swallowed, so the
original application exception remains the visible cause.
- `pos3.mirror()` no longer creates the cache root directory eagerly on
context entry. The leaf directory is still created on demand when a
file is actually downloaded, so the visible behavior of `download()` is
unchanged. Dry-run and `plan_*` paths are now genuinely side-effect
free on the local filesystem.

## [0.3.0] - 2026-05-19

### Added
Expand Down
56 changes: 56 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,62 @@ Lists files/objects in a directory or S3 prefix.

**Returns**: List of full S3 URLs or local paths.

## CLI

`pos3` ships a small command-line interface for the most common one-shot
operations. After `uv pip install pos3` (or `pip install pos3`), `pos3` is on
your `$PATH`:

```bash
# List objects (one full s3:// URL per line on stdout)
pos3 ls s3://bucket/dataset/
pos3 ls -r s3://bucket/dataset/

# Download an S3 prefix or object into the cache (or a custom --local path).
# Only the resulting local path is written to stdout — progress and logs go
# to stderr — so it's safe to capture in a shell variable:
data_dir=$(pos3 download s3://bucket/dataset/)

# One-shot upload. Source defaults to the same cache path `pos3 download`
# would have produced; --local overrides. Errors if the source doesn't exist.
pos3 upload s3://bucket/results/ --local ./out/

# Preview what download/upload would do, without touching anything.
pos3 download -n s3://bucket/dataset/ --local ./data/ --delete
pos3 upload -n s3://bucket/results/ --local ./out/
```

All three subcommands accept `--profile NAME`. The URL form
`s3://<profile>@bucket/...` takes precedence over `--profile` on conflict
(matching the Python API).

`--delete` defaults to **OFF** for both `download` and `upload`, even though
the Python API defaults to `True`. CLI defaults are conservative for
interactive shell use; pass `--delete` explicitly to mirror file removals.

`-n` / `--dry-run` is accepted on `download` and `upload` (not `ls`). It
prints per-file plan lines to stdout in `aws s3 sync --dryrun` style and
performs no transfers, no deletes, and no local directory creation.

The same plan is available from Python via `pos3.plan_download` and
`pos3.plan_upload`, each returning a `pos3.TransferPlan` with
`to_copy: list[tuple[str, str]]` and `to_delete: list[str]`:

```python
with pos3.mirror():
plan = pos3.plan_download("s3://bucket/dataset/", local="./data/")
for src, dst in plan.to_copy:
print(f"would download {src} → {dst}")
```

`download` and `upload` require an `s3://` URL; non-S3 inputs are rejected
with a non-zero exit. `ls` still accepts both `s3://` prefixes and local
paths, matching the Python API.

The CLI is one-shot only — no background sync, no `pos3 sync` subcommand.
Use the Python `pos3.mirror()` context manager when you need an interval-based
loop or bi-directional sync over a job's lifetime.

## Comparison with Libraries

Why use `pos3` instead of other Python libraries?
Expand Down
Loading
Loading