storage: S3/R2 objects join the Storage trait, over an injected HTTP transport

CHANGELOG.md

@@ -5,6 +5,25 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+The S3 release: object storage joins the `Storage` trait, and still nothing third-party enters the tree.
+
+### Added
+
+- Storage: **`S3Storage<T>` implements `Storage`** — AWS S3, **Cloudflare R2**, MinIO, Garage, Ceph RGW behind the same `put`/`get`/`stat`/`delete`/`list`/`get_reader` call sites as `FsStorage` and `DbStorage`. This closes the gap the presigner left open: the swap seam now spans fs ↔ db ↔ bucket, so an app outgrows a single disk (or a Postgres row's few-MB ceiling) by changing the type it constructs. `S3Store::storage(transport)` is the crossing point from credentials to trait, and `S3Store::r2(account, bucket, ak, sk)` is R2 preconfigured (region `auto`, path-style endpoint).
+- Storage: **an outbound-HTTP seam, `HttpTransport`** — one method, so a real S3 client exists without sutegi growing a TLS stack or a dependency. `SystemCurl` delegates the `https` handshake and certificate verification to the system `curl` (the crypto that must not be hand-rolled isn't); `PlainHttp` is pure `std` and **refuses `https`** rather than pretending, for stores on a trusted network path; your own client is `impl HttpTransport`.
+- Storage: **`Authorization`-header SigV4** alongside query presigning (`S3Store::sign_request`, public so the rest of the S3 API — multipart, `CopyObject`, tagging — is reachable through the same transport). The **payload hash is signed** (`x-amz-content-sha256`, never `UNSIGNED-PAYLOAD`), so a body altered in flight is refused by the store — integrity that holds even over `PlainHttp`. Verified against AWS's published known-answer vectors for header-signed `GET Object` and `PUT Object`, next to the presign vector already pinned.
+- Storage: **`ETag` verification on upload and download**, on by default — when the store reports a plain MD5 ETag (single-part, no SSE-C/KMS), the bytes are checked end-to-end; multipart/encrypted ETags are skipped, not faked. `verify_etag(false)` opts out.
+- Storage: `list` follows `ListObjectsV2` continuation tokens and is bounded by `max_list_keys` (default 100 000) — a ten-million-object bucket **errors instead of silently truncating**, and a store that keeps replaying one token cannot spin forever. Keys arrive `encoding-type=url` and are decoded; keys no backend can address (the empty `dir/` markers S3 GUIs create) are skipped.
+- Storage tests: 58 unit cases plus a 9-case wire suite (`tests/s3_roundtrip.rs`) driving the client against a tiny in-process S3 stub over a real socket — full lifecycle, 2400-key pagination across three round trips, unicode/space/plus keys through both path and XML encodings, a tampered download caught by ETag, a dead endpoint, and the same lifecycle again through a real `curl` subprocess (including a 2 MB upload, whose `Expect: 100-continue` interim block the parser skips).
+
+### Security
+
+- Storage: **`S3Store`'s `Debug` redacts credentials.** It previously derived `Debug` over `access_key`/`secret_key`/`session_token`, so printing a store — or any struct holding one — leaked the secret key into logs.
+- Storage: `SystemCurl` keeps credentials out of `argv` (config on stdin, so `Authorization` is invisible to `ps`/`/proc`), pins `--proto =https`, disables redirect following so a 3xx cannot replay a signature at an attacker-chosen host, floors TLS at 1.2, caps the body with `--max-filesize`, and exposes no way to disable certificate verification. `PUT` bodies stage through a `0600`, `O_EXCL` temp file removed on completion (`tmp_dir` points it at a tmpfs).
+- Storage: header values are rejected if they carry control characters, so a caller-supplied `content_type` cannot inject a header; URLs carrying userinfo or whitespace are refused; response bodies, header counts and line lengths are all bounded.
+
 ## [0.9.0] - 2026-07-31
 
 The portable-queue release: a durable job queue no longer requires a Postgres server.

README.md

@@ -92,7 +92,7 @@ fn main() -> std::io::Result<()> {
 | `sutegi-web`  | Router, `App` builder, middleware, groups, extractors, streaming (`sse`/`stream`), `/__introspect`, and the agent tool surface (`App::tool`/`stream_tool`, `schema` helpers, `ToolCtx`, `/__tools`). |
 | `sutegi-orm`  | Typed schema, fluent parameterized query builder, one `Backend` trait, a JSON key/value store, and two runnable backends: **SQLite** (`sqlite`, single-node) and **Postgres** (`postgres`, multi-pod). |
 | `sutegi-pg`   | Pure-`std` PostgreSQL driver: wire protocol v3 over blocking TCP, SCRAM-SHA-256 auth, connection pool. No async runtime, no C library. |
-| `sutegi-storage` | File/object storage behind one `Storage` trait: local fs, database blobs (over `Backend`), and a pure-`std` S3 SigV4 presigner. |
+| `sutegi-storage` | File/object storage behind one `Storage` trait: local fs, database blobs (over `Backend`), and S3/R2 buckets over a pluggable HTTP transport — plus a pure-`std` SigV4 presigner. |
 | `sutegi-macros` | `#[derive(Model)]` (schema, hydration, `save`, `from_input`) and `#[derive(Validate)]` (field-attr rulesets). Compile-time only (syn/quote never reach your binary). |
 | `sutegi-validate` | Fluent `Validator`-style rule sets **and** a JSON Schema subset validator, with structured errors. |
 | `sutegi-queue` | Durable job queue over the `Backend` seam — same SQL on SQLite or Postgres (`UPDATE … RETURNING` claim, `SKIP LOCKED` where the backend has it, visibility-timeout retries, priorities, named queues, dedupe keys, dead-letter). |
@@ -121,7 +121,7 @@ what you use:
 | `template` |   | Blade-style template engine (`{{ }}`, `@if`, `@foreach`, `@include`) |
 | `mail`     |   | `Email` builder + themed messages + `Transport` seam + drivers |
 | `auth-mail` |  | email-verification + password-reset flows on top of both |
-| `storage`  |   | file storage: local fs backend + S3 presigned URLs (pure std) |
+| `storage`  |   | file storage: local fs, S3/R2 objects, presigned URLs (pure std) |
 | `storage-db` |  | blobs in SQLite/Postgres over the same `Backend` seam |
 
 ```toml
@@ -580,18 +580,28 @@ opinion per backend:
 - **`DbStorage<B>`** (`storage-db`) — blobs in a database table over any ORM
   `Backend`. On Postgres that is **multi-pod file storage with zero new
   infrastructure**; honest ceiling ~a few MB per object.
-- **`S3Store`** — a pure-`std` **S3 SigV4 presigner** (AWS, R2, MinIO, …). It
-  mints time-limited GET/PUT/DELETE URLs and the bytes flow **directly between
-  the client and the object store** — no HTTP client, no TLS stack, no bytes
-  proxied. Signing reuses the Postgres driver's SCRAM crypto and is verified
-  against AWS's published known-answer vector.
+- **`S3Storage<T>`** — a real S3-compatible bucket: AWS S3, **Cloudflare R2**,
+  MinIO, Garage, Ceph RGW. The backend for objects past a database row's
+  comfort, and the one that survives ephemeral pods. It moves the bytes itself
+  over an injected `HttpTransport`, which is how sutegi ships an S3 client
+  while still having **no TLS stack and no third-party dependency**.
+- **`S3Store`** — the credentials behind it, and on its own a pure-`std` **SigV4
+  presigner**: time-limited GET/PUT/DELETE URLs whose bytes flow **directly
+  between the client and the object store**, never proxied. Signing reuses the
+  Postgres driver's SCRAM crypto and is verified against AWS's published
+  known-answer vectors — for presigned URLs *and* signed headers.
 
 ```rust
 use sutegi::prelude::*;
 
 let store = FsStorage::new("data/files")?;        // or DbStorage::new(pg)
 store.put("reports/q2.pdf", &bytes, "application/pdf")?;
 
+// Same trait, same call sites, a bucket instead of a disk:
+let r2 = S3Store::r2(&account, "media", &ak, &sk).storage(SystemCurl::new());
+r2.put("reports/q2.pdf", &bytes, "application/pdf")?;
+let objects = r2.list("reports/")?;              // Vec<ObjectMeta>
+
 // The agent-native shape: a tool mints an upload URL, the agent PUTs the
 // bytes itself — your app only ever handles metadata.
 let s3 = S3Store::new("bucket", "eu-central-1", &ak, &sk);
@@ -603,10 +613,35 @@ app.tool("presign_upload", "Mint a time-limited S3 upload URL.",
     })
 ```
 
+### The transport seam (how S3 works without TLS in the tree)
+
+`S3Storage` never opens a socket itself. It hands a signed request to an
+`HttpTransport` — one method — and two implementations ship with it:
+
+- **`SystemCurl`** — `https` by delegating the handshake and certificate
+  verification to the system `curl`. The crypto that must not be hand-rolled
+  isn't, and the dependency count stays at zero. Credentials never reach
+  `argv`: the URL and every signed header go in on **stdin** (`--config -`), so
+  `Authorization` is invisible to `ps`. Protocol pinned with `--proto =https`,
+  redirects off (a 3xx must not replay a signature elsewhere), TLS 1.2 floor,
+  and no knob anywhere that disables verification.
+- **`PlainHttp`** — pure `std`, and it **refuses `https`** rather than
+  pretending. For a store on a trusted path: in-cluster MinIO, sidecar Garage,
+  a dev container. Same stance as the Postgres driver and the SMTP transport.
+- **yours** — `impl HttpTransport for MyClient` if you already pay for `ureq`
+  or `reqwest`.
+
+Every request is signed with the **real payload hash** (`x-amz-content-sha256`,
+never `UNSIGNED-PAYLOAD`), so a body altered in flight is refused by the store —
+integrity that holds even over `PlainHttp`. Downloads and uploads are checked
+against the `ETag` when it is a plain MD5, giving end-to-end verification on top
+of it. `list` follows continuation tokens and is bounded by `max_list_keys`: a
+ten-million-object bucket errors, never silently truncates.
+
 `S3Store` deliberately does not implement `Storage`: minting a URL is a
-different contract than moving bytes. A full proxying S3 client joins the
-trait once TLS lands. See `examples/storage` for the working file server +
-presign tools.
+different contract than moving bytes. `S3Store::storage(transport)` is the
+crossing point. See `examples/storage` for the working file server + presign
+tools.
 
 ## Collections
 

crates/sutegi-storage/Cargo.toml

@@ -6,8 +6,8 @@ rust-version.workspace = true
 license.workspace = true
 repository.workspace = true
 authors.workspace = true
-description = "File/object storage for sutegi: one Storage trait, local-fs and database-blob backends, plus a pure-std S3 SigV4 presigner. Zero third-party dependencies."
-keywords = ["storage", "s3", "presign", "zero-dependency"]
+description = "File/object storage for sutegi: one Storage trait over local-fs, database-blob, and S3/R2 backends, with a pure-std SigV4 signer/presigner and a pluggable HTTP transport. Zero third-party dependencies."
+keywords = ["storage", "s3", "r2", "presign", "zero-dependency"]
 categories = ["web-programming", "filesystem"]
 
 [dependencies]

crates/sutegi-storage/src/lib.rs

@@ -7,16 +7,23 @@
 //!   `Backend`. On Postgres this is **multi-pod file storage with zero new
 //!   infrastructure** — the database you already run is the blob store. Good
 //!   to roughly a few MB per object; past that, reach for real object storage.
-//! - [`S3Store`] — a pure-`std` **SigV4 presigner** for S3-compatible object
-//!   stores (AWS, R2, MinIO, …). It mints time-limited GET/PUT/DELETE URLs;
-//!   the bytes flow **directly between the client (or agent) and S3**, never
-//!   through sutegi. That is why it needs no HTTP client and no TLS — signing
-//!   is HMAC-SHA256, reused from the Postgres driver's SCRAM implementation.
+//! - [`S3Storage`] — an S3-compatible bucket: AWS S3, **Cloudflare R2**, MinIO,
+//!   Garage, Ceph RGW. The backend for objects too big for a database row, and
+//!   the one that keeps working when the pods are ephemeral. It moves the bytes
+//!   itself over an injected [`transport::HttpTransport`], which is how sutegi
+//!   gets a real S3 client while still shipping **no TLS stack and no
+//!   third-party dependency**: [`transport::SystemCurl`] borrows the system
+//!   `curl` for `https`, [`transport::PlainHttp`] is pure `std` for a store on
+//!   a trusted network, and your own client is one method away.
+//! - [`S3Store`] — the credentials behind [`S3Storage`], and on its own a
+//!   pure-`std` **SigV4 presigner**: it mints time-limited GET/PUT/DELETE URLs
+//!   so the bytes flow **directly between the client (or agent) and S3**, never
+//!   through sutegi. Presigning needs no HTTP client at all — signing is
+//!   HMAC-SHA256, reused from the Postgres driver's SCRAM implementation.
 //!
-//! `S3Store` deliberately does **not** implement [`Storage`]: handing out a
-//! URL is a different contract than moving bytes. When a full S3 client lands
-//! (blocked on TLS), it will join the trait; until then the swap seam spans
-//! fs ↔ db, and S3 is the presign-only escape hatch.
+//! `S3Store` deliberately does **not** implement [`Storage`]: handing out a URL
+//! is a different contract than moving bytes. [`S3Store::storage`] is the
+//! crossing point — same credentials, the other contract.
 //!
 //! Keys are `/`-separated paths (`avatars/42.png`), validated identically
 //! across backends — see [`validate_key`].
@@ -35,8 +42,12 @@ use sutegi_json::Json;
 
 pub mod fs;
 pub mod s3;
+pub mod s3_storage;
+pub mod transport;
 pub use fs::FsStorage;
 pub use s3::S3Store;
+pub use s3_storage::S3Storage;
+pub use transport::{HttpTransport, PlainHttp, SystemCurl};
 
 #[cfg(feature = "db")]
 pub mod db;
@@ -70,9 +81,9 @@ impl ObjectMeta {
     }
 }
 
-/// A byte-level object store. Implemented by [`FsStorage`] and [`DbStorage`];
-/// app code holds `impl Storage` (or a concrete type) and swaps backends by
-/// changing the type it constructs, not the call sites.
+/// A byte-level object store. Implemented by [`FsStorage`], [`DbStorage`] and
+/// [`S3Storage`]; app code holds `impl Storage` (or a concrete type) and swaps
+/// backends by changing the type it constructs, not the call sites.
 pub trait Storage {
     /// Store `bytes` at `key` (create or overwrite), recording `content_type`.
     fn put(&self, key: &str, bytes: &[u8], content_type: &str) -> Result<(), String>;