v0.2.0 — Orchestration layer

Slate v0.2.0

The orchestration layer. Slate is no longer just a fast file copier — it now runs jobs from a persistent queue with retry, pipeline chaining, webhooks, and cost tracking.

New in v0.2.0

Worker queue
Jobs are queued and picked up by a background worker (configurable concurrency via SLATE_WORKER_CONCURRENCY, default 4). The API returns immediately on job submission.

Retry with exponential backoff
Failed jobs automatically retry at 30s → 5min → 30min. Configurable max_attempts per job (default 3).

Pipeline chaining

# Job B waits for Job A to complete before starting
curl -X POST localhost:3030/jobs \
  -d '{"src": "s3://raw/data", "dst": "gs://processed/data", "depends_on": "<job-a-id>"}'

Webhook callbacks

curl -X POST localhost:3030/jobs \
  -d '{"src": "...", "dst": "...", "callback_url": "https://your-service/hooks/slate"}'
# Fires HTTP POST when job completes or fails

Cost tracking

curl localhost:3030/cost             # aggregate egress cost across all completed jobs
curl localhost:3030/jobs/<id>/cost   # per-job cost estimate

New job fields: priority, attempt, max_attempts, run_after, depends_on, callback_url, started_at, completed_at, peak_throughput_mbps

Cancel queued jobs

curl -X POST localhost:3030/jobs/<id>/cancel

Install

curl -fsSL https://github.com/tsushanth/slate/releases/latest/download/slate-linux-x86_64.tar.gz \
  | tar xz -C /usr/local/bin

Benchmark: 984 MB/s cross-region (S3 us-east-1 → Hetzner Frankfurt), 4.4× faster than aws s3 cp, within 14% of rclone.