Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ src/ops/ compact / expire / manifests / orphans
delete.rs the one deleter both destructive ops share
reachability.rs what a table still needs — safety-critical
store.rs object listing, which FileIO does not offer
src/sched/ daemon, cron triggers, event debounce
src/sched/ daemon, cron triggers, event debounce, the /status board
src/obs/ observer trait, audit log, metrics
src/cli/ clap commands, human + JSON output
```
Expand Down
25 changes: 21 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,28 @@ and the project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.htm

Bergman is pre-1.0: **breaking changes land in minor versions**, each with a migration note.

## [Unreleased]
## [0.5.0] — 2026-11-20

**Breaking:** `obs::router`, `obs::router_with_events` and `obs::serve` take a `sched::StatusBoard`
— pass `daemon.status()`. `metrics` is a default feature; a library that wants neither `axum` nor
`prometheus-client` sets `default-features = false`.

### Added

- **`bergman status`** asks a running daemon what it is doing and what needs attention, from the
new **`/status`** endpoint. No config or credentials needed; failures and refusals print first;
exits `2` when something needs attention. `Daemon::status()` exposes the same board to embedders.

### Changed

- `cargo install bergman` includes the daemon's HTTP surface and `bergman status`, like every
published binary.

### Security

- `chrono`, `chrono-tz` and `humantime` are marked unmaintained (RUSTSEC-2026-0543, -0555, -0542),
with no patched version and no vulnerability. All three reach Bergman through arrow, datafusion,
iceberg and object_store, so they are ignored in `deny.toml` with that reason.
- `chrono`, `chrono-tz` and `humantime` (RUSTSEC-2026-0543, -0555, -0542) are flagged unmaintained,
with no fix and no vulnerability. They reach Bergman through arrow, datafusion, iceberg and
object_store, and are ignored in `deny.toml` with that reason.

## [0.4.0] — 2026-11-08

Expand Down Expand Up @@ -185,6 +200,8 @@ Initial release. Table health analysis, snapshot expiration with file cleanup,
orphan-file removal, manifest rewriting, and compaction with delete-file
application — as one crate, one binary, and a library, on `iceberg-rust`.

[0.5.0]: https://github.com/hupe1980/bergman/compare/v0.4.0...v0.5.0
[0.4.0]: https://github.com/hupe1980/bergman/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/hupe1980/bergman/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/hupe1980/bergman/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/hupe1980/bergman/releases/tag/v0.1.0
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "bergman"
version = "0.4.0"
version = "0.5.0"
edition = "2024"
authors = ["hupe1980"]
description = "A Rust-native maintenance engine for Apache Iceberg — policy-driven compaction, snapshot expiration, and orphan cleanup with no JVM"
Expand All @@ -17,7 +17,7 @@ rust-version = "1.95"
# Optional dependencies are `dep:` so the feature surface is exactly this list.
# Embedders take `default-features = false` and add what they use.
[features]
default = ["cli", "catalog-rest", "compaction", "storage-all"]
default = ["cli", "catalog-rest", "compaction", "storage-all", "metrics"]

# The only operation that reads and writes data. DataFusion applies equality
# deletes as a hash anti-join; without the feature there is no compaction.
Expand Down
182 changes: 97 additions & 85 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,82 +1,50 @@
# Bergman
# 🏔️ Bergman

**A Rust-native maintenance engine for Apache Iceberg. One crate. One binary. No JVM.**

Bergman keeps Iceberg tables healthy — compacting small files, applying and
retiring delete files, expiring snapshots, re-packing manifests and removing
orphans — driven by declarative policies and built on
[`iceberg-rust`](https://github.com/apache/iceberg-rust) 0.11.0. No Spark, no
Bergman keeps Iceberg tables healthy: it compacts small files, applies and
retires delete files, expires snapshots, re-packs manifests and removes orphans.
Declarative policies drive it, and it is built on
[`iceberg-rust`](https://github.com/apache/iceberg-rust) 0.11.0 — no Spark, no
Trino, no cluster. It is **library-first**: the `bergman` binary is a thin
consumer of the same public API an embedder gets.

```bash
bergman inspect # what is wrong with my tables? (reads only)
bergman plan # what would maintenance do? (reads only)
bergman run # do it
bergman inspect # 🔍 what is wrong with my tables? (reads only)
bergman plan # 🧭 what would maintenance do, and why? (reads only)
bergman run # 🛠️ do it
bergman status # 📡 what is the daemon doing, and what needs me?
```

- **Documentation:** <https://hupe1980.github.io/bergman/>
- **Sister project:** [Rustberg](https://github.com/hupe1980/rustberg), an
authenticated, policy-controlled Iceberg REST catalog. Each works fully
without the other.
📚 **Docs:** <https://hupe1980.github.io/bergman/> ·
🤝 **Sister project:** [Rustberg](https://github.com/hupe1980/rustberg), an
authenticated, policy-controlled Iceberg REST catalog. Each works fully without
the other.

---

## Install
## 📦 Install

```bash
brew install hupe1980/tap/bergman # macOS (Apple Silicon) and Linux
cargo install bergman # anywhere Rust builds
brew install hupe1980/tap/bergman # macOS (Apple Silicon) and Linux
cargo install bergman # anywhere Rust 1.95+ builds
```

As a container — distroless, non-root, statically linked:
🐳 As a container — distroless, non-root, statically linked:

```bash
docker run --rm \
-v ./bergman.toml:/etc/bergman/bergman.toml:ro \
ghcr.io/hupe1980/bergman:latest plan
```

Or a binary from the [releases page](https://github.com/hupe1980/bergman/releases):
static musl Linux for `x86_64` and `aarch64`, and macOS on Apple Silicon, each with
a `.sha256` beside it. From source, Bergman requires Rust 1.95+.
Or grab a binary from the [releases page](https://github.com/hupe1980/bergman/releases):
static musl Linux for `x86_64` and `aarch64`, and macOS on Apple Silicon, each
with a `.sha256` beside it.

---

## What it does

| Operation | Built on |
|---|---|
| Table health analysis | Manifest metadata only; no data file is opened |
| Snapshot expiration | Upstream `ExpireSnapshotsAction` |
| Expiration file cleanup | Bergman — upstream leaves it to the caller |
| Orphan-file removal | Bergman's object-store layer — `FileIO` has no `list` |
| Manifest rewrite | Bergman's commit layer |
| Compaction | Upstream's scan and writers, Bergman's commit layer |
| Dangling delete-file removal | Bergman's commit layer |

Compaction is **triggered**, not scheduled — by small files, oversized files,
delete ratio, and a settle time that leaves a partition being written alone. It
honours the table's `sort-order` and `write.*` properties, and applies equality
deletes with a DataFusion hash anti-join.
[Compaction →](https://hupe1980.github.io/bergman/docs/compaction/)

**Format v3** tables are maintained in full: a rewrite writes each row's
`_row_id` and `_last_updated_sequence_number` physically and the snapshot claims
its row range; deletion vectors are applied and retired by `(path, offset)`.

**Refused, by name, in `bergman plan`:** encrypted tables (snapshot expiration
still runs), anything but Parquet, cross-spec rewrites, and any branch but
`main`. [Status →](https://hupe1980.github.io/bergman/docs/status/)

Bergman owns its commit layer because `iceberg::Transaction` cannot express a
rewrite: it builds the commit from upstream's public writers and `POST`s it to
the table endpoint, behind a `TableCommitter` trait.
[Architecture →](https://hupe1980.github.io/bergman/docs/architecture/)

---

## Quickstart
## ⚡ Quickstart

`bergman.toml` — every setting is optional:

Expand Down Expand Up @@ -104,67 +72,108 @@ skip = true
```

```bash
bergman policy lint # validate offline, for CI
bergman policy explain prod.analytics.events # each setting and where it came from
bergman plan
bergman policy lint # ✅ validate offline, for CI
bergman policy explain prod.analytics.events # 🧾 each setting and where it came from
bergman plan # 🧭 read-only, always safe
```

Settings resolve, most specific first, through the matching **rule**,
**`[defaults]`**, the **catalog's own policy** (`policy_source = "polaris"`),
the **table's properties and `sort-order`**, and the **Iceberg default**.
the **table's properties and `sort-order`**, and the **Iceberg default** — and
`policy explain` shows which layer every value came from.
[Configuration →](https://hupe1980.github.io/bergman/docs/configuration/) ·
[Policies →](https://hupe1980.github.io/bergman/docs/policies/)

---

## Commands
## 🧹 What it does

| | Operation | Built on |
|:--:|---|---|
| 🔍 | Table health analysis | Manifest metadata only; no data file is opened |
| 🗜️ | Compaction | Upstream's scan and writers, Bergman's commit layer |
| 🧽 | Dangling delete-file removal | Bergman's commit layer |
| 📚 | Manifest rewrite | Bergman's commit layer |
| ⏳ | Snapshot expiration | Upstream `ExpireSnapshotsAction` |
| 🗑️ | Expiration file cleanup | Bergman — upstream leaves it to the caller |
| 👻 | Orphan-file removal | Bergman's object-store layer — `FileIO` has no `list` |

🎯 **Compaction is triggered, not scheduled** — by small files, oversized files,
delete ratio, and a settle time that leaves a partition being written alone. It
honours the table's `sort-order` and `write.*` properties, and applies equality
deletes with a DataFusion hash anti-join.
[Compaction →](https://hupe1980.github.io/bergman/docs/compaction/)

🆕 **Format v3** tables are maintained in full: a rewrite writes each row's
`_row_id` and `_last_updated_sequence_number` physically and the snapshot claims
its row range; deletion vectors are applied and retired by `(path, offset)`.

🚫 **Refused by name, never guessed at** — in `bergman plan`, the run report and
`bergman status`: encrypted tables (snapshot expiration still runs), anything but
Parquet, cross-spec rewrites, and any branch but `main`.
[Status →](https://hupe1980.github.io/bergman/docs/status/)

🧱 Bergman owns its commit layer because `iceberg::Transaction` cannot express a
rewrite: it builds the commit from upstream's public writers and `POST`s it to
the table endpoint, behind a `TableCommitter` trait.
[Architecture →](https://hupe1980.github.io/bergman/docs/architecture/)

---

## ⌨️ Commands

| Command | Writes? | What it does |
|---|:--:|---|
| `bergman inspect` | no | Table health: file counts, sizes, delete ratios, snapshots, manifests |
| `bergman plan` | no | The operations maintenance would perform, and why |
| `bergman run` | **yes** | Executes the plan |
| `bergman run --dry-run` | no | Identical to `plan` |
| `… --table <glob>` | — | Scopes `inspect`, `plan` and `run`; an excluded table is never read |
| `… --only <op>[,<op>]` | — | Scopes `plan` and `run` to `compact`, `rewrite-manifests`, `expire-snapshots`, `remove-orphans` |
| `bergman daemon` | **yes** | Runs cycles on schedules, or when told a table changed (`POST /events`, including Lakekeeper CloudEvents) |
| `bergman policy lint` | no | Validates config offline |
| `bergman policy explain <table>` | no | Effective policy with per-value provenance |
| `bergman policy match` | no | Which tables each rule matches |
| `bergman inspect` | — | Table health: file counts, sizes, delete ratios, snapshots, manifests |
| `bergman plan` | — | The operations maintenance would perform, and why — or why not |
| `bergman run` | ✍️ | Executes the plan |
| `bergman run --dry-run` | — | Identical to `plan` |
| `… --table <glob>` | | Scopes `inspect`, `plan` and `run`; an excluded table is never read |
| `… --only <op>[,<op>]` | | Scopes `plan` and `run` to `compact`, `rewrite-manifests`, `expire-snapshots`, `remove-orphans` |
| `bergman daemon` | ✍️ | Runs cycles on schedules, or when told a table changed (`POST /events`, including Lakekeeper CloudEvents) |
| `bergman status` | — | Asks a running daemon what it is doing; what needs attention comes first |
| `bergman policy lint` | — | Validates config offline |
| `bergman policy explain <table>` | — | Effective policy with per-value provenance |
| `bergman policy match` | — | Which tables each rule matches |

Global flags: `--config`, `--format text|json`, `--audit-log <path>`, `--log <level>`.
`bergman run` exits `2` when any operation failed or was refused.
[Operating →](https://hupe1980.github.io/bergman/docs/operating/)
`bergman run` and `bergman status` exit `2` when an operation failed or was
refused, so a scheduler or a check that reads only exit codes still notices.

📡 A daemon started with `--listen` serves `/metrics`, `/health`, `/status` and
(with `--events`) `/events`. `bergman status --daemon http://host:9090` reads
`/status` with no config file and no credentials — point it at a pod from your
laptop. [Operating →](https://hupe1980.github.io/bergman/docs/operating/)

---

## Safety
## 🛡️ Safety

- **Nothing destructive is on by default.** Compaction, orphan deletion and
- 🔒 **Nothing destructive is on by default.** Compaction, orphan deletion and
expiration's file cleanup default off; metadata-only snapshot expiration is on.
- **A table Bergman cannot fully read is refused**, and every manifest walk is
- 📖 **A table Bergman cannot fully read is refused**, and every manifest walk is
all-or-nothing, so nothing is deleted on a partial reading.
- **Orphan removal has eight independent checks**, including a 24-hour grace
- 👻 **Orphan removal has eight independent checks**, including a 24-hour grace
floor, segment-wise containment, re-verification before deleting, a deletion
ceiling, and a refusal when the listing and the metadata share no file.
[Orphans →](https://hupe1980.github.io/bergman/docs/orphans/)
- **One deleter**: orphan removal and expiration cleanup share one blast-radius
- 🎯 **One deleter**: orphan removal and expiration cleanup share one blast-radius
ceiling, audit record and bounded deletion path.
- **A rewrite never drops a delete file another file still needs**, and every
- 🧩 **A rewrite never drops a delete file another file still needs**, and every
file group's row count must land between a floor and a ceiling taken from
metadata.
- **Carried-forward files keep their sequence numbers; manifests keep their
- 🔢 **Carried-forward files keep their sequence numbers; manifests keep their
partition spec; a moved branch keeps its retention.**
- **Commits are compare-and-swap; a lost one is rebuilt, never re-offered.** A
- ⚖️ **Commits are compare-and-swap; a lost one is rebuilt, never re-offered.** A
429/503 is retried; a 409/412 replans.
- **Every operation is auditable**: `--audit-log` writes JSON Lines, and a
- 🧾 **Every operation is auditable**: `--audit-log` writes JSON Lines, and a
deletion manifest is written before the first delete.
- **Secrets never reach a log**: credentials and catalog config redact in
- 🤫 **Secrets never reach a log**: credentials and catalog config redact in
`Debug`. `POST /events` can require a bearer token (`--events-token-env`).

---

## Library
## 🦀 Library

```rust
use bergman::{Bergman, policy::Config};
Expand Down Expand Up @@ -198,19 +207,22 @@ impl MaintenanceObserver for RequireSignoff {
}
```

Embedding the daemon? `Daemon::status()` hands you the same live board
`/status` serves.

| Feature | Default | Carries |
|---|:--:|---|
| `cli` | ✅ | The binary, terminal rendering, logging setup |
| `catalog-rest` | ✅ | Iceberg REST catalog client |
| `compaction` | ✅ | Data-file rewriting, and the DataFusion executor it needs |
| `storage-s3` / `-gcs` / `-azure` | ✅ via `storage-all` | Cloud object stores |
| `metrics` | — | Prometheus metrics, and the daemon's HTTP surface: `/metrics`, `/health`, `/events` |
| `metrics` | ✅ | Prometheus metrics, and the daemon's HTTP surface: `/metrics`, `/health`, `/status`, `/events`; `bergman status` |

[Library →](https://hupe1980.github.io/bergman/docs/library/)

---

## Development
## 🧪 Development

```bash
just test # the full suite
Expand All @@ -226,11 +238,11 @@ just cross-engine # opt-in, needs uv: PyIceberg 0.12.0 writes, Bergman maintain
The suite needs no containers or cloud credentials: it builds real tables on the
local filesystem and commits to an in-memory catalog.

## Changelog
## 📝 Changelog

[CHANGELOG.md](CHANGELOG.md). Bergman is pre-1.0: breaking changes land in minor
versions, with migration notes.

## License
## ⚖️ License

Apache-2.0 OR MIT.
2 changes: 1 addition & 1 deletion site/content/docs/library.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ cargo add bergman --no-default-features --features catalog-rest,storage-s3
| `catalog-rest` | ✅ | The Iceberg REST catalog client |
| `compaction` | ✅ | Data-file rewriting — **and DataFusion**, which applies equality deletes as a hash anti-join |
| `storage-s3` / `-gcs` / `-azure` | ✅ via `storage-all` | Cloud object stores |
| `metrics` | — | Prometheus metrics and the `/metrics` endpoint |
| `metrics` | ✅ | Prometheus metrics and the daemon's HTTP surface: `/metrics`, `/health`, `/status`, `/events` |

`compaction` is the only operation that reads and writes *data*, and the only
one needing a query engine ([why](@/docs/compaction.md#query-engine)). For
Expand Down
Loading
Loading