Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions cmd/bd/backup.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,21 +19,25 @@ var backupCmd = &cobra.Command{
Without a subcommand, exports all tables to JSONL files in .beads/backup/.
Events are exported incrementally using a high-water mark.

For Dolt-native backups (preserves full commit history, faster for large databases):
JSONL backup commands (portable snapshot for backup/restore transport):
bd backup Export the current JSONL backup snapshot
bd backup status Show JSONL + Dolt backup status
bd backup export-git Publish the JSONL snapshot to a git branch
bd backup fetch-git Fetch a backup snapshot from a git branch and restore it
bd backup restore [path] Restore from JSONL backup files

Dolt-native backup commands (preserve full commit history, faster for large databases):
bd backup init <path> Set up a backup destination (filesystem or DoltHub)
bd backup sync Push to configured backup destination

Other subcommands:
bd backup status Show backup status (JSONL + Dolt)
bd backup export-git Export the current JSONL snapshot to a git branch
bd backup restore [path] Restore from JSONL backup files

DoltHub is recommended for cloud backup:
bd backup init https://doltremoteapi.dolthub.com/<user>/<repo>
Set DOLT_REMOTE_USER and DOLT_REMOTE_PASSWORD for authentication.

Note: Git-protocol remotes are NOT recommended for Dolt backups — push times
exceed 20 minutes, cache grows unboundedly, and force-push is needed after recovery.`,
Note: The git-protocol remote warning below applies to Dolt-native backups,
not to 'bd backup export-git'. Git-protocol remotes are NOT recommended for
Dolt backups — push times exceed 20 minutes, cache grows unboundedly, and
force-push is needed after recovery.`,
GroupID: "sync",
RunE: func(cmd *cobra.Command, args []string) error {
state, err := runBackupExport(rootCtx, backupForce)
Expand Down Expand Up @@ -107,6 +111,8 @@ var backupStatusCmd = &cobra.Command{
fmt.Println()
fmt.Println("JSONL backup (portable):")
fmt.Println(" bd backup Run JSONL export now")
fmt.Println(" bd backup export-git Publish snapshot to a git branch")
fmt.Println(" bd backup fetch-git Restore from a backup git branch")
fmt.Println(" Auto-backup runs every 15m when a git remote is detected")
fmt.Println()
fmt.Println("Dolt backup (preserves history, faster for large databases):")
Expand Down
24 changes: 13 additions & 11 deletions cmd/bd/backup_export.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,20 @@ type dbQuerier interface {
}

// backupState tracks watermarks for incremental backup.
type backupCounts struct {
Issues int `json:"issues"`
Events int `json:"events"`
Comments int `json:"comments"`
Dependencies int `json:"dependencies"`
Labels int `json:"labels"`
Config int `json:"config"`
}

type backupState struct {
LastDoltCommit string `json:"last_dolt_commit"`
LastEventID int64 `json:"last_event_id"`
Timestamp time.Time `json:"timestamp"`
Counts struct {
Issues int `json:"issues"`
Events int `json:"events"`
Comments int `json:"comments"`
Dependencies int `json:"dependencies"`
Labels int `json:"labels"`
Config int `json:"config"`
} `json:"counts"`
LastDoltCommit string `json:"last_dolt_commit"`
LastEventID int64 `json:"last_event_id"`
Timestamp time.Time `json:"timestamp"`
Counts backupCounts `json:"counts"`
}

// backupDir returns the backup directory path, creating it if needed.
Expand Down
28 changes: 6 additions & 22 deletions cmd/bd/backup_git_transport.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,11 @@ const (
)

type backupGitManifest struct {
Format string `json:"format"`
BDVersion string `json:"bd_version"`
SnapshotTimestamp time.Time `json:"snapshot_timestamp"`
LastDoltCommit string `json:"last_dolt_commit"`
Counts manifestCounts `json:"counts"`
}

type manifestCounts struct {
Issues int `json:"issues"`
Events int `json:"events"`
Comments int `json:"comments"`
Dependencies int `json:"dependencies"`
Labels int `json:"labels"`
Config int `json:"config"`
Format string `json:"format"`
BDVersion string `json:"bd_version"`
SnapshotTimestamp time.Time `json:"snapshot_timestamp"`
LastDoltCommit string `json:"last_dolt_commit"`
Counts backupCounts `json:"counts"`
}

func normalizeBackupGitRef(value, fallback string) string {
Expand Down Expand Up @@ -202,14 +193,7 @@ func writeBackupGitManifest(dstDir string, state *backupState) error {
BDVersion: Version,
SnapshotTimestamp: state.Timestamp,
LastDoltCommit: state.LastDoltCommit,
Counts: manifestCounts{
Issues: state.Counts.Issues,
Events: state.Counts.Events,
Comments: state.Counts.Comments,
Dependencies: state.Counts.Dependencies,
Labels: state.Counts.Labels,
Config: state.Counts.Config,
},
Counts: state.Counts,
}

data, err := json.MarshalIndent(manifest, "", " ")
Expand Down
3 changes: 3 additions & 0 deletions cmd/bd/backup_restore.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ This command:
Use this after losing your Dolt database (machine crash, new clone, etc.)
when you have JSONL backups on disk or in git.

If your backup snapshots are stored in a git branch, use 'bd backup fetch-git'
to fetch that branch into a temporary worktree and restore from it.

The database must already be initialized (run 'bd init' first if needed).
To initialize and restore in one step, use: bd init && bd backup restore`,
Args: cobra.MaximumNArgs(1),
Expand Down
2 changes: 1 addition & 1 deletion cmd/bd/doctor_pollution.go
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ func runPollutionCheck(_ string, clean bool, yes bool) {
}

fmt.Printf("%s Deleted %d test issues\n", ui.RenderPass("✓"), deleted)
fmt.Printf("\nCleanup complete. To restore, run: bd import %s\n", backupPath)
fmt.Printf("\nCleanup complete. To restore, run: bd init --from-jsonl %s\n", backupPath)
}

func init() {
Expand Down
8 changes: 6 additions & 2 deletions cmd/bd/export.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,12 @@ var exportCmd = &cobra.Command{
Long: `Export all issues to JSONL (newline-delimited JSON) format.

Each line is a complete JSON object representing one issue, including its
labels, dependencies, and comment count. The output is compatible with
'bd import' for round-trip backup and restore.
labels, dependencies, and comment count.

This command is for issue export, migration, and interoperability. It does
not produce the JSONL backup snapshot used by 'bd backup restore'. For
supported backup/restore flows, use 'bd backup', 'bd backup export-git',
and 'bd backup restore'.

By default, exports only regular issues (excluding infrastructure beads
like agents, rigs, roles, and messages). Use --all to include everything.
Expand Down
2 changes: 1 addition & 1 deletion cmd/bd/tips.go
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ func initDefaultTips() {
// This is a proactive health check that trumps educational tips (ox-cli pattern)
InjectTip(
"sync_conflict",
"Run 'bd sync' to resolve sync conflict",
"Run 'bd dolt pull' to resolve sync conflict",
200, // Higher than Claude setup - sync issues are urgent
0, // No frequency limit - always show when applicable
1.0, // 100% probability - always show when condition is true
Expand Down
8 changes: 5 additions & 3 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ bd's core design enables a distributed, Dolt-powered issue tracker that feels li

**Dolt for distribution:** Native push/pull to Dolt remotes (DoltHub, S3, GCS). No special sync server needed. Issues travel with your code. Offline work just works.

**Export for portability:** `bd export` outputs JSONL format for data migration and interoperability. Use `bd init --from-jsonl` to bootstrap a new database from an export.
**Export and backup:** `bd export` outputs issue JSONL for data migration and interoperability. Use `bd backup` and `bd backup restore` for supported JSONL backup snapshots, and `bd backup export-git` / `bd backup fetch-git` when you want those snapshots stored in a git branch.

## Write Path

Expand Down Expand Up @@ -85,7 +85,8 @@ All queries run directly against the local Dolt database:
2. **Sync:** Use `bd dolt pull` to fetch updates from Dolt remotes

Key implementation:
- Import (for bootstrapping/migration): `cmd/bd/import.go`
- JSONL backup restore: `cmd/bd/backup_restore.go`
- Issue bootstrap/migration: `cmd/bd/init.go`
- Dolt storage: `internal/storage/dolt/`

## Hash-Based Collision Prevention
Expand Down Expand Up @@ -296,7 +297,8 @@ Each issue in the Dolt database (and in JSONL exports via `bd export`) has the f
| Dolt implementation | `internal/storage/dolt/` |
| RPC protocol | `internal/rpc/protocol.go`, `server_*.go` |
| Export logic (portability) | `cmd/bd/export.go` |
| Import logic (migration) | `cmd/bd/import.go` |
| JSONL backup restore | `cmd/bd/backup_restore.go` |
| Issue bootstrap/migration | `cmd/bd/init.go` |

## Wisps and Molecules

Expand Down
12 changes: 9 additions & 3 deletions docs/CLI_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -561,13 +561,19 @@ bd gate add-waiter <gate-id> <waiter>

## Database Management

### Export / Bootstrap
### Export / Backup / Bootstrap

```bash
# Export issues to JSONL
# Export issues to issue JSONL
bd export -o issues.jsonl

# Bootstrap a new database from an export
# Write or restore the supported JSONL backup snapshot
bd backup
bd backup restore
bd backup export-git
bd backup fetch-git

# Bootstrap a new database from an issue export
bd init --from-jsonl # Reads .beads/issues.jsonl

# Configure orphan handling for pulls and bootstrapping
Expand Down
2 changes: 1 addition & 1 deletion docs/CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ The sync mode controls how beads synchronizes data with git and/or Dolt remotes.

#### Sync Mode

Beads uses `dolt-native` sync mode exclusively. Dolt remotes handle sync directly with cell-level merge. Use `bd export` for data portability and `bd init --from-jsonl` to bootstrap a new database from an export.
Beads uses `dolt-native` sync mode exclusively. Dolt remotes handle sync directly with cell-level merge. Use `bd export` for issue portability, `bd backup` / `bd backup restore` for supported JSONL backup snapshots, and `bd backup export-git` / `bd backup fetch-git` to move those snapshots through a git branch.

#### Sync Triggers

Expand Down
2 changes: 1 addition & 1 deletion docs/DOLT-BACKEND.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ Beads uses `dolt-native` sync mode exclusively:
- Uses Dolt remotes (DoltHub, S3, GCS, etc.)
- Native database-level sync with cell-level merge
- Supports branching and merging
- `bd export` available for data portability; `bd init --from-jsonl` for bootstrapping from exports
- `bd export` available for issue portability; `bd backup` / `bd backup restore` for JSONL backup snapshots

## Dolt Remotes

Expand Down
2 changes: 1 addition & 1 deletion docs/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ bd dolt push # Push changes to Dolt remote
bd dolt pull # Pull changes from Dolt remote
```

The `bd export` command exists for data portability (e.g., backing up data), and `bd init --from-jsonl` can bootstrap a new database from an export. These are not needed for day-to-day sync.
The `bd export` command exists for issue portability and interchange. For supported backup and restore flows, use `bd backup` to write JSONL backup snapshots, `bd backup restore` to restore them, and `bd backup export-git` / `bd backup fetch-git` if you want those snapshots published to a git branch. None of these are needed for day-to-day Dolt sync.

### What if my database feels stale after a colleague pushes changes?

Expand Down
8 changes: 5 additions & 3 deletions docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -443,12 +443,14 @@ bd config set sync.branch "" # Disable sync branch feature
For **physical database corruption** (disk failures, power loss, filesystem errors):

```bash
# If corrupted, rebuild from a Dolt remote or from an export backup
# If corrupted, rebuild from a Dolt remote or from a backup snapshot
mv .beads/dolt .beads/dolt.backup
bd init
bd dolt pull # Pull from Dolt remote if configured
# Or re-initialize from a backup export:
# bd init --from-jsonl
# Or restore from a local backup snapshot:
# bd backup restore
# Or fetch one from a backup branch:
# bd backup fetch-git
```

For **logical consistency issues** (ID collisions from branch merges, parallel workers):
Expand Down
4 changes: 2 additions & 2 deletions website/docs/architecture/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ flowchart TD
:::info Source of Truth
**Dolt** is the source of truth. Every write auto-commits to Dolt history, providing full version control, branching, and merge capabilities at the database level.

Recovery is straightforward: pull from a Dolt remote with `bd dolt pull`, or use `bd init --from-jsonl` to bootstrap from a JSONL backup.
Recovery is straightforward: pull from a Dolt remote with `bd dolt pull`, restore a local JSONL backup snapshot with `bd backup restore`, or fetch one from a git branch with `bd backup fetch-git`.
:::

### Why Dolt?
Expand Down Expand Up @@ -139,7 +139,7 @@ See [Sync Failures Recovery](/recovery/sync-failures) for sync race condition tr
Dolt's version control makes recovery straightforward:

1. **Lost database?** → Pull from Dolt remote: `bd dolt pull`
2. **Have a JSONL backup?** → Bootstrap from it: `bd init --from-jsonl backup.jsonl`
2. **Have a JSONL backup snapshot?** → Restore it: `bd backup restore`
3. **Merge conflicts?** → Dolt handles cell-level merge natively

### Universal Recovery Sequence
Expand Down
6 changes: 5 additions & 1 deletion website/docs/getting-started/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,11 @@ bd migrate
If you need to restore from a JSONL backup:

```bash
bd init --from-jsonl backup.jsonl
bd init
bd backup restore

# Or fetch the latest snapshot from a backup git branch
bd backup fetch-git
```

Or pull from a Dolt remote:
Expand Down
14 changes: 10 additions & 4 deletions website/docs/reference/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,11 @@ bd doctor --fix
# Or pull from Dolt remote
bd dolt pull

# Or bootstrap from a JSONL backup if available
bd init --from-jsonl backup.jsonl
# Or restore from a local JSONL backup snapshot
bd backup restore

# Or fetch the latest snapshot from a backup git branch
bd backup fetch-git
```

## Dolt Server Issues
Expand Down Expand Up @@ -113,8 +116,11 @@ bd hooks status
### Recovery from backup

```bash
# Bootstrap from JSONL backup
bd init --from-jsonl backup.jsonl
# Restore from a local JSONL backup snapshot
bd backup restore

# Or fetch the latest snapshot from a backup git branch
bd backup fetch-git

# Or pull from Dolt remote
bd dolt pull
Expand Down
20 changes: 11 additions & 9 deletions website/static/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,17 @@ bd show bd-42 --json # Show details
bd update bd-42 --claim # Start work
bd close bd-42 --reason "Done" # Close issue
bd ready --json # Show unblocked work
bd sync # Sync to git
bd dolt push # Push to Dolt remote
bd backup export-git # Publish backup snapshot to a git branch
```

## For AI Agents

- Always use `--json` for programmatic access
- Always include `--description` when creating issues
- Use `--deps discovered-from:<id>` to link found issues
- Run `bd sync` at end of every session
- Use `bd dolt push` / `bd dolt pull` for Dolt remotes
- Use `bd backup` / `bd backup export-git` for portable backups

## Key Concepts

Expand All @@ -41,27 +43,27 @@ bd sync # Sync to git
- **Formulas**: Declarative workflow templates (TOML/JSON)
- **Molecules**: Work graphs from formulas
- **Gates**: Async coordination (human/timer/github)
- **Wisps**: Ephemeral workflows (don't sync to git)
- **Wisps**: Ephemeral workflows (excluded from JSONL backups)

## Recovery

Quick fixes for common issues:

- Database Corruption: `git checkout HEAD~1 -- .beads/`
- Merge Conflicts: Resolve JSONL conflicts, then `bd sync`
- Database Recovery: `bd backup restore` or `bd backup fetch-git`
- Merge Conflicts: Use Dolt merge/recovery tools, then `bd dolt pull`
- Circular Dependencies: `bd doctor` (diagnose only, NEVER --fix)
- Sync Failures: `bd sync --import-only`
- Sync Failures: `bd dolt pull`

Full runbooks: https://steveyegge.github.io/beads/recovery/

## Session Close Protocol

Before ending any AI session:
1. `bd sync` - push changes to git
2. `bd status` - verify clean state
1. `bd dolt push` - if using a Dolt remote
2. `git push` - publish your code changes
3. Resolve conflicts before closing

WARNING: Skipping sync causes data loss in multi-agent workflows.
WARNING: Skipping sync or backup publication causes stale state in multi-agent workflows.

## Documentation

Expand Down