Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Unreleased
* Added "backup" command (full-account export via the CouchDB sync database)
* Added syncServer/syncDatabase/syncUser/syncPassword config keys

# 0.5.1 (2023-03-24)
* Fixed default marvin-cli.json location on linux
* Updated deno and dependencies
Expand Down
58 changes: 56 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ COMMANDS:
get - Read an arbitrary document from your database
today - List Tasks and Projects that are scheduled today
tracking - Get the currently tracked task
backup - Back up your entire account to a JSON file
update - Update a Task, Project, or other (not yet implemented)
delete - Delete a Task, Project, or other (not yet implemented)
help - Help about any command
Expand All @@ -44,11 +45,64 @@ DESKTOP COMMANDS:
run - Start the desktop app (not yet implemented)
quickAdd - Open desktop quick add
list - List Tasks/Projects, optionally filtered
backup - Trigger backups (not yet implemented)
restore - Restore backups (not yet implemented)
quit - Shut down the app (not yet implemented)
```

# Backup

`marvin backup` dumps your entire account to a JSON file (an array of
records, mirroring Marvin's own backup format). Marvin's public API has no
bulk-export endpoint, so this reads directly from your CouchDB/Cloudant sync
database. Find the sync credentials in the API strategy settings or at
https://app.amazingmarvin.com/?api.

You can pass the credentials per run with `--sync-server`, `--sync-database`,
`--sync-user`, and `--sync-password`. Passing `--sync-password` this way keeps
it out of the plaintext config file:

```bash
marvin backup \
--sync-server https://... \
--sync-database XYZ \
--sync-user XYZ \
--sync-password "$MARVIN_SYNC_PASSWORD"
```

Alternatively, save them once with `marvin config` so you can just run
`marvin backup` (note this writes them, including `syncPassword`, to the
plaintext config file):

```bash
marvin config syncServer https://...
marvin config syncDatabase XYZ
marvin config syncUser XYZ
marvin config syncPassword XYZ
```

Either way, control the output with `-o`:

```bash
marvin backup # writes MarvinBackup-<YYYYMMDD>.json
marvin backup -o my-backup.json
marvin backup -o - # write to stdout
```

The backup also surfaces sync conflicts: if any records are in an unresolved
conflict state, a warning is printed and those records keep their internal
`_rev` field as a marker (matching Marvin's own backups). Resolve them via
Marvin's Database Check.

This is a live read of the database, not a guaranteed point-in-time snapshot —
for the most consistent result, back up when Marvin isn't actively syncing.

This is equivalent to Marvin's
[GUI backup](https://help.amazingmarvin.com/en/articles/2278325-how-to-backup-your-data)
with the **Device Local Settings/Data** option left unchecked: it includes
everything in the sync database, but not data that isn't synced between your
devices (trashed items, which sections are opened/closed, etc.), which the sync
database doesn't hold.

# Installation

Download a release for your platform [here](https://github.com/amazingmarvin/marvin-cli/releases).
Expand All @@ -62,4 +116,4 @@ Download a release for your platform [here](https://github.com/amazingmarvin/mar

# Configuring

Get your API Token from https://app.amazingmarvin.com/pre?api (or find it in the API strategy settings) then run `marvin config apiToken XYZ` or use the `--api-token` command line option.
Get your API Token from https://app.amazingmarvin.com/?api (or find it in the API strategy settings) then run `marvin config apiToken XYZ` or use the `--api-token` command line option.
1 change: 0 additions & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
* Finish help command
* Implement run command
* Implement quickAdd command
* Implement backup command
* Implement restore command
* Implement quit command
* Go through help text
168 changes: 167 additions & 1 deletion src/commands/backup.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,170 @@
import { getOptions, globalOptionHelp } from "../options.ts";
import { Params, Options } from "../types.ts";
import { type CouchDoc, fetchAllDocs, uuidOf } from "./backup_couch.ts";

// Back up the entire Marvin account by dumping the CouchDB/Cloudant sync
// database directly. The public API has no bulk-export endpoint, but every
// document lives in a single sync database that we can read with the
// syncServer/syncDatabase/syncUser/syncPassword credentials from the API
// strategy settings. The CouchDB plumbing lives in backup_couch.ts.
export default async function backup(params: Params, cmdOpt: Options) {
console.log("Not yet implemented");
if (cmdOpt.help) {
console.log(backupHelp);
Deno.exit(0);
}

if (params.length !== 0) {
console.error(backupHelp);
Deno.exit(1);
}

const opt = getOptions();

const missing = ["syncServer", "syncDatabase", "syncUser", "syncPassword"]
.filter((key) => !opt[key as keyof typeof opt]);
if (missing.length > 0) {
console.error(
`Missing sync credentials: ${missing.join(", ")}.\n` +
`The backup command reads your Marvin data directly from its CouchDB\n` +
`sync database. Grab syncServer, syncDatabase, syncUser, and\n` +
`syncPassword from the API strategy settings (or\n` +
`https://app.amazingmarvin.com/?api) and save them with e.g.\n` +
` marvin config syncServer https://...\n` +
` marvin config syncDatabase ...\n` +
` marvin config syncUser ...\n` +
` marvin config syncPassword ...`,
);
Deno.exit(1);
}

const server = (opt.syncServer as string).replace(/\/+$/, "");
const database = encodeURIComponent(opt.syncDatabase as string);
const baseUrl = `${server}/${database}`;
const authHeader = "Basic " + btoa(`${opt.syncUser}:${opt.syncPassword}`);

let result: { docs: CouchDoc[]; conflictedIds: string[] };
try {
result = await fetchAllDocs(baseUrl, authHeader);
} catch (err) {
console.error(`Backup failed: ${errMessage(err)}`);
Deno.exit(1);
}

const { docs, conflictedIds } = result;
const json = JSON.stringify(docs, null, 2);

// Where to write: -o/--output, defaulting to a dated file in the current
// directory. Use "-o -" to write to stdout instead.
const output = typeof cmdOpt.output === "string" ? cmdOpt.output : null;
const toStdout = output === "-";
const dest = toStdout ? null : (output ?? defaultFilename());

if (toStdout) {
console.log(json);
} else {
try {
Deno.writeTextFileSync(dest as string, json);
} catch (err) {
console.error(`Failed to write backup to "${dest}": ${errMessage(err)}`);
Deno.exit(1);
}
}

// Diagnostics go to stderr so they don't corrupt piped JSON, and respect
// --quiet. The conflict warning (below) is printed regardless.
if (!opt.quiet) {
console.error(
`Backed up ${docs.length} records to ${toStdout ? "stdout" : dest}`,
);
}
reportConflicts(conflictedIds);
Deno.exit(0);
}

// Warn about records in a sync-conflict state, counting them the way Marvin's
// Database Check does (see uuidOf). Printed even with --quiet, since it signals
// a data-integrity problem.
function reportConflicts(conflictedIds: string[]): void {
if (conflictedIds.length === 0) {
return;
}
const groups = new Set(conflictedIds.map(uuidOf)).size;
console.error(
`⚠ ${groups} unresolved sync conflict(s) detected ` +
`(${conflictedIds.length} conflicted record(s)) — ` +
`resolve via Marvin's Database Check`,
);
}

function errMessage(err: unknown): string {
return err instanceof Error ? err.message : String(err);
}

function defaultFilename(): string {
// Match the name the Marvin GUI uses, e.g. MarvinBackup-20260626.json
// (local date, YYYYMMDD).
const d = new Date();
const stamp = `${d.getFullYear()}` +
`${String(d.getMonth() + 1).padStart(2, "0")}` +
`${String(d.getDate()).padStart(2, "0")}`;
return `MarvinBackup-${stamp}.json`;
}

export const backupHelp = `
marvin backup [-o <file>]

Back up your entire Marvin account.

This reads every record directly from your Marvin CouchDB/Cloudant sync
database (the public API has no bulk-export endpoint) and writes them to a
JSON file as an array of records. The format mirrors Marvin's own backup
files (the internal "_rev" field is omitted), so it can be inspected or
re-imported.

If any records are in an unresolved sync-conflict state, a warning is printed
and those records keep their "_rev" field as a marker (just like Marvin's own
backups) — resolve them via Marvin's Database Check.

Note: this is a live read of the database, not a guaranteed point-in-time
snapshot; for the most consistent result, back up when Marvin isn't actively
syncing.

This is equivalent to Marvin's GUI backup with "Device Local Settings/Data"
left unchecked: it includes everything in the sync database, but not data that
isn't synced between your devices (trashed items, which sections are
opened/closed, etc.), which the sync database doesn't hold.

You must first configure your sync credentials (one-time):
marvin config syncServer https://...
marvin config syncDatabase ...
marvin config syncUser ...
marvin config syncPassword ...
Find these in the API strategy settings or at
https://app.amazingmarvin.com/?api.

EXAMPLE:
# Write a timestamped backup file to the current directory
$ marvin backup

# Write to a specific file
$ marvin backup -o my-backup.json

# Write to stdout (e.g. to pipe elsewhere)
$ marvin backup -o -

OPTIONS:
-o, --output <file>
Where to write the backup. Defaults to MarvinBackup-<YYYYMMDD>.json
in the current directory (matching the Marvin GUI's naming). Use "-"
to write to stdout.

--sync-server <url>
--sync-database <name>
--sync-user <user>
--sync-password <password>
Provide sync credentials for this run instead of reading them from
config. Useful for keeping syncPassword out of the plaintext config
file, e.g. --sync-password "$MARVIN_SYNC_PASSWORD".

${globalOptionHelp}
`.trim();
116 changes: 116 additions & 0 deletions src/commands/backup_couch.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
// CouchDB/Cloudant access for the `backup` command. Kept free of any config,
// filesystem, or process I/O so it can be unit-tested with a mocked `fetch`
// (see backup_couch_test.ts).

// deno-lint-ignore no-explicit-any
export type CouchDoc = Record<string, any>;

// How many documents to fetch per request when paginating _all_docs. Keeping
// this bounded avoids huge responses / timeouts on large accounts.
export const BATCH_SIZE = 1000;

// Abort a single page request that hangs, so a dead connection fails fast
// instead of waiting forever.
export const REQUEST_TIMEOUT_MS = 30_000;

// Page through _all_docs?include_docs=true&conflicts=true, returning every
// document (design documents excluded, since they are CouchDB internals rather
// than Marvin data and aren't part of Marvin's own backup format).
//
// To mirror Marvin's GUI backup format we drop the CouchDB-internal `_rev`
// field, which is present on every document over the API but which the GUI
// omits — except on documents whose revision tree is in a conflicted state,
// where the GUI keeps `_rev` as a marker. `conflicts=true` makes CouchDB add a
// `_conflicts` array to exactly those documents, so we use it to decide which
// records keep `_rev` (and we strip the `_conflicts` array itself, which the
// GUI doesn't emit). The `_id`s of conflicted records are returned so the
// caller can warn about them.
//
// `batchSize` is injectable so tests can exercise pagination without thousands
// of fixtures.
export async function fetchAllDocs(
baseUrl: string,
authHeader: string,
batchSize: number = BATCH_SIZE,
): Promise<{ docs: CouchDoc[]; conflictedIds: string[] }> {
const docs: CouchDoc[] = [];
const conflictedIds: string[] = [];
const seen = new Set<string>();
let startkey: string | null = null;

for (;;) {
let url =
`${baseUrl}/_all_docs?include_docs=true&conflicts=true&limit=${batchSize}`;
if (startkey !== null) {
// startkey is inclusive, so skip=1 avoids re-fetching the boundary doc.
url += `&startkey=${encodeURIComponent(JSON.stringify(startkey))}&skip=1`;
}

let res: Response;
try {
res = await fetch(url, {
method: "GET",
headers: { "Authorization": authHeader },
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
});
} catch (err) {
if (err instanceof DOMException && err.name === "TimeoutError") {
throw new Error(`request timed out after ${REQUEST_TIMEOUT_MS / 1000}s`);
}
throw err;
}

if (!res.ok) {
const text = await res.text();
throw new Error(`[${res.status}] ${text || res.statusText}`);
}

const body = await res.json();
const rows: { id: string; doc?: CouchDoc }[] = body.rows ?? [];
if (rows.length === 0) {
break;
}

for (const row of rows) {
// Skip design docs (CouchDB internals) and any id already collected. The
// dedupe guards against a row reappearing across pages if the database
// changes mid-dump — this is a live read, not a point-in-time snapshot.
if (!row.doc || row.id.startsWith("_design/") || seen.has(row.id)) {
continue;
}
seen.add(row.id);
const doc = row.doc;
if (Array.isArray(doc._conflicts) && doc._conflicts.length > 0) {
// Conflicted: keep `_rev` as Marvin's conflict marker, drop the
// `_conflicts` array (which the GUI backup doesn't include).
delete doc._conflicts;
conflictedIds.push(doc._id);
} else {
// Normal record: the GUI backup omits `_rev`.
delete doc._rev;
}
docs.push(doc);
}

// Loop control uses the raw rows (not the deduped docs): a short page means
// we've reached the end, and the next startkey is the last raw id.
if (rows.length < batchSize) {
break;
}
startkey = rows[rows.length - 1].id;
}

return { docs, conflictedIds };
}

// Group key for a conflicted record's _id. Recurring-task occurrences share one
// UUID (the _id is "YYYY-MM-DD_<uuid>"), so a single conflicted recurring task
// appears as several conflicted occurrence records; grouping by the trailing
// UUID yields a count that matches Marvin's Database Check. This mirrors
// marvin-backup-tools' check_integrity.py so the two agree. It's an approximate
// grouping (a non-occurrence id that happened to contain "_" would group oddly),
// but it only affects the warning count, never the backed-up data.
export function uuidOf(id: string): string {
const i = id.indexOf("_");
return i === -1 ? id : id.slice(i + 1);
}
Loading