Repository navigation
feat(cli): back up to a bucket, not just a disk - #375
Conversation
`create`/`list`/`restore` reached into `node:fs` directly, so the only place a snapshot could ever live was the directory the CLI ran in. Move those reads and writes behind a `BackupDestination` — stage/commit/record/list/materialize — with the local directory as its first implementation. No behaviour change: the export still streams NDJSON to a local path, the manifest is still a single `manifest.json` beside the snapshots, and both are byte-for-byte what they were. Verified by running this handler and alpha's side by side over the same inputs and comparing the resulting directories and the import request bodies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
`lunora backup create` is the tier that answers everything the 30-day in-place window cannot, and it wrote NDJSON to `--dir` — so its durability was whatever disk the CLI happened to run on. `--bucket <name>` sends the same snapshot to an R2 bucket in the operator's account instead; `--prefix` picks the key prefix (default `backups/`). The format does not know where it is going. The export still streams NDJSON to a local path and the manifest still records the same fields; the destination decides whether that path is the backup or a staging file. A snapshot written to a bucket is byte-identical to one written to a directory, including a `v.bigint()` and a `v.bytes()` column, which the tests assert by restoring both and decoding what the import endpoint received. Bytes travel through the worker's admin storage routes under the admin bearer every other backup verb already carries: no R2 credentials in the CLI, nothing about the bucket in the manifest. Uploads take the existing checksum-verified `PUT /_lunora/admin/storage`, so a truncated transfer is refused before anything is written, and the manifest entry is recorded only after the object lands. Reads need a new admin route — `GET /_lunora/admin/storage/object`, backed by a `storageDownload` worker option the generated app wires up — so a bucket restore does not depend on a deployment having configured URL signing. The index is a `<key>.manifest.json` sidecar per snapshot, which is the layout the platform's own scheduled backup already writes, so `list --bucket` shows cron-written and hand-taken snapshots as one history rather than two. `restore --verify` re-digests the snapshot and checks it against the SHA-256 recorded at create time before importing a row, for either destination. A snapshot with no recorded checksum fails rather than passing quietly. One invocation reads one destination: a bucket and a directory are never merged into one listing, because provenance is the whole question when you are picking which copy to restore. BREAKING CHANGE: manifest entries gained a `sha256` field, and `uploadStorageBlob` takes an explicit key instead of a key prefix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
|
Thank you for following the naming conventions! 🙏 |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe backup command now supports local-directory and R2 bucket destinations. It adds shared snapshot layouts, checksum verification, scheduled-backup storage, authenticated object downloads, retention handling, size limits, and related platform and CLI documentation. ChangesBackup storage
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant CLIBackupHandler
participant R2Destination
participant StorageAdminRoute
participant StorageDownload
CLIBackupHandler->>R2Destination: Resolve snapshot key
R2Destination->>StorageAdminRoute: Request object by key and bucket
StorageAdminRoute->>StorageDownload: Read key and bucket
StorageDownload-->>StorageAdminRoute: Return object stream and metadata
StorageAdminRoute-->>R2Destination: Stream snapshot
R2Destination-->>CLIBackupHandler: Materialize snapshot
CLIBackupHandler->>CLIBackupHandler: Verify checksum and import snapshot
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The destination took the bucket name twice — once for its own admin-route URLs and once inside the upload context — which is a way for the two to name different buckets. Read it off the context only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Three constants were exported that nothing outside their own module reads. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/cli/src/commands/backup/handler.ts (1)
111-124: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winRelease the staged directory when the export fails.
destination.stage(name)allocates a resource for the R2 destination.createR2Destination.stagecallsmkdtemp(join(tmpdir(), "lunora-backup-")). The only cleanup is thefinallyblock insidecreateR2Destination.commit.When
runExportCommandreturns a non-zero code, line 123 returns beforecommitruns. The temp directory stays inos.tmpdir(), together with any partial NDJSON thatdiscardPartialExportdid not remove. Every failedlunora backup create --bucket ...leaves one more directory behind.Add an explicit release hook to
BackupDestinationand call it on the failure path. The local destination implements it as a no-op, because its staged path is the final path.🧹 Proposed fix
In
packages/cli/src/commands/backup/destination.ts, add the hook to the interface:interface BackupDestination { commit: (file: string, stagedPath: string, digest: SnapshotDigest) => Promise<void>; /** Drop a staged snapshot that will never be committed (a failed export). */ discard: (stagedPath: string) => Promise<void>; readonly label: string; // … }Implement it in
createDirectoryDestinationas a no-op, and increateR2Destinationasasync (stagedPath) => rm(dirname(stagedPath), { force: true, recursive: true }).Then release on the failure path:
if (result.code !== 0) { + await destination.discard(staged); + return { code: result.code }; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/backup/handler.ts` around lines 111 - 124, Add a discard release hook to the BackupDestination interface, implement it as a no-op in createDirectoryDestination and recursively remove the staged temporary directory in createR2Destination. In the backup handler, invoke destination.discard(staged) before returning when runExportCommand produces a non-zero result, ensuring failed exports release their staged resources.
🧹 Nitpick comments (3)
packages/cli/src/commands/backup/handler.ts (3)
134-142: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low valueRecord
digest.bytesinstead ofresult.bytesin the manifest entry.
digest.sha256anddigest.bytesboth describe the bytes on disk.result.bytesis the count the export accumulated while writing.createR2Destination.commitdeclaressize: digest.bytesto the upload route. The manifest therefore records one byte count while the stored object declares another.The two agree in normal operation. Using
digest.bytesmakes the manifest entry internally consistent with thesha256next to it.♻️ Proposed change
const entry: BackupManifestEntry = { - bytes: result.bytes, + bytes: digest.bytes, createdAt: timestamp, file, id: timestamp, rows: result.rows, sha256: digest.sha256, tables: options.tables, };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/backup/handler.ts` around lines 134 - 142, Update the manifest entry construction in the backup handler to assign digest.bytes to the bytes field instead of result.bytes, keeping the recorded byte count consistent with the adjacent digest.sha256 and uploaded object size.
388-393: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the
defaultbucket sentinel to one local constant.The expression
options.bucket === "default" ? undefined : options.bucketappears twice. Both results are load-bearing:createR2Destinationuses the top-levelbucketforbucketQueryon its direct requests, andcontext.bucketfor the shareduploadStorageBlobandlistStorageObjectshelpers. If one copy changes, the CLI reads one bucket and writes another.As per path instructions, "Follow DRY principles".
♻️ Proposed change
+ // `default` is the CLI's spelling for "the deployment's default bucket", + // which the worker addresses by omitting the `bucket` query parameter. + const bucket = options.bucket === "default" ? undefined : options.bucket; + return createR2Destination({ - bucket: options.bucket === "default" ? undefined : options.bucket, - context: { baseUrl, bucket: options.bucket === "default" ? undefined : options.bucket, fetchImpl, token }, + bucket, + context: { baseUrl, bucket, fetchImpl, token }, logger: options.logger, prefix: options.prefix, });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/backup/handler.ts` around lines 388 - 393, In the destination construction returned by the backup handler, compute the normalized bucket value from options.bucket once in a local constant, then reuse that constant for both the top-level bucket property and context.bucket. Preserve the existing "default" sentinel behavior by mapping it to undefined.Source: Path instructions
206-210: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueSkip the manifest listing when the target is not a backup id.
destination.list()runs on every restore. For the R2 destination,listcallslistStorageObjectsand then issues one authenticated GET per manifest sidecar. A restore by explicit object key pays that full cost, then discards the result becausematchedisundefined.Backup ids are ISO-8601 timestamps with
:and.replaced by-, produced at line 106-108. A shape check ontargetlets a path or key restore go straight tomaterializewithout the listing.♻️ Proposed change
+// A backup id is the snapshot's ISO-8601 timestamp. Anything else is a path or +// an object key, so the manifest cannot resolve it — skip the listing entirely. +const BACKUP_ID = /^\d{4}-\d{2}-\d{2}T\d{2}-\d{2}-\d{2}-\d{3}Z$/u; + const runBackupRestore = async (options: BackupCommandOptions, destination: BackupDestination): Promise<BackupCommandResult> => {- const manifest = await destination.list(); - const matched = manifest.find((entry) => entry.id === target); + const matched = BACKUP_ID.test(target) ? (await destination.list()).find((entry) => entry.id === target) : undefined; const snapshot = await destination.materialize(matched?.file ?? target, matched !== undefined);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/backup/handler.ts` around lines 206 - 210, Update the restore target resolution around destination.list() to first detect whether target matches the backup-id format produced near the backup creation logic (ISO-8601 timestamp with colons and periods replaced by hyphens). Only call destination.list() and search for a matching manifest entry for backup-id-shaped targets; pass explicit paths or object keys directly to destination.materialize without listing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/src/content/docs/concepts/backups.mdx`:
- Around line 183-187: Update the bucket restore drill in the backup
documentation to scope --verify to CLI-created snapshots that include a recorded
sha256 checksum. Alternatively, document a separate drill for scheduled
snapshots that omits --verify, while preserving the existing bucket-based
restore guidance.
In `@packages/cli/docs/index.mdx`:
- Around line 605-608: Correct the backup restore command example to use the raw
ISO timestamp with colons and the period, matching the manifest id assigned by
the backup creation flow. Update only the documented restore argument; keep the
bucket and verify options unchanged.
In `@packages/cli/src/commands/backup/destination.ts`:
- Around line 150-154: Update the manifest write flow in the record callback to
write the serialized manifest to a uniquely named temporary file within
directory, then rename that temporary file over MANIFEST_FILE only after the
write succeeds. Preserve the existing manifest read, append, and formatting
behavior while ensuring failed or interrupted writes do not leave manifest.json
truncated.
In `@packages/cli/src/commands/backup/r2-destination.ts`:
- Around line 77-94: Update the commit handler around uploadStorageBlob so
snapshots are not loaded entirely with readFile(stagedPath). Use an available
streaming or multipart upload mechanism while preserving the existing metadata
and cleanup in the finally block; if neither is supported, validate the snapshot
size against an explicit documented limit before reading and reject oversized
snapshots.
- Around line 129-134: Update the download flow around pipeline and the returned
release callback so failures from pipeline remove the temporary directory via rm
before rethrowing the original error. Preserve the existing successful return of
path and release, and ensure cleanup applies to the directory created by
mkdtemp.
- Around line 55-104: Update createR2Destination to create one bucket-scoped
context using options.bucket, then pass that context to readObject,
uploadStorageBlob in commit, and listStorageObjects in list. Ensure every
storage request uses the same resolved bucket while preserving the existing
bucket query and destination behavior.
---
Outside diff comments:
In `@packages/cli/src/commands/backup/handler.ts`:
- Around line 111-124: Add a discard release hook to the BackupDestination
interface, implement it as a no-op in createDirectoryDestination and recursively
remove the staged temporary directory in createR2Destination. In the backup
handler, invoke destination.discard(staged) before returning when
runExportCommand produces a non-zero result, ensuring failed exports release
their staged resources.
---
Nitpick comments:
In `@packages/cli/src/commands/backup/handler.ts`:
- Around line 134-142: Update the manifest entry construction in the backup
handler to assign digest.bytes to the bytes field instead of result.bytes,
keeping the recorded byte count consistent with the adjacent digest.sha256 and
uploaded object size.
- Around line 388-393: In the destination construction returned by the backup
handler, compute the normalized bucket value from options.bucket once in a local
constant, then reuse that constant for both the top-level bucket property and
context.bucket. Preserve the existing "default" sentinel behavior by mapping it
to undefined.
- Around line 206-210: Update the restore target resolution around
destination.list() to first detect whether target matches the backup-id format
produced near the backup creation logic (ISO-8601 timestamp with colons and
periods replaced by hyphens). Only call destination.list() and search for a
matching manifest entry for backup-id-shaped targets; pass explicit paths or
object keys directly to destination.materialize without listing.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e7623b83-cad5-4bd1-8c88-668ed0968d55
⛔ Files ignored due to path filters (5)
api-snapshots/runtime.api.mdis excluded by none and included by nonepackages/cli/__tests__/commands/backup-r2.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/backup.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/codegen/__tests__/fixtures/simple/expected/_generated/app.tsis excluded by!**/_generated/**,!**/__tests__/**and included bypackages/**packages/runtime/__tests__/storage-admin.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (12)
apps/docs/src/content/docs/concepts/backups.mdxpackages/cli/docs/index.mdxpackages/cli/src/commands/backup/destination.tspackages/cli/src/commands/backup/handler.tspackages/cli/src/commands/backup/index.tspackages/cli/src/commands/backup/r2-destination.tspackages/cli/src/commands/data-transfer/sources/storage-transfer.tspackages/cli/src/commands/data-transfer/storage-blobs.tspackages/codegen/src/emit-app.tspackages/runtime/docs/index.mdxpackages/runtime/src/create-worker.tspackages/runtime/src/storage-admin-routes.ts
Merging this PR will degrade performance by 3.93%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | in-batch: single IN(...) query + id->doc re-projection |
787 µs | 945.6 µs | -16.77% |
| ⚡ | baseline (Object.keys + toInternal + path spread per field) |
72.1 µs | 65.1 µs | +10.9% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing feat/313-backup-to-r2 (efde1e2) with alpha (f8a19ae)
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
The scheduled backup wrote no `sha256`, so `lunora backup restore --verify` refused exactly the snapshots nobody watches being taken, while snapshots run by hand verified fine. Digest the NDJSON and hand it to `put()` — R2 validates it on write and reports it afterwards — then record it in the manifest sidecar alongside the fields the CLI already writes. Two comments nine lines apart claimed opposite things about memory, and the code followed the pessimistic one: the body was buffered, not streamed. The claim that it was "never held in worker memory at once" was left over from an abandoned attempt. It is gone, and the real constraint is written down instead: `orchestrateExport` resolves every shard's rows before the first byte is written, so the snapshot is assembled in the isolate no matter what the upload does. A multipart upload would remove one copy of the encoded bytes and not raise that ceiling. So the run now bounds itself: 64 MiB of NDJSON, checked as rows arrive, failing with `BACKUP_TOO_LARGE` and naming the ways out rather than being killed for running out of memory. The rows are encoded once into the exact bytes that get hashed and stored, which is also less memory than the previous string array plus Blob. `R2BucketLike.put` gains the `sha256` option it always accepted — `@lunora/storage` has been passing one all along, and the canonical projection did not say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Three ways a bucket-backed backup could leave the operator worse off than no backup at all: - The manifest was written in place, so a crash or a failed write left an unparseable index — which `readManifest` then refuses to overwrite, turning one bad write into a stuck backup schedule. Write a sibling temp file and rename over it. - A snapshot was read into memory to upload it with no bound, so a large backup died as an out-of-memory crash after a successful export. Refuse above a stated 256 MiB with the workarounds named. The upload route takes one body, so lifting the bound means multipart, not a bigger read. - A failed download left its temp directory (holding whatever arrived) behind, because nothing returns `release` on that path. Adds the cross-tier test that would have caught the missing checksum on the scheduled tier: one bucket, both writers, comparing the manifests they land and restoring a cron-written snapshot through the CLI with a bigint and a bytes column intact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The backup docs listed flags and left the reader to infer the shape of the feature. State it: PITR is in place and covers the last 30 days, snapshots are portable and cover everything older, and the two share nothing. Adds the section this change exists for — restoring from a bucket when the machine that took the backups is gone — plus the id-vs-filename distinction (the restore examples used the file-name form, which `restore` can never match against a manifest id), the size ceilings for both tiers, and a plain statement that a "bucket" on the Node host is an fs-backed directory rather than a separate failure domain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Unformatted on alpha, so the repo-wide prettier gate fails on any branch that runs it. Redundant parentheses around one property access; no behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/cli/src/commands/backup/destination.ts (1)
171-175: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftSerialize manifest read-modify-write operations.
Two
backup create --dirprocesses can both read the same manifest, append different entries, and rename their own result. The last rename loses the other snapshot entry.Use a cross-process lock, or redesign the local index as immutable per-snapshot records. Keep the lock across
readManifest,push, andwriteManifest.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/cli/src/commands/backup/destination.ts` around lines 171 - 175, Serialize the manifest read-modify-write sequence in the record callback by acquiring a cross-process lock before readManifest and holding it through manifest.push and writeManifest, then releasing it reliably afterward. Ensure concurrent backup create --dir processes cannot overwrite each other’s entries.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/runtime/docs/index.mdx`:
- Around line 218-219: Update the documentation text near orchestrateExport to
remove taking snapshots more often as a remedy for reducing snapshot size.
Recommend backupTables or moving the job off-platform for size reduction, and
only mention increased frequency as limiting future data growth if needed.
- Around line 220-221: Update the backup documentation around `lunora backup
create --bucket` and the related `runExportCommand` behavior to remove the
unsupported fixed 256 MiB limit. Describe the limit as higher than the 64 MiB
isolate limit and dependent on the host/container, and recommend streaming
larger snapshots to `--dir` when needed; keep the wording consistent across the
backup documentation.
In `@packages/runtime/src/create-worker.ts`:
- Around line 4190-4229: The scheduled backup currently retains both the chunk
array and the concatenated body, exceeding the intended memory limit. Update the
flow around writeRow, streamExportRows, and concatChunks to avoid simultaneous
full-snapshot representations—prefer streaming encoded chunks directly into the
upload and hashing path—or reduce MAX_SCHEDULED_BACKUP_BYTES to cover all peak
allocations, while preserving size enforcement before any partial object or
manifest is written.
- Around line 4203-4207: Remove the schedule-frequency advice from the
oversized-backup remediation in packages/runtime/src/create-worker.ts lines
4203-4207 by deleting “shorten the interval” while preserving the remaining
guidance. Also update the corresponding remediation hint in
packages/errors/src/catalog.ts lines 193-201 to remove “run the cron more
often”; no other backup-limit messaging should change.
---
Outside diff comments:
In `@packages/cli/src/commands/backup/destination.ts`:
- Around line 171-175: Serialize the manifest read-modify-write sequence in the
record callback by acquiring a cross-process lock before readManifest and
holding it through manifest.push and writeManifest, then releasing it reliably
afterward. Ensure concurrent backup create --dir processes cannot overwrite each
other’s entries.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 19cdea99-fab8-4fd2-a059-1bf5d76e1746
⛔ Files ignored due to path filters (6)
api-snapshots/errors.api.mdis excluded by none and included by noneapi-snapshots/platform.api.mdis excluded by none and included by noneapi-snapshots/runtime.api.mdis excluded by none and included by nonepackages/cli/__tests__/commands/backup-tier-parity.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/scheduled-backup.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (10)
apps/docs/src/content/docs/concepts/backups.mdxpackages/cli/docs/index.mdxpackages/cli/src/commands/backup/destination.tspackages/cli/src/commands/backup/handler.tspackages/cli/src/commands/backup/r2-destination.tspackages/errors/src/catalog.tspackages/platform/src/bindings.tspackages/runtime/docs/index.mdxpackages/runtime/src/create-worker.tspackages/runtime/src/storage-admin-routes.ts
🚧 Files skipped from review as they are similar to previous changes (4)
- packages/runtime/src/storage-admin-routes.ts
- packages/cli/docs/index.mdx
- packages/cli/src/commands/backup/handler.ts
- apps/docs/src/content/docs/concepts/backups.mdx
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## alpha #375 +/- ##
========================================
Coverage 87.09% 87.09%
========================================
Files 1172 1177 +5
Lines 63383 63567 +184
Branches 15447 15759 +312
========================================
+ Hits 55202 55365 +163
- Misses 7654 7680 +26
+ Partials 527 522 -5
🚀 New features to boost your workflow:
|
Both backup writers share a prefix and a sidecar suffix so that one bucket reads as one history. That also made an operator's snapshot indistinguishable from a cron's to `pruneBackups`, which collected every `.manifest.json` under the prefix, sorted, and deleted past `backupRetain`. Two `lunora backup create --bucket` snapshots plus one cron fire with `backupRetain: 1` removed both, no log line on either side — from the bucket the docs recommend, at the moment somebody had just taken a pre-migration copy. Retention now reads each candidate's sidecar and deletes only entries carrying this backup's own `scheduledTime`. An unreadable sidecar counts as somebody else's and is kept. `BackupStore` gains `get` for it; a raw R2 binding already satisfies that. The 24 MiB cap says what it actually does. It was documented in six places as failing "instead of an out-of-memory kill", which it cannot promise: `orchestrateExport` resolves every shard's rows before the first row is encoded, so for shard-local tables the row set is resident before the first check runs — it bounds incrementally only on the `.global()` generator branch. It is now described as a cap on the snapshot, set well below the isolate's limit because it cannot bound the larger allocation, and lowered from 64 MiB accordingly. The test drives it through the streaming branch and asserts the generator is abandoned early, rather than materialising 200 MiB up front and calling that a demonstration. Per-row chunks are released after the contiguous copy is built, which is what the comment claimed already happened. `runScheduledBackup` and `pruneBackups` move to `scheduled-backup.ts` (precedent: `export-stream.ts`), and the layout both writers depend on — prefix, sidecar suffix, key template, manifest shape, the `id` = ISO rule — moves to `backup-layout.ts` and is exported. Five conventions were written twice across two packages, one of them under a comment reading "Must match `@lunora/runtime`'s scheduled backup". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
`create --bucket` refused above 256 MiB, but `uploadStorageBlob` routes past 32
MiB to a signed PUT with no server-side digest — so "checksum-verified" was
false from 32 MiB up, and a 33 MiB snapshot failed with "no signed PUT URL
could be minted" on any app without URL signing configured, which is most of
them. The cap is now the verified route's own, so the claim holds and the
refusal names the size.
Staging has a lifetime. `stage` returns `{ path, release }` and `create` wraps
the run in `try/finally`, so a failed export no longer leaves a temp directory
of production rows behind — cleanup used to live only in `commit`, which a
failed export never reaches. That also removes `commit`'s
`rm(dirname(stagedPath))`, which was safe only because the R2 `stage` happened
to own that directory; the same call against the directory destination would
have deleted the operator's backups.
One damaged sidecar no longer blocks recovery. `restore` reads the index before
it can reach a snapshot named directly by key, and the R2 listing threw on any
unreadable object under the prefix — so an unrelated `.manifest.json` made
`restore <exact key>` fail while the named object was fine. Unreadable sidecars
are skipped with a warning, and the reads are windowed instead of one
unbounded `Promise.all`.
Also: the manifest's `bytes` is the file's own length rather than the export's
wire counter; `--prefix backups` no longer yields `backupslunora-backup-…`;
`materialize` takes the matched entry instead of a boolean the R2 side ignored;
`bucketQuery` is imported rather than hand-rolled a second time; and the
--verify message no longer blames the scheduled backup for a missing checksum
it now records.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The matrix codegen trusts said nothing about a feature that adds an admin route, a `WorkerOptions` entry and a backup destination. `objectStorageBackups` is native on Cloudflare and emulated on the Node host, where the bucket is a directory on the same machine the CLI runs on — so a bucket-backed backup there is not the separate failure domain it is on Cloudflare. That caveat had reached the docs and not the matrix. Regenerates the two committed example workers whose `_generated/app.ts` lacked `storageDownload`; only the golden fixture had been updated, so a bucket restore against either would have answered STORAGE_DOWNLOAD_NOT_CONFIGURED. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The ceilings changed and the reasons differ per tier: 32 MiB for `create --bucket` (the verified upload route's body cap) and 24 MiB for the scheduled backup (a Worker isolate). Both stated, with the reason, in place of the old 256/64 pair. The last-resort recovery procedure told the reader to fetch `lunora-backup-<id>.ndjson` two sections after a callout explaining that the id and the key are not the same string. It now shows the key form, with the list command beside it. Also documents that `list --bucket` reads one object per snapshot, and that a damaged sidecar is skipped rather than fatal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/runtime/src/scheduled-backup.ts`:
- Line 153: Update the prune logic around scheduled backup deletion to delete
each snapshot object before its manifest sidecar, awaiting the snapshot deletion
before starting the sidecar deletion so a partial failure remains retryable.
Also isolate or handle prune errors in runScheduledBackup so a failed prune does
not reject an otherwise successful backup write.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 26fb3f4d-1380-44b6-8f41-ab770d55a42c
⛔ Files ignored due to path filters (9)
api-snapshots/errors.api.mdis excluded by none and included by noneapi-snapshots/lunora.api.mdis excluded by none and included by noneapi-snapshots/platform.api.mdis excluded by none and included by noneapi-snapshots/runtime.api.mdis excluded by none and included by noneexamples/blog/lunora/_generated/app.tsis excluded by!**/_generated/**and included by noneexamples/team-chat/lunora/_generated/app.tsis excluded by!**/_generated/**and included by nonepackages/cli/__tests__/commands/backup-r2.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/backup-tier-parity.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/scheduled-backup.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (13)
apps/docs/src/content/docs/concepts/backups.mdxpackages/cli/docs/index.mdxpackages/cli/src/commands/backup/destination.tspackages/cli/src/commands/backup/handler.tspackages/cli/src/commands/backup/r2-destination.tspackages/cli/src/commands/data-transfer/storage-blobs.tspackages/errors/src/catalog.tspackages/platform/src/capabilities.tspackages/runtime/docs/index.mdxpackages/runtime/src/backup-layout.tspackages/runtime/src/create-worker.tspackages/runtime/src/index.tspackages/runtime/src/scheduled-backup.ts
🚧 Files skipped from review as they are similar to previous changes (8)
- packages/runtime/docs/index.mdx
- apps/docs/src/content/docs/concepts/backups.mdx
- packages/cli/src/commands/data-transfer/storage-blobs.ts
- packages/cli/docs/index.mdx
- packages/cli/src/commands/backup/r2-destination.ts
- packages/cli/src/commands/backup/destination.ts
- packages/cli/src/commands/backup/handler.ts
- packages/errors/src/catalog.ts
…sidecar Retention tested `typeof scheduledTime === "number"` — "written by a cron", where the docblock claimed "written by *this* cron". Two workers on one bucket, `0 3 * * *` and `*/30 * * * *` with `backupRetain: 2`, and the daily run deleted the other's snapshots; both deployments silently got half the retention they configured. It now matches the cron expression, stamped on each sidecar as custom metadata when it is written. That marker also removes the reads. The previous fix issued one `get` per sidecar under the prefix on every run — 40 operator sidecars meant 40 reads and zero deletions, scaling with the bucket rather than the retention window, and R2 calls come out of the Worker subrequest budget. Custom metadata rides on the listing (`include: ["customMetadata"]`), so retention is now one list page per 1000 objects and no per-object requests. A failed prune no longer fails the backup. It runs after both `put`s, so the snapshot and its manifest have already landed; propagating meant an operator saw a failing cron over a backup that was fine, and the actual symptom — retention stopped — was invisible. It is caught and reported instead. Deletes go snapshot first, sidecar second. The reverse order can leave a snapshot with no sidecar, and since the marker retention prunes on lives on the sidecar, such an object is invisible to retention forever. Also: `normalizeBackupPrefix` moves into `backup-layout` and runs inside `backupObjectKey`, so `backupPrefix: "snapshots"` stops producing `snapshotslunora-backup-…` — the CLI had a guard for that and the scheduled writer did not. `isBackupManifestEntry` moves there too, because the side that deletes had no shape guard while the side that only reads did. `toHex` is imported rather than reimplemented, and `BACKUP_MANIFEST_SUFFIX` / `STORAGE_OBJECT_PATH` are no longer exported with no consumer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
A throwing `release()` in the create/restore `finally` turned an exit code non-zero after `logger.success` had already printed the backup — a scripted run would retry a snapshot sitting in the bucket. Both sites warn instead. The 32 MiB cap is imported from the route that enforces it rather than restated; two literals drifting apart is what decides whether an upload silently takes the unverified path, and the docblock already claimed it mirrored the runtime. `StagedSnapshot` is its own type: `stage` returned a `MaterializedSnapshot`, whose doc says "a snapshot made readable as a local file", for a file that does not exist yet. `commit`'s doc no longer promises a throw the directory implementation cannot make. `normalizePrefix` and `isManifestEntry` are gone in favour of the shared ones, so the guard and the prefix rule have one definition each, and the directory destination computes its layout through `locate`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
"Passing `createStorage(...).download` satisfies it" was the only remediation an operator got when a bucket restore failed, and it does not compile — `download`'s second parameter is a byte range, not a bucket, so the two signatures share no properties and the assignment is a TS2322. The sentence was copied from the `storageDelete`/`storageList` variants, where it is correct. All three sites now show what the generated worker actually emits. Also: retention's scope stated (only this cron's snapshots, others left alone), the 24 MiB cap no longer reads as "the cap is what makes it fit", the size paragraph no longer sits under a heading covering the uncapped `--dir`, and `list` is described consistently as one read per snapshot in both places. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/runtime/src/storage-admin-routes.ts (2)
212-245: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winReject unknown bucket names before downloading.
handleStorageObjectpasses thebucketquery parameter straight tostorageDownload, and the generated selector falls back todefaultfor an unknown non-empty bucket. An invalidrestore --bucket <typo>can therefore read a snapshot from the default bucket. Validate the requested bucket before downloading, or make the injected selector fail closed for invalid names.Add object-download coverage for default, valid named, and invalid named buckets.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/storage-admin-routes.ts` around lines 212 - 245, Update handleStorageObject to validate the optional bucket query parameter against the configured bucket names before calling storageDownload, rejecting unknown non-empty names instead of allowing fallback to the default bucket. Preserve default-bucket behavior when omitted and valid named-bucket behavior, and add coverage for default, valid named, and invalid named object downloads.
212-245: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winPrevent caching of backup downloads.
handleStorageObjectreturns backup bytes under admin authorization, but the handler and outer response decoration do not setcache-control: no-store. Add it before returning the response.Proposed header
return new Response(object.body, { headers: { + "cache-control": "no-store", "content-type": object.httpMetadata?.contentType ?? "application/octet-stream",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/runtime/src/storage-admin-routes.ts` around lines 212 - 245, Add a Cache-Control header with a no-store directive to the Response returned by handleStorageObject, preserving the existing content type, content length, and status headers.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/backup/destination.ts`:
- Around line 195-196: Update the staging implementation around stage and its
returned release/commit handlers so failed or incomplete exports remove the
local staging file from directory. Stage through a temporary path and rename it
only during commit, or make release conditionally delete the staged path until
commit succeeds; preserve committed files and ensure cleanup targets only
uncommitted staging artifacts.
In `@packages/runtime/src/backup-layout.ts`:
- Around line 78-88: Keep isBackupManifestEntry limited to validating manifest
shape, and add filesystem path-containment validation in the local restore flow
around the destination path construction in destination.ts. Resolve the manifest
file against the backup directory and reject any result outside that directory
before reading or importing it; keep this separate from shared R2 object-key
validation, and ensure the TypeScript types remain valid.
---
Outside diff comments:
In `@packages/runtime/src/storage-admin-routes.ts`:
- Around line 212-245: Update handleStorageObject to validate the optional
bucket query parameter against the configured bucket names before calling
storageDownload, rejecting unknown non-empty names instead of allowing fallback
to the default bucket. Preserve default-bucket behavior when omitted and valid
named-bucket behavior, and add coverage for default, valid named, and invalid
named object downloads.
- Around line 212-245: Add a Cache-Control header with a no-store directive to
the Response returned by handleStorageObject, preserving the existing content
type, content length, and status headers.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 67737ee3-fb4c-4a43-b84e-d4084cf09a08
⛔ Files ignored due to path filters (6)
api-snapshots/errors.api.mdis excluded by none and included by noneapi-snapshots/lunora.api.mdis excluded by none and included by noneapi-snapshots/runtime.api.mdis excluded by none and included by nonepackages/cli/__tests__/commands/backup-tier-parity.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/cli/__tests__/commands/backup.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/scheduled-backup.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (12)
apps/docs/src/content/docs/concepts/backups.mdxpackages/cli/src/commands/backup/destination.tspackages/cli/src/commands/backup/handler.tspackages/cli/src/commands/backup/r2-destination.tspackages/cli/src/commands/data-transfer/storage-blobs.tspackages/errors/src/catalog.tspackages/runtime/docs/index.mdxpackages/runtime/src/backup-layout.tspackages/runtime/src/create-worker.tspackages/runtime/src/index.tspackages/runtime/src/scheduled-backup.tspackages/runtime/src/storage-admin-routes.ts
🚧 Files skipped from review as they are similar to previous changes (8)
- packages/errors/src/catalog.ts
- packages/runtime/docs/index.mdx
- packages/cli/src/commands/backup/r2-destination.ts
- apps/docs/src/content/docs/concepts/backups.mdx
- packages/runtime/src/scheduled-backup.ts
- packages/cli/src/commands/data-transfer/storage-blobs.ts
- packages/cli/src/commands/backup/handler.ts
- packages/runtime/src/create-worker.ts
Phases 0-1 and WS5 shipped in #375. WS4 did not: retention still deletes inside every cron run, narrowed by a cron-expression marker rather than made explicit, which is not what §4.4 promised. That gap is also where the branch's worst defect came from - unifying the key layout is what makes list --bucket one history and what made the two writers indistinguishable to the existing prune. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
* docs(plans): file 313, move the long-term backup tier to R2 lunora backup already has both tiers and names them: pitr covers the last 30 days in place, and create/restore is the portable one for everything older. The second writes through node:fs to a local dir, so the tier that exists for the case the 30-day window cannot cover depends on whichever disk the CLI ran on, automated by an external cron. The snapshot format, manifest, export/import endpoints and R2 bindings all exist, so this is a destination rather than a redesign. Acceptance is a bigint and bytes round-trip, which is what plan 265 showed the export path could silently lose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP * docs(plans): record 313's outcome and its one deviation Phases 0-1 and WS5 shipped in #375. WS4 did not: retention still deletes inside every cron run, narrowed by a cron-expression marker rather than made explicit, which is not what §4.4 promised. That gap is also where the branch's worst defect came from - unifying the key layout is what makes list --bucket one history and what made the two writers indistinguishable to the existing prune. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP * docs(plans): mark 313 shipped in the index Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`materialize` joined the backup directory with a manifest entry's `file` and read whatever came out, so an entry of `file: "../../outside.ndjson"` — or an absolute path — made `restore <id>` read and import a file from anywhere on disk. The entry had passed `isBackupManifestEntry`, which knows `file` is a string and nothing more: a shape guard is not a safety guard, and that predicate is shared with retention, which compares object keys where path semantics do not apply. Containment lives at the join instead. The candidate is resolved and rejected unless it stays under the resolved directory, compared with a separator so `/backups-evil` cannot pass for `/backups`. A target the operator typed themselves is untouched — naming any file they can read is what that form is for. Tested with a relative escape, an absolute path, and an escape wearing a plausible-looking name. The directory destination also staged straight to the final path, so a failed export, digest, commit or record left a partial `.ndjson` with no manifest entry — invisible to `list`, accumulating silently. It is the same leak fixed on the R2 side last round, on the sibling path. Staging now writes a `.partial` file in the same directory and `commit` renames it into place, which is atomic and also makes the directory `commit` mean what the interface says instead of being a no-op that happens to be safe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/cli/src/commands/backup/destination.ts`:
- Around line 174-182: Update resolveInsideDirectory to check the candidate
exists, resolve both directory and candidate through realpath, then enforce
containment using canonical root-or-child comparisons while correctly handling a
filesystem root without adding a duplicate separator. Preserve the existing
LunoraError rejection for entries whose symlink-resolved target lies outside the
backup directory.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8088dc79-0d2d-440f-9748-0d34c2b0b278
⛔ Files ignored due to path filters (1)
packages/cli/__tests__/commands/backup.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (2)
packages/cli/src/commands/backup/destination.tspackages/cli/src/commands/backup/r2-destination.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/cli/src/commands/backup/r2-destination.ts
`resolve()` is text manipulation — it makes no filesystem query at all — so the
containment guard closed `../` and absolute paths and left a symlink inside the
backup directory pointing out of it wide open. Same trust boundary as the
finding it fixed: a tampered backup directory.
`realpath` on both sides now does the check the filesystem can answer. Both,
not just the candidate: a backup directory is itself often reached through a
symlink — every macOS temp directory is — so canonicalizing one side rejects
ordinary paths. Three existing restore tests plus a new positive control fail
if only the candidate is resolved.
The lexical check stays in front of it, because it catches a text-level escape
whether or not the file exists, and a manifest pointing outside the directory
is worth naming either way. Order is lexical, existence, canonical: `realpath`
throws on a missing path, and a missing snapshot is ordinary — "not found" and
"escaped" must not collapse into one answer.
`isInside` compares with a separator, and does not append one when the root
already ends in it, which is what a directory at the filesystem root looks
like: `resolve("/")` is `/`, and `//` matches nothing.
Verified the symlink test fails against the lexical-only guard, so it proves
the link is followed rather than passing for an unrelated reason.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The local file a downloaded snapshot lands in was named by `key.split("/")`,
and `key` comes off a `.manifest.json` in the bucket — data. Splitting on the
POSIX separator leaves `..\..\evil.txt` intact as a single "segment", which is
harmless on POSIX and walks out of the temp directory under `path.win32.join`:
`C:\evil.txt`, written with the operator's privileges and attacker-controlled
bytes, and left behind because `release()` only removes the `mkdtemp`
directory. The comment above it asserted basename semantics that `split("/")`
provides on one platform only.
`basename` is separator-correct per platform. It can still return `.`, `..` or
`""` — a key ending in a separator, or one that is only a separator — each of
which resolves to the directory or its parent rather than a file in it, so
those become a fixed name.
The rule is a pure function taking the `path` API it uses, and the test drives
it through `path.win32` and `path.posix` explicitly: CI is ubuntu-only and
`@lunora/cli` declares no OS restriction, so nothing else would ever exercise
the platform this bug is on. Verified the previous implementation fails it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The same guard was hand-rolled in four places — a Convex snapshot's blob ids,
the upload walk, the doctor's shadowed-CLI check, and a backup manifest's
`file` — three of them security guards, each carrying its own copy of the two
subtleties: compare with a separator (`/backups-evil` is not inside
`/backups`), and do not append one when the root already ends in it (a
directory at the filesystem root, where `//` matches nothing).
Hoisted to `util/path-containment`, with the reasoning that was scattered
across four comments written once. Deliberately not `path.relative(...)
.startsWith("..")`, which false-negatives on a legitimately named `..foo`
directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The `BackupDestination` docblock still described the pre-`.partial` behaviour — `release` "a no-op where the staged file is the backup", `commit` "nothing to do" — for the second time on this branch. It is the only thing an author of a third destination reads, and as written it permitted exactly the partial-snapshot leak that was fixed. Rewritten as what a caller may rely on, so it stops going stale each time an implementation changes. `MaterializedSnapshot` and `StagedSnapshot` were structurally identical while one's doc claimed it was "deliberately not" the other; TypeScript is structural, so that was a boundary the compiler never had. One `SnapshotFile`. The `STORAGE_DOWNLOAD_NOT_CONFIGURED` hint compiled but discarded the bucket, so a hand-wired worker would read `--bucket archive` from the default bucket. It now forwards `opts.bucket` the way the generated worker does — a wrong answer that compiles is worse than the one that did not. The sibling-prefix test never tested a sibling prefix: the root is `.lunora-backups`, so `../lunora-backups-evil` failed on the leading dot, not the separator. One character, and the rule the guard exists for is now exercised. Also: `resolve()` described accurately (it consults the working directory and never the filesystem); root-side canonicalisation justified generally rather than by macOS temp directories alone; `materialize` no longer re-checks existence the entry branch already did; four exports with no importer dropped, one of them (`backupObjectKeyOfManifest`) a public API commitment in two snapshots bought for a caller one module away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Both halves shipped - the object-storage destination in #375, and in #389 the part that made the plan's own decision true: the scheduled backup no longer deletes, prune is the only thing that removes a backup, and retention previews what it would take. The row keeps what git alone would not tell you. Six review rounds, and every defect after the first was in the remediation rather than the original work, all of one shape - a claim about a second party's behaviour never checked against the second party. Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
What
lunora backup create | list | restorecan now keep snapshots in an R2 bucketinstead of a directory on whatever machine ran the CLI.
Scope
Plan 313 phases 0–1 (WS1–3), plus WS5: the platform's scheduled backup
already existed on
alpha(backupCron/backupStore), and putting the twowriters in one bucket turned up real defects in it, so it is changed here too.
Read the
@lunora/runtimecommits as WS5 work rather than incidental.Retention/prune (WS4) is not implemented here — but the existing prune had to be
made safe, see below.
The shape of it
One encoding, two destinations. The export streams NDJSON to a local path;
the destination decides whether that path is the backup or a staging file on its
way up. Neither destination has snapshot code of its own.
One layout, two writers. Key template, default prefix, sidecar suffix,
manifest shape and the
id= ISO rule live in@lunora/runtime'sbackup-layoutand are imported by both the CLI and the cron. They were writtentwice across two packages, one copy under a comment reading "Must match
@lunora/runtime's scheduled backup" — a comment doing a type's job, and theplan's §3 STOP condition.
No new auth surface. Bytes travel through the worker's admin storage routes
under the admin bearer the backup verbs already carry. Uploads take the
checksum-verified
PUT /_lunora/admin/storage; reads needed one new route,GET /_lunora/admin/storage/object, so a restore does not depend on URL signinga deployment may never have configured.
restore --verifyre-digests the snapshot and checks it against the SHA-256recorded at create time, for both destinations and both tiers.
pitris untouched. Restore stays explicit, and a bucket and a directory arenever merged into one listing.
What sharing a bucket cost, and how it is paid
Unifying the layout is what makes
list --bucketone history. It also made anoperator's snapshot indistinguishable from a cron's to the existing
pruneBackups, which deleted every.manifest.jsonpastbackupRetain— sotwo
backup create --bucketsnapshots plus one cron fire withbackupRetain: 1removed both, silently. Retention now reads each candidate's sidecar and deletes
only entries carrying its own
scheduledTime; an unreadable sidecar counts assomebody else's and is kept. A test pins that a cron fire cannot delete a
CLI-written snapshot.
The scheduled tier
restore --verifyrefused precisely thesnapshots nobody watches being taken. It now digests the NDJSON, hands it to
put()(R2 validates on write), and records it in the sidecar.memory; it was. The replacement states the real constraint —
orchestrateExportresolves every shard's rows before the first row isencoded — so the cap is described as a bound on the snapshot, not on peak
memory, and set well below the isolate's limit (24 MiB) because it cannot
bound the larger allocation.
create --bucketcaps at 32 MiB, theverified-upload route's own limit, so "checksum-verified" is true at every
size it accepts.
R2BucketLike.putgained thesha256option it always accepted.runScheduledBackup+pruneBackupsmoved toscheduled-backup.ts(precedent:
export-stream.ts).Verification
by side over the same inputs, comparing backup directories and import request
bodies; confirmed non-vacuous by perturbing the manifest indent.
v.bigint()+v.bytes()through all three writers — directory, bucket,and the platform's cron — decoded from what the import endpoint received.
backup-tier-parity.test.ts): both writers into onebucket, asserting the cron manifest carries every field the CLI's does, that
the
idvalues are ISO timestamps and the keys derive from them via theshared helper, that each digest matches its object, and that snapshot and
manifest land in the same bucket.
backupRetain: 1prunes its own oldersnapshot and leaves an operator's untouched.
generator is abandoned early — not by materialising a large array up front.
--verifywith nothing imported; a failed export leaves no staging directory.Platform parity
objectStorageBackupsis now in the matrix:nativeon Cloudflare,emulatedonplatform-node, where the bucket is a directory on the samemachine — so it is not the separate failure domain it is on Cloudflare. The two
committed example workers whose
_generated/app.tslackedstorageDownloadareregenerated.
Also here
packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.tswas unformatted onalpha, failing the repo-wide prettier gate on any branch that runs it.Formatting only.
🤖 Generated with Claude Code
https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Summary by CodeRabbit
New Features
Bug Fixes
Documentation