feat: add purgeRuns API and retainRuns auto-cleanup option#109
Merged
Conversation
- `durably.purgeRuns({ olderThan, limit })` for manual batch deletion of
terminal runs (completed, failed, cancelled) with cascading cleanup of
steps, logs, and labels
- `retainRuns: '30d'` option on createDurably for automatic periodic purge
(runs once per 60s during worker polling, batch size 100)
- Migration v2: adds (status, completed_at) index for efficient purge queries
- Duration parser supports 'd' (days), 'h' (hours), 'm' (minutes)
- 11 new tests covering all purge scenarios
Closes #88
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📝 WalkthroughWalkthrough自動・手動で端末状態のランを削除する機能を追加： Changes
Sequence Diagram(s)sequenceDiagram
participant App as アプリケーション
participant Durably as Durably インスタンス
participant Poll as ポーリング／ライフサイクル
participant Store as ストレージ層
participant DB as データベース
App->>Durably: processOne() 呼び出し
Durably->>Poll: 処理ループ開始
Poll->>Poll: lastPurgeAt + PURGE_INTERVAL_MS をチェック
alt パージ間隔を超過
Poll->>Durably: 非同期で purge をトリガー
Durably->>Store: purgeRuns({ olderThan, limit })
Store->>DB: SELECT ids WHERE status IN TERMINAL_STATUSES AND completed_at < cutoff
DB-->>Store: 対象 run ID を返す
Store->>DB: cascadeDeleteRuns(ids) — labels, steps, logs, runs を削除
DB-->>Store: 削除完了
Store-->>Durably: 削除件数を返す
Durably->>Poll: lastPurgeAt を更新
else パージ不要
Poll-->>Durably: 通常処理継続
end
Poll-->>App: processOne() 完了
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Extract cascadeDeleteRuns() helper to eliminate duplication between deleteRun() and purgeRuns() - Extract TERMINAL_STATUSES constant to module scope - Move PURGE_INTERVAL_MS to module scope (was recreated per processOne call) - Make auto-purge fire-and-forget (void) so it doesn't block job claiming or lease renewal in the worker polling loop Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
No backward compatibility needed — merge the (status, completed_at) index into the single v1 migration instead of keeping a separate v2. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move TERMINAL_STATUSES after RunStatus type definition (ordering) - Replace void fire-and-forget with .catch() to prevent unhandled rejection - Use single Date.now() variable in purge block instead of calling 3 times - Add comment documenting intentional immediate purge on first cycle Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…hot path The claimNext SQL already contains an activeLeaseGuard subquery that checks NOT EXISTS for active leases with the same concurrency key. The JS-side getRuns → filter → excludeConcurrencyKeys flow was redundant and added a full table scan + JOIN + JSON parse per poll cycle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nal string Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
packages/durably/tests/shared/purge.shared.ts (1)
208-235: This never exercises the positive auto-purge path.The test only proves that a freshly completed run survives 200ms. It would still pass if the retention scheduler never called
purgeRuns(). Please add an actually expired run here—via fake timers or by backdatingcompleted_at—and assert that polling removes it.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/durably/tests/shared/purge.shared.ts` around lines 208 - 235, Add a positive auto-purge assertion by creating a run, marking it completed and making it appear older than the 1m retention, then asserting the worker polling actually removes it: e.g., after triggering the run with d.jobs.testJob.trigger and waiting for completion via d.getRun, either (A) use vitest fake timers (vi.useFakeTimers()/vi.setSystemTime( Date.now() + 2*60*1000 )) before letting the poller run and then assert d.getRun(run.id) is null, or (B) backdate the run row directly (via d.db or the test DB helper) to set completed_at = new Date(Date.now() - 2*60*1000) then start the poller (d.start()), wait a polling interval, and expect await d.getRun(run.id) to be null; ensure you restore timers/cleanup (vi.useRealTimers()/d.stop()/d.db.destroy()) and keep the existing assertions that freshly completed runs are not purged when not expired.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/durably/docs/llms.md`:
- Line 39: Clarify that the retainRuns TTL is only enforced during active worker
polling by stating that cleanup is triggered from processOne() while the worker
polls (at most once per minute) and runs purge in batches of up to 100 terminal
runs; make this explicit next to the retainRuns example and in the section
covering lines ~233-246 so users know no background cleanup occurs when no
worker is running.
In `@packages/durably/src/durably.ts`:
- Around line 917-933: The retention purge is currently invoked before
storage.claimNext and therefore gets enqueued into the same write mutex ahead of
the claim, stalling the hot path; fix by moving the purge logic out of the
pre-claim path — call storage.claimNext(workerId, now, state.leaseMs) first,
then (if state.retainRunsMs !== null and Date.now() - state.lastPurgeAt >=
PURGE_INTERVAL_MS) update state.lastPurgeAt and trigger storage.purgeRuns({
olderThan: cutoff, limit: 100 }) asynchronously (e.g. void setImmediate(() =>
storage.purgeRuns(...).catch(() => {})) or setTimeout(...,0)) so the purge runs
fire-and-forget without serializing with claimNext.
In `@packages/durably/src/migrations.ts`:
- Around line 83-88: The migration adds an index but it's bundled into migration
1 while LATEST_SCHEMA_VERSION remains 1, so existing databases will skip it;
update the migrations by moving the createIndex call out of the existing
migration-1 block into a new migration function/entry (e.g., create a migration
for version 2) and increment LATEST_SCHEMA_VERSION accordingly so the new index
(idx_durably_runs_status_completed) runs for already-upgraded DBs; ensure the
new migration includes the same
.createIndex(...).ifNotExists().on('durably_runs').columns(['status','completed_at']).execute()
logic and any related rollback or idempotency safeguards.
---
Nitpick comments:
In `@packages/durably/tests/shared/purge.shared.ts`:
- Around line 208-235: Add a positive auto-purge assertion by creating a run,
marking it completed and making it appear older than the 1m retention, then
asserting the worker polling actually removes it: e.g., after triggering the run
with d.jobs.testJob.trigger and waiting for completion via d.getRun, either (A)
use vitest fake timers (vi.useFakeTimers()/vi.setSystemTime( Date.now() +
2*60*1000 )) before letting the poller run and then assert d.getRun(run.id) is
null, or (B) backdate the run row directly (via d.db or the test DB helper) to
set completed_at = new Date(Date.now() - 2*60*1000) then start the poller
(d.start()), wait a polling interval, and expect await d.getRun(run.id) to be
null; ensure you restore timers/cleanup
(vi.useRealTimers()/d.stop()/d.db.destroy()) and keep the existing assertions
that freshly completed runs are not purged when not expired.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 19ea6a51-7f8e-4842-ac58-30ff86e2dd7f
📒 Files selected for processing (9)
packages/durably/docs/llms.mdpackages/durably/src/durably.tspackages/durably/src/migrations.tspackages/durably/src/storage.tspackages/durably/tests/node/migration-consolidated.test.tspackages/durably/tests/node/purge.test.tspackages/durably/tests/shared/purge.shared.tspackages/durably/tests/shared/storage.shared.tswebsite/public/llms.txt
💤 Files with no reviewable changes (1)
- packages/durably/tests/shared/storage.shared.ts
Comment on lines
+83
to
+88
| await db.schema | ||
| .createIndex('idx_durably_runs_status_completed') | ||
| .ifNotExists() | ||
| .on('durably_runs') | ||
| .columns(['status', 'completed_at']) | ||
| .execute() |
Contributor
There was a problem hiding this comment.
Don't fold new DDL into migration 1.
Line 12 still leaves LATEST_SCHEMA_VERSION at 1, so databases that already recorded version 1 will skip this block forever. Existing installs won't get idx_durably_runs_status_completed, which means the new purge query loses its supporting index.
💡 Safer migration shape
-export const LATEST_SCHEMA_VERSION = 1
+export const LATEST_SCHEMA_VERSION = 2
const migrations: Migration[] = [
{
version: 1,
up: async (db) => {
- await db.schema
- .createIndex('idx_durably_runs_status_completed')
- .ifNotExists()
- .on('durably_runs')
- .columns(['status', 'completed_at'])
- .execute()
},
},
+ {
+ version: 2,
+ up: async (db) => {
+ await db.schema
+ .createIndex('idx_durably_runs_status_completed')
+ .ifNotExists()
+ .on('durably_runs')
+ .columns(['status', 'completed_at'])
+ .execute()
+ },
+ },
]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| await db.schema | |
| .createIndex('idx_durably_runs_status_completed') | |
| .ifNotExists() | |
| .on('durably_runs') | |
| .columns(['status', 'completed_at']) | |
| .execute() | |
| export const LATEST_SCHEMA_VERSION = 2 | |
| const migrations: Migration[] = [ | |
| { | |
| version: 1, | |
| up: async (db) => { | |
| // ... existing v1 migration logic ... | |
| }, | |
| }, | |
| { | |
| version: 2, | |
| up: async (db) => { | |
| await db.schema | |
| .createIndex('idx_durably_runs_status_completed') | |
| .ifNotExists() | |
| .on('durably_runs') | |
| .columns(['status', 'completed_at']) | |
| .execute() | |
| }, | |
| }, | |
| ] |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/durably/src/migrations.ts` around lines 83 - 88, The migration adds
an index but it's bundled into migration 1 while LATEST_SCHEMA_VERSION remains
1, so existing databases will skip it; update the migrations by moving the
createIndex call out of the existing migration-1 block into a new migration
function/entry (e.g., create a migration for version 2) and increment
LATEST_SCHEMA_VERSION accordingly so the new index
(idx_durably_runs_status_completed) runs for already-upgraded DBs; ensure the
new migration includes the same
.createIndex(...).ifNotExists().on('durably_runs').columns(['status','completed_at']).execute()
logic and any related rollback or idempotency safeguards.
Owner
Author
There was a problem hiding this comment.
Intentionally kept in v1. This library has no production databases yet — we explicitly consolidated all migrations into v1 earlier in this PR to avoid carrying dead migration versions. The ifNotExists guard makes it safe for any edge case.
Contributor
There was a problem hiding this comment.
@coji, understood — thanks for the context. A clean consolidation into v1 while the library is still pre-production is a perfectly reasonable approach, and the ifNotExists guard ensures idempotency for any edge cases. I'll stand down on this one.
✏️ Learnings added
Learnt from: coji
Repo: coji/durably PR: 109
File: packages/durably/src/migrations.ts:83-88
Timestamp: 2026-03-16T07:56:24.962Z
Learning: In `packages/durably/src/migrations.ts`, all schema migrations (including new indexes like `idx_durably_runs_status_completed`) are intentionally consolidated into migration version 1. The library has no production databases yet, so there is no need to carry separate incremental migration versions. `LATEST_SCHEMA_VERSION` is kept at 1 by design. The `ifNotExists` guard on DDL statements is used as the idempotency safeguard. Do not flag folding new DDL into migration v1 as an issue for this repository.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
🧠 Learnings used
Learnt from: CR
Repo: coji/durably PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-03-12T13:21:21.863Z
Learning: Applies to **/*.{js,ts} : Use the dialect injection pattern - pass a Kysely dialect to `createDurably()` to abstract SQLite implementations
Learnt from: coji
Repo: coji/durably PR: 101
File: packages/durably/tests/node/db-semantics.postgres.test.ts:4-4
Timestamp: 2026-03-08T05:03:43.723Z
Learning: In `packages/durably/tests/node/db-semantics.postgres.test.ts`, `createPostgresSchemaResource()` is intentionally called with parentheses. It returns an object `{ createDialect, setup, cleanup }` (a SharedDialectResource), not a plain dialect factory function. The `createDbSemanticsTests` harness in `packages/durably/tests/shared/db-semantics.shared.ts` (around line 18) handles both patterns: a bare factory function OR a pre-invoked resource object with `createDialect`, optional `setup`, and optional `cleanup` methods.
Learnt from: CR
Repo: coji/durably PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-03-12T13:21:21.863Z
Learning: Applies to **/*.{js,ts,mjs} : Use ESM-only syntax; CommonJS is not supported. Always use top-level `await` for async initialization (e.g., `await durably.migrate()`). Do not wrap in async IIFE or Promise chains.
Learnt from: coji
Repo: coji/durably PR: 101
File: docs/rfcs/runtime-rearchitecture/ja/core-runtime.md:0-0
Timestamp: 2026-03-08T05:06:39.050Z
Learning: Reviewers should verify that step execution uses step.run() and that step output data is persisted. By default, preserveSteps is false (step history is deleted when runs reach a terminal state). If audit/debug history is required, ensure preserveSteps is set to true. Be aware that cleanupSteps used to exist with inverted semantics (cleanupSteps: true meant delete), and this was renamed to preserveSteps in Phase 1. Apply this guidance to all JS/TS code and docs that reference step persistence.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
website/api/create-durably.md (1)
157-157: Clarify cutoff applies to completedAt timestamp.The inline comment says "cutoff date" but doesn't specify which timestamp field is compared. For clarity, consider mentioning that it compares against
completedAt.📝 Suggested clarification
- olderThan: Date // cutoff date — runs completed before this are deleted + olderThan: Date // cutoff date — runs with completedAt before this are deleted🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@website/api/create-durably.md` at line 157, Update the inline comment on the olderThan field to state that the cutoff is compared against the run's completedAt timestamp; e.g., change the comment for the olderThan: Date field so it explicitly reads that runs with completedAt before this date will be deleted, referencing the olderThan property and the completedAt timestamp.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@website/api/create-durably.md`:
- Line 162: Update the endpoint description to clarify that it purges all
terminal runs (not just those with status 'completed'): change wording that
currently says "runs completed before this" to explicitly state "terminal runs
(completed, failed, cancelled) with completedAt older than the cutoff are
deleted", and mention that deletion uses the run.status and completedAt fields
so readers know which runs are targeted.
- Line 31: Update the docs for the retainRuns option to state that
createDurably() validates the retainRuns string synchronously and will throw
immediately (before returning the instance) for invalid formats; explicitly
document the accepted pattern (\d+(d|h|m)) and add a short note below the
options table (or inline with the retainRuns description) indicating that
invalid values cause an eager/throwing validation when createDurably() is
called.
In `@website/api/index.md`:
- Line 221: The docs show purgeRuns(options?) as optional but the actual
implementation requires options; update the API docs to match the implementation
by removing the optional marker so the signature reads purgeRuns(options) and
ensure any surrounding text or examples reflect that options is required; locate
the purgeRuns declaration/implementation (the purgeRuns function in Durably /
durably.ts) to mirror its parameter requirements when editing the markdown
entry.
---
Nitpick comments:
In `@website/api/create-durably.md`:
- Line 157: Update the inline comment on the olderThan field to state that the
cutoff is compared against the run's completedAt timestamp; e.g., change the
comment for the olderThan: Date field so it explicitly reads that runs with
completedAt before this date will be deleted, referencing the olderThan property
and the completedAt timestamp.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 34a1ebdc-3e28-426a-aee6-89286dc32eef
📒 Files selected for processing (3)
CLAUDE.mdwebsite/api/create-durably.mdwebsite/api/index.md
- Move auto-purge after claimNext so it never serializes with job claiming - Add positive auto-purge test with backdated completed_at - Clarify retainRuns runs only during worker polling in docs - Fix purgeRuns(options?) → purgeRuns(options) in API quick reference - Clarify olderThan matches completedAt, not just "completed" status Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Owner
Author
|
Re: nitpick about the auto-purge test only covering the negative path — fixed in 85a366c. Added a new test |
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/durably/src/durably.ts (1)
1073-1074:lastPurgeAtはメモリ内のみで保持される点について。
lastPurgeAtがデータベースに永続化されないため、ワーカー再起動時に最初のアイドルサイクルで即座にpurgeが実行されます。これは許容可能な動作ですが、複数ワーカーインスタンスが同時に起動した場合、各インスタンスが独立してpurgeを試みる可能性があります。現在の実装では、
storage.purgeRunsがトランザクション内で実行されるため、データ整合性は保たれます。高頻度のワーカー再起動がない限り、この動作は問題ありません。🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/durably/src/durably.ts` around lines 1073 - 1074, 現在の実装は lastPurgeAt をメモリのみで保持しているためワーカー再起動時に即座に purge が走る可能性がある点を修正してください：最後にパージした時刻を永続化するか（例：データベースの小さなキー/テーブルに保存・読み出しし、storage.purgeRuns 実行後に更新する）、起動時に lastPurgeAt を現在時刻に初期化して最初のアイドルサイクルでの即時実行を防ぐかいずれかを選んで実装し、retainRunsMs と比較するロジック（lastPurgeAt の読み書き）を durably の初期化周りで確実に使うようにしてください（参照シンボル: lastPurgeAt, retainRunsMs, storage.purgeRuns）。
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@packages/durably/src/durably.ts`:
- Around line 1073-1074: 現在の実装は lastPurgeAt をメモリのみで保持しているためワーカー再起動時に即座に purge
が走る可能性がある点を修正してください：最後にパージした時刻を永続化するか（例：データベースの小さなキー/テーブルに保存・読み出しし、storage.purgeRuns
実行後に更新する）、起動時に lastPurgeAt
を現在時刻に初期化して最初のアイドルサイクルでの即時実行を防ぐかいずれかを選んで実装し、retainRunsMs と比較するロジック（lastPurgeAt
の読み書き）を durably の初期化周りで確実に使うようにしてください（参照シンボル: lastPurgeAt, retainRunsMs,
storage.purgeRuns）。
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ae3cc121-ef83-454b-a068-3fd48a436fc8
📒 Files selected for processing (6)
packages/durably/docs/llms.mdpackages/durably/src/durably.tspackages/durably/tests/shared/purge.shared.tswebsite/api/create-durably.mdwebsite/api/index.mdwebsite/public/llms.txt
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/durably/tests/shared/purge.shared.ts
- website/api/create-durably.md
- website/public/llms.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
durably.purgeRuns({ olderThan, limit })— manual batch deletion of terminal runs (completed/failed/cancelled) with cascading cleanup of steps, logs, and labelsretainRuns: '30d'option oncreateDurably()— automatic periodic purge during worker polling (every 60s, batch size 100)(status, completed_at)index for efficient purge queriesd(days),h(hours),m(minutes)Details
Phase 1: Manual purge API
Phase 2: Automatic retention
Auto-purge runs at most once per 60 seconds during the worker polling loop with a batch limit of 100, to avoid interfering with normal job processing.
Test plan
Closes #88
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
ドキュメント
Migrations
テスト