Repository navigation
feat(platform-node): complete the Node platform host - #328
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR adds a Node deploy driver, neutral resource projection, durable SQLite-backed Node services, workflow and object-storage adapters, platform composition updates, conformance support, package exports, and shard namespace compatibility. ChangesNode platform support
Estimated code review effort: 5 (Critical) | ~90 minutes Sequence Diagram(s)sequenceDiagram
participant NodePlatform
participant NodeSchedulerHost
participant SQLite
participant onDispatch
NodePlatform->>NodeSchedulerHost: create with shared database
NodeSchedulerHost->>SQLite: persist and re-arm jobs
NodeSchedulerHost->>onDispatch: dispatch scheduled job
onDispatch-->>NodeSchedulerHost: report success or failure
NodeSchedulerHost->>SQLite: delete, retry, or dead-letter job
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
0d7064e to
672fed3
Compare
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (2)
packages/platform-node/src/conformance/index.ts (1)
88-90: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThe cleanup order contradicts the order
createNodePlatform.close()uses.The comment states that this routes through the same disposers as
createNodePlatform'sclose(). The disposers match, but the order is inverted.close()inpackages/platform-node/src/node-platform.tsdisposes the scheduler first and documents why:disposeShard()closes the connection, and a job timer that fires in the window would find it closed. HeredisposeShard()runs first.The guards in
deliverkeep this benign today. Align the order so the guard stays the backstop rather than the mechanism.♻️ Proposed fix
- disposeShard(); disposeScheduler(); + disposeShard(); registry.close();🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/platform-node/src/conformance/index.ts` around lines 88 - 90, Update the cleanup sequence in the conformance teardown to match createNodePlatform.close(): call disposeScheduler() before disposeShard(), then close registry as currently done. Keep the existing disposers and guard behavior unchanged.packages/platform-node/src/node-global-store.ts (1)
71-82: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winCache prepared statements instead of re-preparing on every call.
database.prepare(statement)compiles a new statement object on eachall/runcall. This exec wraps every read and write that the.global()table core issues throughSqlCtxExec, so the same SQL text gets recompiled repeatedly on a hot path. Cache the prepared statement per SQL string and reuse it.♻️ Proposed statement cache
+const statementCache = new WeakMap<Database.Database, Map<string, Database.Statement>>(); + +const preparedStatement = (database: Database.Database, statement: string): Database.Statement => { + let cache = statementCache.get(database); + + if (cache === undefined) { + cache = new Map(); + statementCache.set(database, cache); + } + + let prepared = cache.get(statement); + + if (prepared === undefined) { + prepared = database.prepare(statement); + cache.set(statement, prepared); + } + + return prepared; +}; + export const createNodeSqlExec = (database: Database.Database): SqlCtxExec => { return { // eslint-disable-next-line `@typescript-eslint/require-await` -- the exec seam is async so a networked engine can await; better-sqlite3 is synchronous - all: async (statement, parameters) => database.prepare(statement).all(...(parameters as unknown[])) as Record<string, unknown>[], + all: async (statement, parameters) => preparedStatement(database, statement).all(...(parameters as unknown[])) as Record<string, unknown>[], // eslint-disable-next-line `@typescript-eslint/require-await` -- see `all` run: async (statement, parameters) => { - const result = database.prepare(statement).run(...(parameters as unknown[])); + const result = preparedStatement(database, statement).run(...(parameters as unknown[])); return { rowsAffected: result.changes }; }, }; };Based on path instructions, "Check for performance optimizations" applies to this path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/platform-node/src/node-global-store.ts` around lines 71 - 82, Update createNodeSqlExec to cache prepared statements by SQL text and reuse the cached statement in both all and run instead of calling database.prepare on every invocation. Keep parameter binding and the existing result shapes unchanged.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/platform-node/package.json`:
- Around line 63-67: Align the cron-parser declaration in the platform-node and
scheduler package manifests: either use the established range format
consistently, or, if pinning is intentional, update the scheduler manifest to
the same exact 5.6.2 version. Preserve the existing dependency rule across both
package.json files.
In `@packages/platform-node/src/index.ts`:
- Around line 20-25: The capability note in the introductory comment of the Node
platform incorrectly lists cross-shard fan-out and the query coordinator as
missing. Update that paragraph to reflect the implemented emulated
crossShardFanout capability and in-process query coordinator, while keeping the
genuinely unsupported capabilities unchanged.
In `@packages/platform-node/src/node-platform.ts`:
- Around line 86-95: Extend NodePlatformOptions with the registry option type,
then update createNodePlatform to pass the platform options (or the relevant
registry options) into createNodeShardRegistry. Preserve the existing shard, KV,
socket, and scheduler construction while ensuring path and onAlarm reach the
registry.
In `@packages/platform-node/src/node-scheduler-host.ts`:
- Around line 306-316: Both post-dispatch re-arm paths must honor the terminal
closed state. In packages/platform-node/src/node-scheduler-host.ts lines
306-316, update the failure guard around parkJob and armJob to stop when closed
or the connection is not open; in lines 357-359, require the host to be both not
closed and open before scheduling the next cron tick.
- Around line 412-432: Update deadLetter.requeue to return false immediately
when closed or !isOpen(), before invoking requeueJob or armJob. Preserve the
existing requeue behavior for open connections.
- Around line 441-457: Update the job ID generation used by schedule and related
identity operations so newly created IDs cannot collide with persisted
_lunora_scheduler_jobs rows after restart; replace the restart-local nextId()
strategy with collision-free IDs, using the existing node:crypto randomUUID
import approach, while preserving consistent IDs for armJob, cancel, list, and
deadLetter.requeue.
In `@packages/platform-node/src/node-shard-host.ts`:
- Around line 154-159: Update the alarm scheduling logic in arm within
NodeShardHost to avoid passing delays above 2^31 - 1 ms to setTimeout: when the
remaining delay exceeds that limit, schedule only a maximum-sized chunk and
re-arm until the target timestamp is reached, preserving the existing alarm
callback behavior. Apply the same chunked-delay approach to armJob in
NodeSchedulerHost, using a named function or the project’s existing
no-use-before-define allowance for recursive re-arming.
In `@packages/platform-node/src/node-shard-registry.ts`:
- Around line 61-73: Update encodeShardKey to percent-encode each
non-whitelisted character using its UTF-8 byte sequence, emitting one uppercase
%XX escape per byte instead of the character’s Unicode code point. Preserve the
existing readable whitelist and decodeShardKey behavior so all shard keys,
including non-ASCII values such as café, round-trip through decodeURIComponent.
---
Nitpick comments:
In `@packages/platform-node/src/conformance/index.ts`:
- Around line 88-90: Update the cleanup sequence in the conformance teardown to
match createNodePlatform.close(): call disposeScheduler() before disposeShard(),
then close registry as currently done. Keep the existing disposers and guard
behavior unchanged.
In `@packages/platform-node/src/node-global-store.ts`:
- Around line 71-82: Update createNodeSqlExec to cache prepared statements by
SQL text and reuse the cached statement in both all and run instead of calling
database.prepare on every invocation. Keep parameter binding and the existing
result shapes unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0f96a391-ec79-4b5e-bf57-9d7b70e6b168
⛔ Files ignored due to path filters (14)
api-snapshots/runtime.api.mdis excluded by none and included by nonepackages/codegen/__tests__/platform-target.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/config/__tests__/driver-registry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/config/__tests__/node-driver.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/cross-shard-fanout.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/engine-conformance.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/global-store.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/lifecycle.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/shard-do-integration.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/shard-registry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/shard-state.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/create-worker.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/resolve-shard.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**pnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by none
📒 Files selected for processing (19)
packages/config/src/cloudflare/cloudflare-driver.tspackages/config/src/driver-registry.tspackages/config/src/node/node-driver.tspackages/config/src/resource-graph.tspackages/platform-node/.releaserc.jsonpackages/platform-node/package.jsonpackages/platform-node/src/conformance/index.tspackages/platform-node/src/index.tspackages/platform-node/src/node-global-store.tspackages/platform-node/src/node-platform.tspackages/platform-node/src/node-scheduler-host.tspackages/platform-node/src/node-shard-directory.tspackages/platform-node/src/node-shard-host.tspackages/platform-node/src/node-shard-registry.tspackages/platform-node/src/node-shard-state.tspackages/platform-node/src/node-socket-host.tspackages/platform/src/capabilities.tspackages/runtime/src/create-worker.tspackages/runtime/src/resolve-shard.ts
💤 Files with no reviewable changes (1)
- packages/platform-node/src/node-shard-directory.ts
Merging this PR will regress 2 benchmarks
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | 4 shards |
1.4 ms | 1.6 ms | -10.69% |
| ❌ | flat 3 primitives (the notify.send attribute shape) |
55.5 µs | 61.8 µs | -10.1% |
| ⚡ | single row — 10 columns |
137.8 µs | 111.3 µs | +23.77% |
| ⚡ | + resolveIdentity (userId + claims → x-lunora-identity) |
2 ms | 1.7 ms | +19.87% |
| ⚡ | baseline (Object.keys + toInternal + path spread per field) |
71.9 µs | 65.2 µs | +10.28% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing feat/platform-node-host (765f997) with alpha (e32cb26)2
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(2d29a80) during the generation of this report, so e32cb26 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/platform-node/src/node-r2-bucket.ts`:
- Around line 373-386: The object publication flow around the write path must
atomically expose bytes and metadata as one recoverable version. Replace the
direct rename followed by writeMeta with a per-key versioned object-directory
scheme and atomically update a pointer only after both are written; update
get/head to resolve the published version, and make delete use the same
publication mechanism so interrupted writes or deletes never pair a new body
with stale NodeObjectMeta.
- Around line 196-218: Update packages/platform-node/src/node-r2-bucket.ts at
lines 196-218 to hash and write ReadableStream upload chunks directly to a
temporary file instead of collecting them in memory. At lines 316-326, expose
body through a file range stream so range reads do not materialize the full
object. At lines 368-376, publish the staged file only after the streaming hash
completes; preserve existing validation and upload behavior.
- Around line 59-84: Update the key-to-filesystem-path handling around
validateKey and the R2 bucket path helpers to use a reversible encoding for each
key segment before joining it with the bucket directory. Ensure backslashes, "."
and ".." segments, empty segments, and equivalent-looking keys remain distinct
and cannot escape the bucket; do not rely on host-specific separators or path
normalization, and use the same mapping consistently for reads, writes, deletes,
and listings.
In `@packages/platform-node/src/node-workflow-host.ts`:
- Line 227: Update createNodeWorkflowHost around the options.store
initialization to prevent an implicit MemoryStore fallback. Require callers to
provide a durable WorkflowStore, or replace the fallback with an established
durable Node storage implementation before returning a restart-capable host.
- Around line 297-299: Update the terminate callback in the workflow host so it
preserves a terminated tombstone or lifecycle state instead of deleting the run
record. Ensure subsequent status() calls for the same instance return the
binding’s terminal terminated status, while invalid instance IDs remain
distinguishable.
- Around line 307-316: The WorkflowBindingLike create implementation currently
ignores createOptions.id, so caller-selected IDs are not preserved. Update the
create path around runtime.trigger and instanceFor to pass the supplied ID when
supported; when the runtime cannot honor it, reject create({ id }) with a
NOT_IMPLEMENTED error instead of triggering a different run. Ensure createBatch
preserves its existing behavior unless its contract also accepts caller-supplied
IDs.
In `@packages/platform/src/capabilities.ts`:
- Around line 137-146: Clarify the emulation claim in the capability overview
comment: explicitly limit “the two” to Cloudflare product bindings if that is
intended, while acknowledging the other emulated capabilities such as
globalTables, websocketHibernation, shardAlarms, crossShardFanout, and
scheduler. Keep the capability ratings and gateAgainstMatrix reference
unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3a6d8e79-ffa4-4d7a-8796-a6224315b884
⛔ Files ignored due to path filters (5)
packages/platform-node/__tests__/node-r2-bucket.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/node-workflow-host.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/234-node-host-findings.mdis excluded by none and included by nonepnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by nonepnpm-workspace.yamlis excluded by none and included by none
📒 Files selected for processing (6)
packages/config/src/node/node-driver.tspackages/platform-node/package.jsonpackages/platform-node/src/index.tspackages/platform-node/src/node-r2-bucket.tspackages/platform-node/src/node-workflow-host.tspackages/platform/src/capabilities.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/platform-node/src/index.ts
- packages/platform-node/package.json
419d2ee to
a95ef90
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/platform-node/src/node-workflow-host.ts`:
- Around line 89-108: Update toMs to reject non-finite results for both numeric
durations and parsed amount × unit values by validating with Number.isFinite and
throwing the existing VALIDATION_ERROR for invalid inputs. Also validate the
delay computed by sleepUntil before passing it to context.sleep, so invalid Date
values produce the same validation error rather than propagating NaN.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1ab6e7c2-2fe7-43d9-8122-78186f2dfa69
⛔ Files ignored due to path filters (19)
api-snapshots/runtime.api.mdis excluded by none and included by nonepackages/codegen/__tests__/platform-target.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/config/__tests__/driver-registry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/config/__tests__/node-driver.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/cross-shard-fanout.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/engine-conformance.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/global-store.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/lifecycle.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/node-r2-bucket.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/node-workflow-host.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/shard-do-integration.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/shard-registry.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/shard-state.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/create-worker.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/runtime/__tests__/resolve-shard.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/234-node-host-findings.mdis excluded by none and included by noneplans/234-node-host-lint-fix.mdis excluded by none and included by nonepnpm-lock.yamlis excluded by!**/pnpm-lock.yamland included by nonepnpm-workspace.yamlis excluded by none and included by none
📒 Files selected for processing (21)
packages/config/src/cloudflare/cloudflare-driver.tspackages/config/src/driver-registry.tspackages/config/src/node/node-driver.tspackages/config/src/resource-graph.tspackages/platform-node/.releaserc.jsonpackages/platform-node/package.jsonpackages/platform-node/src/conformance/index.tspackages/platform-node/src/index.tspackages/platform-node/src/node-global-store.tspackages/platform-node/src/node-platform.tspackages/platform-node/src/node-r2-bucket.tspackages/platform-node/src/node-scheduler-host.tspackages/platform-node/src/node-shard-directory.tspackages/platform-node/src/node-shard-host.tspackages/platform-node/src/node-shard-registry.tspackages/platform-node/src/node-shard-state.tspackages/platform-node/src/node-socket-host.tspackages/platform-node/src/node-workflow-host.tspackages/platform/src/capabilities.tspackages/runtime/src/create-worker.tspackages/runtime/src/resolve-shard.ts
💤 Files with no reviewable changes (1)
- packages/platform-node/src/node-shard-directory.ts
🚧 Files skipped from review as they are similar to previous changes (17)
- packages/platform-node/.releaserc.json
- packages/config/src/resource-graph.ts
- packages/platform-node/src/node-global-store.ts
- packages/config/src/node/node-driver.ts
- packages/runtime/src/resolve-shard.ts
- packages/runtime/src/create-worker.ts
- packages/config/src/driver-registry.ts
- packages/platform-node/src/index.ts
- packages/config/src/cloudflare/cloudflare-driver.ts
- packages/platform-node/src/node-r2-bucket.ts
- packages/platform/src/capabilities.ts
- packages/platform-node/src/node-shard-registry.ts
- packages/platform-node/src/node-socket-host.ts
- packages/platform-node/src/node-shard-host.ts
- packages/platform-node/src/node-scheduler-host.ts
- packages/platform-node/src/node-platform.ts
- packages/platform-node/src/conformance/index.ts
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (4)
packages/platform-node/src/node-workflow-store.ts (2)
40-50: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConfirm the connection pragmas for the multi-process claim.
The header states that two processes can share one database file and that
acquirekeeps the claim race-free. Cross-process safety also needs the connection to run in WAL mode with a non-zerobusy_timeout. Without them a concurrent writer fails immediately withSQLITE_BUSYinstead of waiting. This store receives an already-openDatabase, so the pragmas belong to the caller. Document that requirement here, or apply the pragmas where the shared connection is created.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/platform-node/src/node-workflow-store.ts` around lines 40 - 50, Document the shared-connection requirements for the workflow store near its database initialization: callers must configure WAL mode and a non-zero busy_timeout before using the already-open Database. If the connection creation path is available, apply those pragmas there instead, while preserving the race-free acquire behavior across processes.
89-92: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winRun the two deletes in one transaction.
deleteissues two statements. A failure between them leaves an orphan lease row.better-sqlite3providesdatabase.transaction()for exactly this case, and the combined write is also faster than two implicit transactions.♻️ Proposed fix
+ const deleteRun = database.transaction((runId: string) => { + deleteRunStatement.run(runId); + deleteLeaseStatement.run(runId); + }); @@ delete: async (runId) => { - deleteRunStatement.run(runId); - deleteLeaseStatement.run(runId); + deleteRun(runId); },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/platform-node/src/node-workflow-store.ts` around lines 89 - 92, Update the delete implementation around the delete method to execute deleteRunStatement and deleteLeaseStatement within a single better-sqlite3 database.transaction callback, preserving the existing runId parameter and statement order.packages/platform-node/src/node-r2-bucket.ts (1)
496-508: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winFlush the staged file and clean it up if
renamefails.The module states that an interrupted
putleaves the previous version intact. Two gaps weaken that claim:
- The handle closes without
handle.sync(). After a crash the rename can be durable while the body bytes are not, which publishes a truncated object with a trailer that no longer matches.renameruns outside thetryblock. If it fails, for example when the key path collides with an existing directory, the staged file stays in.lunora-tmpforever.♻️ Proposed fix
await handle.write(encodeTrailer(meta)); + await handle.sync(); } finally { await handle.close(); } - } catch (error: unknown) { - await unlinkIfPresent(temporaryPath); - - throw error; - } - // The one operation that publishes the object — body and metadata are - // in this file together, so there is no window where they disagree. - await rename(temporaryPath, filePath); + // The one operation that publishes the object — body and metadata are + // in this file together, so there is no window where they disagree. + await rename(temporaryPath, filePath); + } catch (error: unknown) { + await unlinkIfPresent(temporaryPath); + + throw error; + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/platform-node/src/node-r2-bucket.ts` around lines 496 - 508, Update the staged-write flow around handle.write, handle.close, and rename to call handle.sync() before closing so all body and trailer bytes are flushed before publication. Include the rename operation in the cleanup try/catch so any rename failure calls unlinkIfPresent(temporaryPath) before rethrowing, preserving the previous object and preventing orphaned staged files.packages/platform-node/src/node-workflow-host.ts (1)
268-289: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueResolve each workflow once instead of validating twice.
The first loop validates every entry and fills
idByExport. The map then repeatsisWorkflowDefinitionand re-readsoptions.workflows[exportName], and theas stringcasts exist only becauseidByExport.getcannot prove presence. Collect validated{ exportName, definition, id }records in the first pass and iterate them afterwards. This removes the duplicate guard and both casts.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/platform-node/src/node-workflow-host.ts` around lines 268 - 289, Update the workflow-processing logic around idByExport and visulimaWorkflows to resolve each entry once: in the first pass, validate the value and collect its exportName, definition, and computed id in a typed record. Iterate those records when building visulimaWorkflows, removing the second isWorkflowDefinition check, re-read from options.workflows, idByExport lookup, and related casts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/platform-node/src/node-r2-bucket.ts`:
- Around line 326-333: Update walkObjects so the TMP_DIR entry is skipped only
when traversing the bucket root, matching validateKey’s first-segment
restriction; nested entries named TMP_DIR must remain listable and continue
through the relativePath traversal.
- Around line 155-166: Update readTrailer to treat malformed or invalid trailer
metadata as no object: guard JSON.parse and metadata validation so non-JSON
bytes or parsed values missing sha256Hex return undefined instead of throwing.
Preserve the existing valid trailer result and bodySize behavior, allowing get,
head, and list to skip undecodable files.
- Around line 443-456: Update the cursor construction in the listing method
around the scanned `page` and filtered `objects` so it uses the last scanned key
from `page`, not `objects.at(-1)?.key`. Preserve the existing cursor behavior
for empty pages while ensuring truncated pages always advance past all scanned
keys, including entries whose trailer reads fail.
In `@packages/platform-node/src/node-workflow-host.ts`:
- Around line 325-331: Update the `restart` callback’s `LunoraError` message to
replace the misspelled engine name “visulama” with “visulima,” matching the
naming used elsewhere in the surrounding workflow host code.
- Around line 363-374: Update the terminate handler to cancel or abort the
active runtime before writing the tombstone through store.save, ensuring the run
cannot resume or overwrite its terminated state and its lease is released. If
the hosting engine exposes no cancellation API, make terminate return a
not-implemented result instead of persisting the tombstone.
---
Nitpick comments:
In `@packages/platform-node/src/node-r2-bucket.ts`:
- Around line 496-508: Update the staged-write flow around handle.write,
handle.close, and rename to call handle.sync() before closing so all body and
trailer bytes are flushed before publication. Include the rename operation in
the cleanup try/catch so any rename failure calls unlinkIfPresent(temporaryPath)
before rethrowing, preserving the previous object and preventing orphaned staged
files.
In `@packages/platform-node/src/node-workflow-host.ts`:
- Around line 268-289: Update the workflow-processing logic around idByExport
and visulimaWorkflows to resolve each entry once: in the first pass, validate
the value and collect its exportName, definition, and computed id in a typed
record. Iterate those records when building visulimaWorkflows, removing the
second isWorkflowDefinition check, re-read from options.workflows, idByExport
lookup, and related casts.
In `@packages/platform-node/src/node-workflow-store.ts`:
- Around line 40-50: Document the shared-connection requirements for the
workflow store near its database initialization: callers must configure WAL mode
and a non-zero busy_timeout before using the already-open Database. If the
connection creation path is available, apply those pragmas there instead, while
preserving the race-free acquire behavior across processes.
- Around line 89-92: Update the delete implementation around the delete method
to execute deleteRunStatement and deleteLeaseStatement within a single
better-sqlite3 database.transaction callback, preserving the existing runId
parameter and statement order.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 926daf31-afb0-4bcf-bea9-1a7c813fa681
⛔ Files ignored due to path filters (4)
packages/platform-node/__tests__/node-r2-bucket.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform-node/__tests__/node-workflow-host.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/platform/__tests__/contracts.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/234-node-host-findings.mdis excluded by none and included by none
📒 Files selected for processing (5)
packages/platform-node/src/index.tspackages/platform-node/src/node-r2-bucket.tspackages/platform-node/src/node-workflow-host.tspackages/platform-node/src/node-workflow-store.tspackages/platform/src/capabilities.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/platform/src/capabilities.ts
2205947 to
b26045b
Compare
e403e4b to
3028a40
Compare
Every `QueryCoordinator` fan-out entry point — `fanOut` and the whole `orchestrate*` family — took a `ShardNamespaceLike` requiring `idFromName` and `get`. `@lunora/platform`'s `ShardDirectory` spells the first `idForName`, and makes the second optional because a name-only registry has no id to materialize from. The two shapes were otherwise identical, so a fully conforming directory was rejected over one method name. Nothing caught it because the only namespace ever passed was a real `DurableObjectNamespace`. It is the coupling the platform-parity rule describes — a feature reaching past the contracts into a provider API — and it surfaced the first time a second host tried to fan out. Both members are now optional, `idForName` is accepted alongside `idFromName`, and the two call sites that reached for `idFromName` directly go through `resolveShard`. A namespace supplying neither spelling, or neither `getByName` nor `get`, throws where it is resolved rather than at the first `undefined is not a function` inside a fan-out leg. Cloudflare bindings satisfy the widened shape unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Takes `@lunora/platform-node` from the plan-234 spike to a host that implements what it declares, and lifts the `private` flag the parent commit used to keep the spike off npm. Durability is the through-line. The spike persisted state and never restored it: alarms wrote a timestamp nothing re-armed, scheduler jobs lived in in-process Maps, socket attachments and tags did too. Each contract now owns both halves, and construction is the wake — alarms and jobs re-arm from their rows, an elapsed one fires late rather than never, and socket state comes back from `_lunora_sockets`. Two restart tests pin what the TCK's simulated recycle cannot reach. The directory used to resolve a shard key to a stub whose `fetch` echoed the key: it passed both TCK directory legs with no shard behind it. `createNodeShardRegistry` resolves *and* hosts — one key, one database, one `ShardHost`, one KV store, one socket registry — and seeds its fan-out key set from the shard files on disk, without which a fan-out after a restart returns an empty result indistinguishable from "no rows matched". Cross-shard fan-out is proven against the real query coordinator over three databases, including a failing shard degrading to a partial answer. Adds `.global()` tables over the `@lunora/sql-store` core, a `@lunora/config` deploy driver so `--target node` resolves, and `createNodeShardState`, which presents a shard in the `DurableObjectState` shape `ShardDO` expects — `ShardDO` builds the Cloudflare adapters from `state` rather than accepting the contracts a host already implements, so adapting back is ~100 lines that delete themselves once that seam moves, against 8,700 that would diverge if forked. `NODE_CAPABILITIES` is rewritten to match, and rates almost nothing `native` on purpose: the matrix defines that as "the platform provides it", and a bare Node process provides none of this. Fan-out, `.global()` tables, alarms, scheduler and sockets are `emulated` with notes saying how well; every Cloudflare product binding stays `unsupported`. Still absent, and stated as such: a dev server, so there is no `lunora dev --target node` yet. `plans/234-node-host-findings.md` carries the full record, including the three findings this construction surfaced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RED ON PURPOSE — do not push until the fix lands. Every existing suite proves a layer: the TCK proves host primitives, `engine-conformance` proves the engine, `shard-state` proves the two Cloudflare adapters accept a Node state. Nothing constructed the class that actually serves a request, and this repo holds four separate SQLite-on-Node implementations that "mirror" `ShardDO` without being it — so every suite could agree with itself while the real path was broken. It was. `createNodeShardState.acceptWebSocket` discards the handle and the attachment, so a socket carries no subscriptions, no identity and no admin flag; every read returns `undefined` behind an optional chain and nothing throws. Two legs, both against a real `ShardDO`: - construction passes, which is load-bearing — the constructor eagerly calls `createShardHost(state)` and `createSocketHost(state)`, so a structurally insufficient state fails here rather than on first request. - the attachment round-trip fails, asserted on the payload rather than a count, through `createSocketHost(state).accept(...)` — the exact call the upgrade path makes. A count assertion is what let this ship green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- forward registry options (path/onAlarm) from createNodePlatform - honor the closed flag in scheduler re-arm and deadLetter.requeue - mint collision-free job ids with randomUUID instead of a counter - chunk setTimeout delays above 2^31-1 in shard arm and scheduler armJob - encode shard keys as escaped UTF-8 bytes so decodeURIComponent round-trips - align the conformance teardown dispose order with createNodePlatform.close - cache prepared statements in createNodeSqlExec - drop the stale 'still missing' list from the package doc comment - route the SocketHost attachment stamp into the wrapper handle so acceptWebSocket no longer discards it; fixes the ShardDO integration probe
- Add createNodeWorkflowHost: visulima-backed workflow engine with bindings/env/runtime, step adapter (do/sleep/waitForEvent), status mapping (completed→complete, failed→errored) - Add createNodeR2Bucket: fs-backed R2BucketLike implementation (put/get/head/delete/list with range support, checksums, cursor paging) - Flip workflows + objectStorage to emulated in NODE_CAPABILITIES - Trim UNSUPPORTED in node-driver.ts (queues + containers remain) - Add 20 tests for both adapters (10 each) - Update findings doc with gaps and next steps Known gaps: pause/restart not implemented, ctx.run fails without HTTP server, MemoryStore not durable across restarts.
- Fix jsdoc indentation and HTML escaping in node-r2-bucket.ts - Move exports to end of file (exports-last) - Extract nested ternary in toObject() into getUploadedDate() helper - Rename abbreviated variables (dir → currentPath, rel → relativePath) - Replace await-in-loop with Promise.all pattern - Add localeCompare to sort for deterministic ordering - Replace bitwise >> with Math.floor - Use undefined instead of null where applicable (R2 contract) - Fix require-await in arrayBuffer/text methods - Use optional chaining for prefix check - Add expect.hasAssertions() to all tests - Fix no-await-expression-member in test files - Fix no-promise-executor-return in sleep helper All 92 tests pass, lint clean, types clean.
R2 objects were bytes in `directory/key` plus a metadata sidecar under
`.lunora-meta/`. Two files, and no pair of filesystem operations publishes
both at once: a crash between the body rename and the sidecar write left the
new bytes carrying the previous checksum, size and content-type, which
`get`/`head` then reported as fact. Objects are now one file with the
metadata in a trailer, so the single rename that publishes the bytes
publishes their metadata with them.
The same rewrite removes the buffering. `put` streams into the staged file
while hashing, `get` streams the requested range straight off it, and a range
read no longer loads the whole object to slice it.
`validateKey` only split on `/`, so on Windows `..\outside` reached
`join(directory, key)` and resolved outside the bucket, and `a/./b`, `a//b`
and `a/b` all mapped to the same file. Every segment must now be a plain
name. `walkObjects` also joined key segments with the platform separator,
which would have listed `a/b` as `a\b`.
Workflows:
- `store` is required. The default was the engine's in-process `MemoryStore`,
so a caller who said nothing got a host that lost every run on restart.
`createNodeWorkflowStore` is a SQLite `WorkflowStore` with an atomic
single-statement lease, pinned by a test that resumes a waiting run from a
second host over the same file.
- `terminate` writes a tombstone instead of deleting the run, so `status()`
reports `terminated` rather than the `unknown` an invalid instance id also
returns.
- `create({ id })` throws `NOT_IMPLEMENTED`. It was accepted and dropped, so a
retried create fanned out into distinct runs and `get(id)` found none of them.
- Non-finite sleep durations are rejected. `NaN`, `Infinity` and an overflowing
parse reached `context.sleep` as an unsatisfiable wake-at and parked the run
in `waiting` with nothing to explain it; `sleepUntil` did the same for an
invalid `Date`.
The Node capability spot-check asserted `objectStorage` was `unsupported`
while this package emulates it — red on the branch. It now names a feature
nothing is building towards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hosts
Two of these are defects the previous commit introduced.
**`get()` was not atomic against a concurrent `put`.** `readTrailer` opened the
file, read the trailer and closed the handle; the body read then reopened the
path. An overwrite in between served the new version's bytes — trailer JSON
included — under the old version's size and checksum, which is exactly the
tearing the single-file layout exists to prevent. `get` now holds one handle
from the trailer read through the body read, so the `rename` that publishes a
new version cannot change what an in-flight read sees.
That handle also has to be released. `body` was built eagerly, so every `get`
of an object over 64 KiB leaked a descriptor unless the caller drained the
stream — and the `@lunora/storage` path calls `text()`/`arrayBuffer()` and never
touches `body`. `body` is now a lazy getter, and any of the three reads closes
the handle; measured, the leak on those paths is gone. The object is read once
as a result, which is how R2's own body behaves.
**`list()` could report `truncated: true` with no cursor.** The cursor came from
the last surviving object rather than the last key scanned, so a page whose
entries all dropped out stranded a `while (truncated)` caller, and a partial
drop rewound over keys already served.
The rest:
- `create({ id })` is honoured through a durable alias row instead of throwing.
Refusing it was wrong in a way the earlier review missed: `ctx.spawn` mints a
child id, calls `create({ id })`, then `get(id)`, so the refusal took fan-out
from degraded to dead. Aliasing gives the contract what it actually asks for —
one run per id, findable by it.
- `terminate` no longer fabricates a tombstone for an id that was never a run,
and clears the run's lease instead of leaving it held until expiry. It is
still not a barrier against an in-flight activation; that is now stated.
- `resume` on a terminated run raises instead of leaking the engine's
"no workflow registered" error for the tombstone's sentinel id.
- A corrupt or truncated trailer reads as absent, like every other malformed
file. It threw out of `JSON.parse` before, which took down the whole `list()`.
- `validateKey` shares `hasControlChar` with `@lunora/storage` and applies the
same 1024-byte ceiling, so key grammar no longer diverges per host. It also
reserves the staging directory at every depth — `a/.lunora-tmp/b` was
writable and readable but invisible to `list()`.
- A key colliding with a prefix of itself raises a `LunoraError` naming the
limitation rather than an `EISDIR`/`EEXIST` from an `fs` call the caller did
not make. Non-finite range and limit values are clamped rather than reaching
`Buffer.alloc`.
- One `toArrayBuffer`, shared with the socket host, instead of two same-named
copies in one package. Three loops over the workflow entries collapse into
one, which deletes the id `Map`, both `as string` casts and a duplicate guard.
Tests: multi-chunk streaming (1 MiB, with ranges spanning a chunk boundary and
abutting the trailer), the mid-read overwrite, corrupt/truncated objects, cursor
advance past dropped entries, `ctx.spawn` end to end, and the store's
`acquire`/`release`/`due`/`delete` — including two connections to one file
contending for a lease, which is what the store's design argues hardest for and
had no coverage at all. The mid-put test now uses a different-length replacement;
its previous one was the same 8 bytes as the original and could not fail.
The Node capability notes and `plans/234` are updated to match, including the
key-folding divergence on case-insensitive filesystems, which the header
previously claimed did not exist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ShardNamespaceLike` is the structural projection of a real `DurableObjectNamespace`. Making `get` and `idFromName` optional on it let a `@lunora/platform` `ShardDirectory` through the fan-out entry points, and degraded every other consumer at the same time: ~74 call sites and every app's `env.SHARD` inherited two members that were suddenly `possibly undefined`. Six projects failed to typecheck as a result — five examples through `createDoAuthWiring`, whose `AuthNamespaceLike` needs both members, and the playground calling `env.SHARD.idFromName(...)` directly. All six were red on this branch and neither review pass covered them, because both were scoped to the last commit's diff rather than the branch. The widening belongs on the *input*, not on the binding type. `fanOut` and the `orchestrate*` family now take `ShardNamespaceInput = ShardDirectory | ShardNamespaceLike`, and `toDirectory` reads whichever spelling is present off an internal projection of both. A conforming directory is still accepted; a Cloudflare binding is back to having the two members it has always had. The generated auth declaration is typed `AuthNamespaceLike` rather than `ShardNamespaceLike` for the same reason — `createDoAuthWiring` resolves through `idFromName` + `get` with no `getByName` fallback, so both are load-bearing there and the declaration should say so. Regenerating the examples touches exactly those two lines; no unrelated drift came with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Everything the last round documented as a known limitation rather than fixing.
**`terminate` is now a barrier.** It wrote its tombstone through the store
directly, outside the engine's per-run activation queue, so an activation
already in flight ran to completion and its `save` landed after the tombstone —
the run reported `terminated`, then reported its output. The engine has no
cancellation hook, but every write it makes goes through the store it was
handed, so the guard goes there: writes for a terminated id are dropped. The
same wrapper hides tombstone and alias rows from the runtime, which is what made
a reloaded terminated run surface as "No workflow registered with id
@lunora/platform-node:terminated" instead of "this run is gone".
**A `get` whose result is never read no longer retains a descriptor.** Reading
always released deterministically, but an object that became unreachable
untouched held its handle until the process exited. A `FinalizationRegistry`
closes it on collection — measured at 38 descriptors held across 40 unread gets,
0 after a collection.
**The key → path mapping is injective.** Filesystems fold names R2 does not: a
case-insensitive volume made `A` and `a` one object, Windows strips a trailing
dot or space, and `x:y` names an NTFS alternate data stream. Each segment is now
percent-escaped on exactly those characters and decoded on the way out.
Lowercase keys — nearly all of them — still map byte-identically, so the bucket
directory stays browsable.
**The workflow suite runs against both stores.** It only ever ran on
`MemoryStore`, whose `structuredClone` round-trip is more permissive than the
JSON one the durable SQLite store (and the engine's own contract) uses. The
parameterisation immediately caught one: a timed-out `waitForEvent` resolves
`{ payload: undefined, type }`, and JSON drops the key. Asserted on the
observable rather than key presence, which holds for both.
Also: `journal_mode = WAL` on the workflow store, matching the two sibling
SQLite stores that already set it, instead of a docstring telling the caller to;
concurrency coverage for racing puts on one key and for `.lunora-tmp` draining;
and `R2BucketLike.put` accepts `ArrayBufferView`, which real R2 and this file's
own `uploadPart` both accept — the projection had drifted from what it projects.
`platform-node` joins the api-snapshot guard. It was held out as a spike whose
surface was expected to move; it now implements what it declares, and gating it
is how the second host stops drifting from the contracts unnoticed. That is the
whole reason `platform` and `platform-cloudflare` are gated, and coverage is not
a stability promise.
`plans/234-node-host-lint-fix.md` is deleted — a scratch to-do list for lint
fixes that all landed, and the file failing the prettier gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`check-roadmap-tiers` runs from `postinstall` and refuses to let `scripts/api-snapshot.js` and `ROADMAP.md` disagree about which stability tier a package is in. Adding `platform-node` to `TIER_1` claimed a full-SemVer-at-1.0 promise the roadmap does not make for it — the roadmap lists it as experimental, correctly — so every job on the branch died in its setup step rather than at the check it looked like. The roadmap is right and the script was wrong. `TIER_3` is precisely the case: covered by the guard, so the surface cannot drift unnoticed, with the snapshot header saying it carries no SemVer promise until the package graduates. That is already how `agent` and `ai` are handled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`NODE_CAPABILITIES.queues` was `unsupported` — the one `ctx.*` surface with no
Node backing at all. `createNodeQueueHost` gives it one: a `QueueBindingLike`
producer per declared queue over `_lunora_queue_messages`, and a batched
consumer feeding the same `dispatchQueueBatch` the Cloudflare host uses.
The contract is message-shaped, not job-shaped, and that is what decides the
implementation. A consumer receives a batch and settles each message
individually — `ack()`, `retry({ delaySeconds })`, or undecided, which workerd
treats as an implicit ack when the handler returns and a retry when it throws.
No general-purpose Node queue library expresses that; they deliver one job to
one handler and take its return value. Wrapping one would mean rebuilding the
batch layer anyway, on top of a store this package already has.
So it is a table on the same connection as the shard, KV, socket, scheduler and
workflow tables, and a queue survives a restart for the reason those do.
- `visible_at` carries both `delaySeconds` (capped at 12 hours) and the retry
backoff, and doubles as the in-flight window: claiming a batch pushes it out
by `visibilityTimeoutMs`, so a crash mid-handler redelivers rather than loses.
- Claiming counts the delivery; rescheduling only moves the message. Doing both
in one statement double-counted every retried attempt.
- Exhausting `maxRetries` re-enqueues onto a declared `deadLetterQueue` as a
real message, so that queue's consumer sees it like any other. With none
declared the message parks `state = 'dead'` rather than vanishing.
- `maxBatchSize` / `maxBatchTimeout` assembly: a full batch goes at once, a
partial one waits, so a trickle is not delivered one message at a time.
- All four content types, with `v8` the only one that round-trips a `Map`.
- `mode: "pull"` queues are written but never consumed — nothing here serves the
HTTP endpoint an external worker would poll.
Delivery is driven by `poll(now)` rather than a timer, because this host has no
dev server to own one, and an injectable clock is what makes the delay and
backoff behaviour testable without waiting on wall time. Settlement uses that
same clock; using `Date.now()` instead drifted from it by however long the
handler ran.
`plans/306-pluggable-queue-drivers.md` and #360 propose the follow-on: a driver
seam so a deployment picks its own backend. SQLite's single writer is the
ceiling here — right for one process, wrong for several.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first draft was prose with a provider table. Rewritten to `plans/TEMPLATE.md` (§0–§9) so it is executable rather than merely informative: - **§1** carries file:line evidence for every claim about the current contract — the template's rule is that a claim without a reference is a guess, and this plan's whole argument rests on what the contract actually says. - **§1.3** links every provider to its own docs for the specific mechanism claimed (SQS `ReceiveMessage`, RabbitMQ `prefetch`, JetStream `fetch`), so the survey can be re-checked without redoing it. Where a claim could not be verified it says so: `pg-boss` advertises batch work and dead-letter redrive, but its published README contains none of the option names, so adapting it starts with confirming them. - **§4** records each decision against the alternative it beat, including why the engine sits above the driver and why receive/settle beat subscribe/handle. - **§8** gets STOP conditions that can actually fire — a TCK that cannot be written identically for all drivers means the contract is under-specified, and an extraction that changes any of the 9 existing SQLite tests changed behaviour. - **§10** is new: the `QueueDriver` contract, the engine that consumes it, and what swapping a backend looks like at a call site. A `limits` member lets a driver declare what it cannot do natively — SQS caps delay at 15 minutes against the contract's 12 hours, and Kafka cannot settle one message without its neighbours — so the engine can compensate and the TCK can assert the degradation instead of it surfacing in production. Also states the parity row (§6): a driver does not make `ctx.queues` native on a target, because the platform is not providing the feature — a driver is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The API sketch was designed from the contract alone. It matched none of the
conventions this repo already has for the same problem, and it missed four
things a project that has actually shipped a pluggable-backend queue got right.
**In-repo (§2.1).** Three vocabularies exist and are not interchangeable:
`*Driver` is a deploy target (`DeployDriver`), `*Adapter` is a stateless
provider translator (`PaymentAdapter`, replica's SQLite adapters), `*Store` is
pluggable persistence (`RateLimitStore`). A queue backend is the middle one, so
`QueueDriver` became `QueueAdapter` throughout — the old name claimed something
this is not.
`@lunora/payment` solves the same shape for five payment providers, so its
decisions carry: a `readonly identifier`, a *named* `ProviderCapabilities`
interface rather than an inline object, and optional methods to mean "this
provider cannot do X" with the fallback documented on each. That last idiom is
better than the boolean it replaced — an absent method cannot be called by
mistake, where `supportsPartialSettle: false` can.
Disposal follows `NodePlatform`'s `close()` + `Symbol.dispose`, with the async
variant used here and the divergence stated: a network-backed adapter has
in-flight requests to drain.
**External (§2.2).** `@openqueue/sdk`'s contract was read at source rather than
skimmed, being the only surveyed project solving the pluggable-backend problem
rather than the queue problem. Four things adopted:
- `specVersion` plus a `validateQueueAdapter()` guard, so a third-party adapter
built against an older contract fails with "upgrade in lockstep" instead of at
the first missing method.
- Capabilities enforced before dispatch via `assertCapability` and a typed
error, not merely documented — an unsupported call should fail by name rather
than degrade silently.
- `migrations` as an optional member, present only on adapters that own SQL
schema. Both SQLite and Postgres do, and the plan previously had no answer for
who creates the table or how you know it is current.
- Import-cleanliness as a **gate**: the contract entry point must import no
backend client, or every consumer pays for all of them.
And one thing that confirms a decision already taken: their delivery interface
is `consume(queue, { process(job) })` — one job at a time — and having chosen
subscribe/handle, the framework has no batch delivery at all. That is the
clearest evidence available that receive/settle is the right shape here, from
the project that went furthest the other way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…host
Two blockers, both mine, both from the same commit.
**The capability said `emulated` and nothing composed the host.** `queues` was
flipped from `unsupported` in the commit that added `createNodeQueueHost`, but
`createNodePlatform` never built one — the only references to it in the repo
were its own export and its test — while `node-driver.ts` still warned that
`ctx.queues` would be unavailable. `gateAgainstMatrix` omits a surface and emits
`platform_unsupported_feature` only for `"unsupported"` and undeclared, so
`"emulated"` un-gated the whole `ctx.queues` surface with no diagnostic
anywhere. A node-target app declaring a queue got emitted types, a warning
saying it would not work, and no binding.
`createNodePlatform` now builds the host from `queues` + `onQueueBatch`, and is
generic over the declared queues so the binding map keeps its keys — erased to
`Record<string, …>` every binding reads as possibly-undefined and the wiring is
typed uselessly. Passing `queues` without `onQueueBatch` throws rather than
storing messages nothing consumes. The stale node-driver warning is gone.
**`maxRetries` was off by one, in the direction that disagrees with the helper
this host feeds.** `dispatchQueueBatch` sets `deadLettered` on
`attempts > maxRetries`; this used `>=`, so it buried every message one delivery
early and each one was captured `deadLettered: false` — the Studio Queues panel
could never show one. `maxRetries: 1` retried zero times. Cloudflare counts
retries after the initial delivery, so `maxRetries: 2` is three deliveries; the
test froze the wrong number and now asserts the right one.
Also in the queue host:
- **The claim is one statement.** `SELECT` then a separate `UPDATE` is safe
within a process — better-sqlite3 is synchronous, nothing interleaves — but
two processes on one file both read the same pending rows and both deliver,
each telling its handler `attempts: 1`. Reproduced. Now a single
`UPDATE … WHERE id IN (SELECT …) RETURNING *` inside an `immediate()`
transaction, which takes the write lock up front instead of upgrading. Since
`RETURNING` yields post-update rows, `row.attempts` *is* the delivery count
and the `+ 1` that compensated for the old ordering is gone.
- **A dead-letter queue that names itself redelivered forever** — the re-enqueue
resets `attempts`. One that names an undeclared queue wrote a row nothing
polls and nothing reaps, which is the silent drop the parking behaviour exists
to prevent. Both are rejected at construction, where Cloudflare rejects them.
- **Producers and `poll()` share one clock.** `send` used `Date.now()` while
`poll(now)` took an injectable one, so a test that captured `now`, sent, and
polled at `now + delay` missed by the elapsed milliseconds — the delay test was
already failing intermittently in full-suite runs. `options.now` is threaded
through both.
- `deadLettered()` became `deadLetters.list` + `.requeue`, matching
`SchedulerHost.deadLetter`. A parked message you can read but not requeue is
half of what "a dropped message with no trace is impossible to debug" asks
for, and the api-snapshot had frozen the half.
- `maxBatchTimeout` is clamped to the documented 0-60; a corrupt body no longer
fails the whole dead-letter listing.
R2, from the key encoding added last round:
- `list()` threw a raw `URIError` on any filename that is not a valid escape —
a hand-dropped `100%` took down the entire listing, contradicting the design
one function over, where `readTrailer` returns `undefined` so unrelated files
drop out of the page instead of failing it.
- Escaping triples an uppercase character, so a 200-character key inside the
documented 1024-byte ceiling produced `ENAMETOOLONG` from `rename`. The
segment limit is now checked on the encoded form.
Runtime: `toDirectory` returns a `ShardDirectory` unchanged instead of
rebuilding it — it was allocating an object and three closures to re-derive
`idForName` from `idForName`. That early return is what removes the cast and
the all-optional projection, since everything past it is a binding where `get`
and `idFromName` are required. Narrowed with a type guard on `idFromName`,
because `DirectShardDirectory` declares `idForName` optionally and an `in` check
cannot exclude it.
Workflows: `create({ id })` naming an existing run is refused. The alias write
is an upsert, so it replaced that run's row with a pointer to a new one and the
completed run read back as `unknown` with its output gone — which is what the
natural "retry the create with the id I was given" pattern did.
New tests: cross-process redelivery after a consumer dies holding a batch (the
guarantee the docstring leads with and nothing covered), `maxRetries` counted as
retries, self-referencing and undeclared dead-letter rejection, the 12-hour
delay clamp, content-type validation, the undecodable filename, the escaped
segment ceiling, and the platform binding what it declares. 144 in the package.
Docs: `plans/234` said queues remain unsupported two lines above describing the
implementation; the rejected-alternative preamble moved to `plans/306`, which is
where it already lives at greater length. Plan 306 answers its own blocking
question rather than opening on a decision it declines to make, defines the four
types §9 referenced but never declared, merges the duplicated workstream and
phase tables, and stops restating its headline four times.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abe4bd6 to
765f997
Compare
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
What this is
Takes
@lunora/platform-nodefrom the plan-234 spike to a host that implements what it declares — so it arrives reviewable and testable as a whole rather than assembled in place across releases.Durability was the through-line
The spike persisted state and never restored it. Alarms wrote a timestamp nothing re-armed; scheduler jobs lived in in-process
Maps; socket attachments and tags did too. Every one of those reported success while losing the work.Each contract now owns both halves, and construction is the wake: alarms and jobs re-arm from their durable rows, one whose time elapsed while the process was down fires late rather than never, and socket state comes back from
_lunora_sockets. Two restart tests — a second host over the same database file — pin what the TCK's simulated recycle cannot reach.The directory resolved without dispatching
It answered
new Response(name)— the shard key echoed back. That passed both TCK directory legs with no shard behind it, which made cross-shard fan-out unimplementable.createNodeShardRegistryresolves and hosts: one key, one database, oneShardHost, one KV store, one socket registry.listShardKeysseeds from the shard files on disk — without that, a fan-out after a restart returns an empty result indistinguishable from "no rows matched". Fan-out is proven against the real@lunora/runtimequery coordinator over three databases, including a failing shard degrading to a partial answer rather than an exception.Object storage and workflows
createNodeR2Bucketis anR2BucketLikeover the filesystem, andcreateNodeWorkflowHostcompilesdefineWorkflowhandlers onto the@visulima/workflowengine. Two design points are worth a reviewer's time:An R2 object is exactly one file with a metadata trailer (
<body><trailer JSON><uint32be length>"LNR1"), not a body file plus a sidecar. Two files cannot be published by one filesystem operation, so a crash between them left new bytes carrying the previous checksum, size and content-type — whichget/headthen reported as fact. The structural cost was larger than the crash window: because a sidecar can always be missing, every read carried a fall-back-to-statpath that fabricated an etag for anything it could not explain, including files that were never ours. One file makesreadTrailerFromtotal and deletes that branch.getholds a single handle from trailer through body, so an overwrite mid-read cannot serve the new bytes under the old metadata; the body is single-use, as R2's is.Workflow
storeis required. The only available default was the engine's in-processMemoryStore, and a host that silently loses every run on restart should not be what a caller gets for saying nothing.createNodeWorkflowStoreis a SQLiteWorkflowStorewith a single-statement atomic lease.create({ id })is honoured through a durable alias row rather than refused —ctx.spawnmints a child id, callscreate({ id }), thenget(id), so refusing it takes fan-out out entirely.Also
.global()tables over the@lunora/sql-storecore.@lunora/configdeploy driver, so--target noderesolves.inferis shared with Cloudflare verbatim;provisionreports what this target cannot serve instead of writing config;toolchainis absent because there is no vendor CLI to shell out to.createNodeShardState, presenting a shard in theDurableObjectStateshapeShardDOexpects.Two things worth a reviewer's attention
A porting blocker in
@lunora/runtime, and the shape of its fix. EveryQueryCoordinatorfan-out entry point required Cloudflare'sidFromName; the platform contract spells itidForName. A fully conformingShardDirectorywas rejected over one method name, and nothing caught it because the only namespace ever passed was a realDurableObjectNamespace.The first attempt made
getandidFromNameoptional onShardNamespaceLike. That was wrong, and the branch carried it for several commits: that interface is the structural projection of a realDurableObjectNamespace, so ~74 call sites and every app'senv.SHARDinherited two members that were suddenlypossibly undefined— six example/app typechecks went red. The widening belongs on the input.fanOutand theorchestrate*family now takeShardNamespaceInput = ShardDirectory | ShardNamespaceLike, and the binding type is unchanged from before this branch. The generated auth declaration is typedAuthNamespaceLike, which requires both members, becausecreateDoAuthWiringresolves throughidFromName+getwith nogetByNamefallback.ShardDOcannot be handed a platform. It builds the Cloudflare adapters fromstaterather than accepting the contracts a host already implements, so the seam stops one layer below where a second host needs it.createNodeShardStateadapts backwards as the smaller mistake: ~100 lines that delete themselves onceShardDOtakes aShardPlatform, against 8,700 that would diverge if forked. The real fix belongs in@lunora/doand is filed as a finding.What is still missing, and is stated as such
No dev server, so there is no
lunora dev --target nodeyet.NODE_CAPABILITIESrates almost nothingnativeon purpose — the matrix defines that as "the platform provides it", and a bare Node process provides none of this. Fan-out,.global()tables, alarms, scheduler, sockets, workflows and object storage areemulatedwith notes saying how well; every Cloudflare product binding staysunsupported.Three limitations are recorded in the capability notes rather than fixed:
terminateis a barrier only within one process (the engine exposes no cancellation hook, so the guard lives on the store handed to the runtime);ctx.rundispatches to an endpoint no Node HTTP server serves; andctx.parallel's join cannot interleave inside one synchronoustriggeractivation.API surface
platform-nodejoins the api-snapshot guard at the experimental tier — the snapshot records how the surface moves and carries no SemVer promise until the package graduates, which is howagentandaiare already handled.ROADMAP.mdpublishes that tier andcheck-roadmap-tiersfails the install if the two disagree.Verification
platform-node 129/129 · runtime 845/845 · config 584/584 · codegen 1068/1068 · platform 45/45 ·
api:check47/47 snapshots · types, eslint, prettier, package.json sort clean.plans/234-node-host-findings.mdcarries the full record, including the findings this construction surfaced.🤖 Generated with Claude Code
Summary by CodeRabbit