feat: GROUNDSKEEPER (v7.2.0) — Self-Managing Infrastructure #8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix version reality: v7.1.0 complete on main (unreleased), GROUNDSKEEPER in progress - Reorder GROUNDSKEEPER: FRONTIER first (multiplier), IDX second, GC last - Make GC and index auto-rebuild opt-in with warn-by-default semantics - WEIGHTED mixed-version sync: fail fast with E_SCHEMA_UNSUPPORTED, never silently drop - Elevate auto-materialize coalescing from edge-case test to core invariant - HANDSHAKE deletion guards: two-layer validation (best-effort at build, authoritative at commit) - LIGHTHOUSE receipts: reinforce zero-cost invariant when disabled - Frontier storage: CBOR authoritative, JSON as debug artifact
O(writers) method comparing stored writer tip SHAs against current refs. Returns true if any writer tip has changed since the last materialize(), enabling cheap "has anything changed?" polling without full materialization. Closes GK/FRONTIER/1.
Adds frontier.cbor (authoritative) and frontier.json (debug) to the bitmap index tree when a frontier Map is provided to serialize/finalize. Threads frontier option through IndexRebuildService.rebuild(). Closes GK/IDX/1.
New IndexStalenessChecker with loadIndexFrontier() and checkStaleness(). On index load, reads frontier metadata, compares against current refs, and warns when stale. Opt-in autoRebuild triggers automatic rebuild. Closes GK/IDX/2.
After materialize(), check GC metrics and warn when thresholds are exceeded. Execute GC only when gcPolicy.enabled === true. GC failure never breaks materialize. Adds logger parameter to WarpGraph.open(). Closes GK/GC/1.
📝 WalkthroughWalkthroughAdds frontier metadata serialization (CBOR + canonical JSON), frontier-aware index staleness detection with optional auto-rebuild, WarpGraph frontier-change checking via hasFrontierChanged(), and opt-in post-materialization GC wiring and logging; tests exercise frontier serialization, staleness logic, hasFrontierChanged, and GC behavior. Changes
Sequence DiagramssequenceDiagram
participant Client as Client
participant WarpGraph as WarpGraph
participant Storage as Storage
Client->>WarpGraph: hasFrontierChanged()
activate WarpGraph
WarpGraph->>Storage: read writer tips / frontier
Storage-->>WarpGraph: current frontier (Map)
WarpGraph->>WarpGraph: compare current vs _lastFrontier
WarpGraph-->>Client: Promise<boolean>
deactivate WarpGraph
Note over WarpGraph,Storage: materialize() updates _lastFrontier and may trigger GC
Client->>WarpGraph: materialize()
activate WarpGraph
WarpGraph->>WarpGraph: perform materialization
WarpGraph->>WarpGraph: _lastFrontier = getFrontier()
WarpGraph->>WarpGraph: _maybeRunGC(state)
alt gcPolicy.enabled && thresholds met
WarpGraph->>WarpGraph: run GC
WarpGraph->>WarpGraph: log info (logger)
else thresholds met && !enabled
WarpGraph->>WarpGraph: log warning (logger)
end
WarpGraph-->>Client: materialize result
deactivate WarpGraph
sequenceDiagram
participant Client as Client
participant IndexRebuild as IndexRebuildService
participant Staleness as IndexStalenessChecker
participant Storage as Storage
participant Builder as Bitmap/StreamingBuilder
Client->>IndexRebuild: load(treeOid, { currentFrontier, autoRebuild })
activate IndexRebuild
IndexRebuild->>Storage: read shard OIDs
Storage-->>IndexRebuild: shard OIDs
IndexRebuild->>Staleness: loadIndexFrontier(shardOids)
activate Staleness
Staleness->>Storage: read frontier.cbor (or frontier.json)
Storage-->>Staleness: frontier blob
Staleness-->>IndexRebuild: indexFrontier (Map)
deactivate Staleness
IndexRebuild->>Staleness: checkStaleness(indexFrontier, currentFrontier)
Staleness-->>IndexRebuild: { stale, reason, details }
alt stale && autoRebuild
IndexRebuild->>IndexRebuild: rebuild({ frontier: currentFrontier })
IndexRebuild->>Builder: serialize/finalize({ frontier })
Builder->>Storage: write shards + frontier.cbor + frontier.json
IndexRebuild-->>Client: rebuilt index
else stale && !autoRebuild
IndexRebuild-->>Client: stale index + warning
else not stale
IndexRebuild-->>Client: current index
end
deactivate IndexRebuild
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
index.d.ts (1)
506-508:⚠️ Potential issue | 🟠 MajorUpdate BitmapIndexBuilder.serialize typings to accept frontier.
The implementation now supports
serialize({ frontier }); the type definition still exposes a zero-arg signature, which blocks TS users from passing the new option.🧩 Proposed fix
- /** Serializes the index to a tree structure of buffers */ - serialize(): Record<string, Buffer>; + /** Serializes the index to a tree structure of buffers */ + serialize(options?: { frontier?: Map<string, string> }): Record<string, Buffer>;
🤖 Fix all issues with AI agents
In `@ROADMAP.md`:
- Line 227: Add a language identifier to the fenced code block to satisfy
markdownlint MD040 by changing the opening triple-backtick fence to include a
language (e.g., use "```text" instead of "```"); locate the lone fenced block in
ROADMAP.md (the opening "```" shown in the diff) and update it accordingly so
the block is "```text" and the closing fence remains unchanged.
In `@src/domain/services/IndexStalenessChecker.js`:
- Around line 15-28: The loadIndexFrontier function currently assumes the
decoded envelope contains a valid frontier object and will throw an opaque
TypeError if malformed; update loadIndexFrontier to validate the parsed envelope
(after cborDecode or JSON.parse) by checking that envelope is an object,
envelope.frontier exists and is an object (or map-like), and that
Object.entries(envelope.frontier) is safe to call; if validation fails, throw a
clear, descriptive error mentioning the blob type and shardOids key (e.g.,
'invalid frontier envelope for frontier.cbor' or 'frontier.json') so callers can
diagnose corrupted or future-version blobs; keep validation logic adjacent to
the cborDecode and JSON.parse usage and ensure you use the same variable names
(envelope, buffer, cborOid, jsonOid) shown in loadIndexFrontier.
In `@src/domain/services/StreamingBitmapIndexBuilder.js`:
- Around line 355-365: The frontier.json must be written using canonical
(sorted-key) JSON; change the JSON serialization for envelope (the code that
creates the jsonOid via
this.storage.writeBlob(Buffer.from(JSON.stringify(envelope)))) to use a
canonical JSON encoder instead (either a small helper that recursively
serializes objects with Object.keys(...).sort() or a
stable-json-stringify/canonicalize utility) so keys are lexicographically
ordered across the entire envelope; ensure you use that helper in the
frontier.json write path (where jsonOid is created) and keep the existing sorted
frontier object and envelope structure.
🧹 Nitpick comments (2)
src/domain/services/IndexRebuildService.js (2)
5-109: Document the newfrontieroption forrebuild().
The frontier propagation looks good, but the JSDoc doesn’t mention the new option, which makes it harder to discover.📚 Suggested JSDoc update
- * `@param` {AbortSignal} [options.signal] - Optional AbortSignal for cancellation support. + * `@param` {AbortSignal} [options.signal] - Optional AbortSignal for cancellation support. + * `@param` {Map<string, string>} [options.frontier] - Frontier to persist alongside the rebuilt index.
282-315: Document staleness options onload().
The staleness check and auto‑rebuild flow are clear, but the newcurrentFrontier,autoRebuild, andrebuildRefoptions aren’t described in the JSDoc.📚 Suggested JSDoc update
- * `@param` {boolean} [options.strict=true] - Enable strict integrity verification (fail-closed). + * `@param` {boolean} [options.strict=true] - Enable strict integrity verification (fail-closed). + * `@param` {Map<string, string>} [options.currentFrontier] - Frontier to compare for staleness. + * `@param` {boolean} [options.autoRebuild=false] - Auto-rebuild when a stale index is detected. + * `@param` {string} [options.rebuildRef] - Ref to rebuild from when autoRebuild is true.
- Update BitmapIndexBuilder.serialize typings to accept frontier option - Add language identifier to ROADMAP.md fenced code block (MD040) - Validate frontier envelope in loadIndexFrontier before accessing fields - Use canonical sorted-key JSON for frontier.json serialization - Document frontier param in IndexRebuildService.rebuild() JSDoc - Document currentFrontier/autoRebuild/rebuildRef in IndexRebuildService.load() JSDoc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
graph.hasFrontierChanged()— O(writers) cheap change detection without materializationfrontier.cbor+frontier.jsonat build time for staleness trackingIndexStalenessChecker— on index load, compare frontier metadata against current refs; warn when stale, opt-inautoRebuildmaterialize()checks GC thresholds post-run; warn by default, execute only whengcPolicy: { enabled: true }Test plan
npm run test:local— 1605 tests pass (71 files, including no-coordination regression suite)npm run lint— cleannode scripts/roadmap.js status— GROUNDSKEEPER at 100% (4/4)WarpGraph.frontierChanged.test.js(7),BitmapIndexBuilder.frontier.test.js(8),IndexStalenessChecker.test.js(13),WarpGraph.autoGC.test.js(6)Summary by CodeRabbit
New Features
Tests
Documentation